Innovative Routines International (IRI), Inc. (The CoSORT Company) respects your time and privacy. You can stop or start future quarterly mailings at any time: click here or on the links at the bottom of this message. Please forward this newsletter to anyone interested in IRI or our high-performance data manipulation and management  tools, or open systems data processing technology.

 

FAst extraCT Logo       CoSORT Logo       RowGen Logo
The CoSORT Journal: Data Sorting and ETL News
                                                                                  Quarter 3, 2006
In this issue:

Coming Soon: FAst extraCT v2
ETL Tool MD Bridges to CoSORT
Next Trade Conference Exhibits
More International CoSORT Agents
Tech Tip: /ALTSEQ
ETL Tool File Metadata Use in CoSORT & RowGen

CoSORT (IRI, Inc.) and Meta Integration Technology, Inc. (MITI) recently announced a new metadata export bridge from file layout metadata repositories in Informatica, DataStage and other  tools to SortCL and RowGen data definition file (.DDF) format.

The new DDF export bridge in the Meta Integration Model Bridge (MIMB) suite helps data warehouse architects run (faster) SortCL file transforms and build RowGen test data alongside existing ETL tools -- leveraging file metadata already created for the ETL tool.
The bridge’s automatic metadata conversion preserves users’ investments in their metadata and eliminates the need to manually re-define file layouts for CoSORT/SortCL or RowGen use. 

So for example, instead of sorting and joining flat files using an ETL or database engine, you can quickly export the metadata in SortCL format for faster CoSORT executions, taking the load off  the tools which are not optimized for such high data volumes.

Similarly, the RowGen DDF Export Bridge facilitates the creation of realistic test data because its descriptions already exist for processing in your BI, ETL, or EAI tool. Because the MIMB .DDF bridge migrates file and field layouts perfectly, you can easily create ideal ETL and DB test data in real file and table formats.

Click here
for more information on CoSORT's MIMB connections.

Next IRI Exhibits: ACUCOBOL, Natural & Oracle

CoSORT / IRI, Inc. will be a silver sponsor at the ACUCOBOL-GT:

ACUCOBOL

2006 Developers' Conference
Vendor Forum
September 13-15, 2006
Manchester Grand Hyatt
San Diego, California

An IRI engineer will demonstrate a fast CoSORT replacement for ACUCOBOL-GT sorting, and SortCL's unique collation, conversion, and reporting capabilities for Vision index files.

The following month, CoSORT is on display at the 14th annual:

Software AG Natural Conference

Software AG Natural Conference
Booth P
October 17-20, 2006
Hyatt Regency - Penn's Landing
Philadelphia, Pennsylvania

IRI's exhibit will feature CoSORT's drop-in replacement to the SAG Natural sort and the new RowGen test data and file synthesizer.

The following week will mark IRI's ninth consecutive exhibit at:

Oracle OpenWorld
Booth #1124

October 22-26, 2006
Moscone  Convention Center
San Francisco, Ca
lifornia

With the innovations described at left, CoSORT and FACT users will be able to process more data in the same I/O pass during combined Oracle Extract-Transform-Report-Load operations!

Greece, New Zealand & Switzerland!

IRI is pleased to announce that CoSORT, FACT and RowGen are represented by resellers and service providers in: Greece by LogiDIS, in New Zealand by Generic Systems, and in Switzerland by DPCS.

If you work in these countries, please click on the links above to email your new agent, and mention some of your data processing challenges on open systems. In addition to IRI products, these companies offer long consulting experience and software solutions for legacy migrations, data warehousing and data quality, and decision support for many vertical industries.

About CoSORT / IRI, Inc. and The CoSORT Journal
  • CoSORT solutions serve data warehouse (ETL) architects, very large database (VLDB) administrators, mainframe sort migrators, and developers and independent software vendors (ISVs) building faster sorting and data transformation into their applications.
  • CoSORT delivers the IT industry's fastest parallel UNIX sort engine combined with a versatile flat-file manipulation and reporting program -- SortCL -- which combines: 
    row filtering and conditional selection, sort/merge and joins, drill-down aggregation and cross-row calculation, conversion and collation of more than 100 data types, database sequencing, and multi-target, multi-level output reformatting for reports, hand-offs, and DB load utilities.
  • Other special CoSORT features include: coroutine sort architecture; fully tunable and scalable parallel sort performance on all multi-CPU UNIX and Windows servers; cross-calculation on aggregated values and aggregation on cross-calculated values, cross-table joins (matching) integrated with data conversion and expression logic; multinational date and timestamp support; cross-platform Java GUI; and, e-commerce reporting via CSV/CLF and IP Address manipulation, plus ELF input and HTML output.
  • CoSORT offers parameter converters for legacy JCL sort software, as well as drop-in sort verb replacements for:
    ACUCOBOL-GT, Amdocs Ensemble, DataStage Server; Informatica PowerCenter; Cincom Supra; the DB2 loader, MF COBOL Workbench, Net and Server Express; SAS 8, Software AG Natural; Sun MRP, and UNIX (/bin/sort).
  • IRI offers other data manipulation and management solutions like: FACT for fast unloads from Oracle; RowGen for custom data generation and format simulations; netCONVERT for mainframe tape data conversion and reformatting; x-PRESS for fast, and secure data (de)compression; Logon for controlling and auditing access to UNIX systems; and Permitas for licensing and activating software applications.
  • The CoSORT Journal is a quarterly Email newsletter designed to keep subscribers updated on salient news and events at IRI, Inc. Past editions are archived here.

To remove or add an Email address in future CoSORT Journal mailings, please email news@iri.com. To contact an IRI agent, click here, call 1-800-333-SORT, or email info@iri.com.

 

FAst extraCT (FACT) Version 2 is Near 

The next major release of IRI's unload software for faster ETL and reorgs will ship soon. Among the new features in FACT for Oracle on UNIX & Linux are:

1) Support for CLOB and BLOB data extraction.

2) Support for more .ini environment variables; i.e.
    UserID, password, and select query math.

3) Support for LDAP connections to Oracle 10G.

4) Outfile extension entries (.dat., .log, .ctl, etc.) and  
    outfile naming (with paths) can now be specified
    through one parameter, reducing .ini file sizes.

5) New MAX ROWS parameter sets the processing 
    buffer size for extraction performance tuning.

6) TIMESTAMPFORMAT option now supports the
    Oracle 9i format of YYYYMMDDHH24MISSFF
    (previously YYYYMMDDHH24MISS.FFFFFFFFF)

 
Tech Tip: Special Collating Sequences

Beyond Mainframe ALTSEQ 

If the collating sequence of your data differs with the natural ordering of that data type, you can either write a custom compare procedure (/KEYPROCEDURE), or use SortCL's /ALTSEQ statement. ALTSEQ supports input fields declared as ASCII (the default) or EBCDIC. 

As usual, SortCL exceeds the above mainframe sort functionality by also enabling ALTSEQ value substitutes in output field displays, and for conditional evaluations applicable in SortCL.

The syntax for specifying an alternate sequence is:

/INFILE(S)=path(s)/filename(s) /ALTSEQ=(hex_value1Ahex_value1B
     
[,hex_value2Ahex_value2B][,etc.])
where the character represented by hex_value1A will be replaced with the character represented by hex_value1B. You can also specify additional replacements (offset with a comma) such as substituting hex_value2A for hex_value2B, and so on. Using hex values allows for non-printable characters.

For example, given this input file, sales.dat:

A book           6.98
C blouse        23.45
1 tablet         2.45
B yarn          10.78
3 skirt         78.98
C coat         235.97
2 thread         4.25
A pen,blk        2.98

The SortCL job specification file below redefines the 'dept' field to replace every hex 41 value (=A) with a hex 31 value (=1) and so on. The virtual /INREC record definition converts the dept field before the sort.

/INFILE=sales.dat
    /ALTSEQ=(4131,4232,4333)
    /FIELD=(dept,pos=1,size=1)
    /FIELD=(alt_dept,pos=1,size=1) 

    /FIELD=(item,pos=3,size=10)
    /FIELD=(amount,pos=14,size=8, NUMERIC)
/INREC
    /FIELD=(dept,pos=1,size=1)
    /FIELD=(alt_dept,pos=2,size=1, ALTSEQ)
    /FIELD=(item,pos=3,size=10)
    /FIELD=(amount,pos=14,size=8, NUMERIC)
/SORT
    /KEY=(dept, ALTSEQ)
    /KEY=item
/OUTFILE=dept1
    /INCLUDE WHERE alt_dept == "1"
    /FIELD=(alt_dept,pos=1,size=1)
    /FIELD=(item,pos=3,size=10)
    /FIELD=(amount,pos=14,size=8, NUMERIC)
/OUTFILE=dept2
    /INCLUDE WHERE alt_dept == "2"
    /FIELD=(dept,pos=1,size=1)
    /FIELD=(item,pos=3,size=10)
    /FIELD=(amount,pos=14,size=8, NUMERIC)
/OUTFILE=dept3
    /INCLUDE WHERE alt_dept == "3"
    /FIELD=(dept,pos=1,size=1)
    /FIELD=(item,pos=3,size=10)
    /FIELD=(amount,pos=14,size=8, NUMERIC)

The end result are three conditionally segmented output files in the same sort order; first by the converted dept values, then by item. 

In the first output file, dept1, note that we displayed the changed 'dept' field value: 

1 book           6.98
1 pen,blk        2.98
1 tablet         2.45

whereas in the output of dept2 and dept3, the original dept field value is kept (though its sort order is based on the alternate dept value). dept2:

2 thread         4.25
B yarn          10.78

dept3:
C blouse        23.45
C coat         235.97
3 skirt         78.98

As always, email questions to  support@iri.com.

Copyright © 2006 Innovative Routines International (IRI), Inc. 2194 Highway A1A, Suite 303, Melbourne, FL 32937-4932. All rights reserved.

CoSORT
®, SortCL, Rowgen and Permitas are trademarks of IRI. FACT is a trademark of IDS Ltd. (CoSORT Korea). Other product or brand names mentioned herein may be (registered) trademarks or copyrights of their respective owners.