Innovative Routines International (IRI), Inc., The CoSort Company

High-Speed Control of High-Volume Data

What Is SortCL?

A) A parallel ETL engine for data staging and integration
B) A file manipulation, comparison and conversion tool
C) A custom detail, delta, and summary report generator
D) 'Beyond the mainframe' sort/merge/sum software
E) A protection and governance tool for flat-file data
F) All of the above

Answer: F) All of the above. See also: FAQ > ETL

The Sort Control Language (SortCL) program is the most powerful and popular tool in IRI's flagship CoSort package. SortCL uses an open, explicit metadata for data definition and manipulation. SortCL's self-documenting 4GL job scripts perform - and combine - high-speed data transformation, conversion, reporting, and protection in large file environments.

From a solution standpoint, SortCL can be used to:

• Replace slower 3GL, Perl and SQL jobs
• Feed, speed, or re-seed BI/DW tools
• Relieve application and system overhead
• Filter, integrate, and stage large files
• Generate custom reports and hand-offs
• Accelerate bulk DB reorgs and loads
• Mask and encrypt fields and records
• Convert legacy sorts and data types
• Migrate between flat and index files
• Synthesize safe test data in real formats
• Detect, capture, and audit changed data

What Can SortCL Do?

CoSort's Sort Control Language (SortCL) program can accept multiple inputs (files, pipes and procedures) in multiple formats, process them in many ways, and produce one or more custom-formatted reports and output targets, all at once.

Specifically, SortCL can, in one job script and I/O pass, rapidly perform and combine all of these data transformations:

Filter	At the byte, field and record level, plus duplicate removal and saving
Segment	Conditional (include/omit) selection with if-then-else, else-if logic
Sort	Multiple keys, directions, sequences
Merge	Two or more pre-sorted files
Join	Two or more un/sorted files on many conditions for file compares and change data capture (deltas)
Re-map	Resize, reposition, and realign fields
Convert	Change data types (e.g. EBCDIC<>ASCII, Packed<>Numeric)
Re-format / Interchange	Convert between file formats (e.g. Text <>XML, VS<>RS, ISAM<>Vision, LDIF<>CSV)
Aggregate	Parallel roll-up and drill-down sum, min, max, average, and count values. Accumulation. Ranking.
Calculate	Math and trig functions across detail and summary rows
Sub-string	Bit-level manipulations and Perl-compatible regular expression logic for pattern matching, etc.
Validate	Check that character and field attributes match their specifications (i.e. "iscompares", gap analysis)
Sequence	For custom indexing, reporting, and database load operations
Lookup	Discrete field substitutions, pseudonymization, etc. using "set" file field dimensions
Protect	Encrypt data at the field level and audit data security measures. Also: anonymization, de-identification, filtering, and pseudonymization
Prototype	Create randomly-generated or set-selected (safe) test data fields
Report	Custom-formatted, segmented detail and summary targets
Custom	Complex field-level user functions (e.g. 3rd-party DQ libraries)

How Do I Run SortCL?

SortCL job scripts are called from the command line, in a batch script, through a job scheduler, from a program (as an executable or C API call), or via CoSort's free Java gui2scl (Graphical User Interface to SortCL) client. The GUI allows you to import, modify, create, save, print, share, validate, and execute SortCL jobs across the network.

What about Logging?

Application-level statistics can be output with each job; either to the screen or a file. In addition, the CoSort system logs runs in a self-appending file, and debugging information in a self-replacing file. On-screen monitoring options are available at various verbosity levels for runtime progress assessment. You can also enable and secure an XML audit log for validating compliance and performing forensic application and data analysis.

Where Can I Get More Information on SortCL?

SortCL plays in an integral role in almost every Solution area described on this web site. Explore this linked list for a description of SortCL's capabilities in:

• Business Intelligence
• Data Governance
• Data Transformation
• ETL/DB Acceleration
• Field Protection
• File Interchange
• Third-party Sort Migration