Version 9 is Here!

Announcing a New Platform for High-Volume Data Transformation and Protection

• Anonymization/obfuscation
• Encryption (256-bit AES and more)
• De-identification
• Redaction/filtering
• Safe test data synthesis
  

• A choice of data protections applied to fields on a need-to-know basis
• Portable (flat-file) protection that is not limited to one database
• Protection for data in motion and at rest (secure data to/from the DB)
• Availability of non-sensitive database, file, and disk information
• Combined protections in the same job script and I/O pass with high-volume data transformation, reporting, and test data synthesis
• A query-ready XML audit trail to help verify compliance with privacy rules, perform limited data forensics, and support your risk and controls framework
  

• Multi-file joins (3 or more extracted tables)
• Joins over pre-sorted or unsorted files
• Joins based on multiple conditions
• Mix and match multiple join types
• Multi-dimensional lookup (set) files
  

These powerful join features allow you to reduce database workloads and optimize large data warehouse ETL operations. SortCL continues to allow simultaneous aggregation, calculation, conversion, and multi-target output formatting.

V9's new field-matching functionality also includes multi-level value lookups on flat files. Lookup transformations during the action and output phases of your job script -- and value substitutions made from columns in delimited lookup files -- are designed to save time over direct computations and complex join operations.

See also: http://mcsv.net/cgi-bin/redir?MCid=g72Sbcq9lQ[UNIQID]
http://mcsv.net/cgi-bin/redir?MCid=Zx8Btecz6e[UNIQID]


Multi-Byte Collation & Conversion

Sorting Native Characters in Double-Byte Order

CoSort V9 now makes faster sorting and processing available for Asian data sets using double- and multi-byte characters. This is necessary for accurate permutation and conversion of large flat files containing Chinese, Japanese, and Korean key fields represented in native formats like BIG5, Shift_JIS and eHangul, respectively, among others. Please contact your IRI agent for more details.


Field-Level Functions, Custom Data Cleansing Support

User Libraries Enable Complex Transformations

CoSort's SortCL tool has always supported field filtering and scrubbing using built-in commands and user conditions. But now there are more possibilities for integrated, custom data cleansing. Data cleansing is important because it helps make field values and formats accurate and consistent with related data sets throughout your systems.

CoSort V9 now supports custom field-level transformations during the inrec and outfile phases of a SortCL job script. This means you can run custom transform functions - like data cleansing - on every field value, up to two times per job. Integrated field-level functions are important because they can - like so many other data manipulations - be combined in the same SortCL job script and I/O pass. This saves steps and time in the design and execution of high-volume transformation, reporting, and protection jobs.

Based on your business needs and modeling rules, you can plug in your own functions or those in data quality vendor libraries. Just specify your library paths at the top of the script, and your functions at the field level. You can also combine cleansing with scrubbing and bulk data reduction so you can simultaneously remove or save duplicate records, and include or omit others based on business and compliance rules.

See also: http://mcsv.net/cgi-bin/redir?MCid=XHvyDPFX5E[UNIQID]

Manipulate and Convert Large XML Files

Need XML Files From Other, Large Flat File Formats? Try CoSort 9

Although XML is an increasingly popular file interchange format, it has not been a practical structure for large files. That's because a 30GB XML file may contain 20GB worth of tags. And, until now, there simply has been no efficient way to rapidly convert, process, protect, or create large transaction files in XML. However, that changes with the new CoSort V9 package.

CoSort's SortCL tool can now process, convert, and create large, flat XML files. You can direct SortCL to filter, transform and output valid, well formed XML files that contain structured data from XML or any other supported flat file source -- of any size. The reverse is also true; SortCL can process huge, flat XML files and output the data in other formats, including ACUCOBOL Vision, CSV, ISAM, Sequential, and ELF.

See also: http://mcsv.net/cgi-bin/redir?MCid=uTlU0IwqG0[UNIQID]

Manipulate and Convert Large LDIF Files

CoSort Puts DAP Data Under Your Control

LDIF is an interchange format for representing LDAP (Lightweight Directory Access Protocol) contents and update requests. LDAP directories hold information with similar attributes, organized both logically and hierarchically -- e.g. an address book sorted by name, with email and phone data attached. While LDIF records may hold large volumes of customer and transaction information, LDIF records are in a format that most applications cannot readily import or process. CoSort's SortCL tool can now process LDIF data, and simultaneously convert files in LDIF to other file formats and vice versa. As usual, your SortCL job script would reference or specify your field layouts, and the input and output file types. For example, on input, the declaration under input file could be /PROCESS=LDIF, and on output it could be /PROCESS=CSV.

"The Comcast Data Engineering and Management Integration (DEMI) organization works with 10 terabytes of LDAP data on a daily basis as we work to distribute business critical information resources to the rest of the company. The fact is we would not be able to pull this off successfully without CoSort. It accurately and quickly processes billions of rows of DAP data and allows us to join and analyze this information in connection with our other data warehouse processes. No other tool gives us this much speed and flexibility and allows the processing of the volume of flat-file LDIF records to be achieved. The very talented CoSort team worked directly with us to develop their module and was able to turn it around very quickly. In turn they have developed a long-term customer relationship with America's largest Cable Operator and a large (50 TB) and growing data warehouse."

See also: http://mcsv.net/cgi-bin/redir?MCid=2uTpOoOGoK[UNIQID]

Create Safe Test Data in Production Formats

CoSort 9 Can Also Synthesize Prototype Files in SortCL Jobs

In addition to simultaneous data processing (transformations), presentation (reporting), and protection (field-level encryption, etc.) functionality, CoSort V9's SortCL tool also supports data prototyping.

SortCL can now generate safe test data in real file and report formats. You can create output fields in any supported data type and layout using randomly-generated or randomly-selected values. Test data allows you to:

• Benchmark hardware and software
• Populate test databases
• Prototype and develop applications
• Stress-test programs with fuller data ranges
• Safely outsource real file formats

The CoSort Test Data Tool, RowGen, is also available for standalone generation, transformation, and presentation of safe test data in production file and report formats. RowGen is a lower cost product because it does not transform real data. But the same data definition and manipulation metadata work across both RowGen and SortCL, so you can move easily from prototyping to production later if you use only RowGen now.

See also: http://mcsv.net/cgi-bin/redir?MCid=mSwuPfwYoy[UNIQID]


New Dashboard Option Speeds Big Data Visualization

Marry Back-End Efficiency to Front-End Presentations

You may already use CoSort's SortCL tool to rapidly integrate, stage, and digest massive file volumes in ETL, federated data, or ODS operations that feed business intelligence tools and processes. Now you can also populate AND use a best-of-breed dashboard application to facilitate the development of business insight.

Specifically, SortCL can rapidly churn billions of records to produce output file subsets in XML or CSV formats that dashboards can import. The dashboard solution IRI now offers lets you cross-integrate that data with database, Excel, and other sources -- and then customize dynamic charts.

By combining SortCL with the CoSort Dashboard add-on option, you have a cost-effective, technically proven solution for saving time and money when transforming massive data volumes into drill-down dashboard charts.

See also: http://mcsv.net/cgi-bin/redir?MCid=Irj7t3F2dE[UNIQID]


CoSort Survey Thank You

Guess Who Won the Zune