Success Story
Automotive Data Warehouse - LG01, Ltd.
(published January 2001, DM Review)
Background
LG01 Ltd. is a UK Consultancy specializing in Data-Warehousing solutions.
Although relatively new to the market, we are experienced developers
with a deep understanding of the technical issues associated with
data movement and management in large enterprises. We are based
in York, Northern England.
Hardware
CoSORT running on Sun E3000 server (Solaris 7), 4 CPUs and ¾ Gb
of RAM. A mixture of local and network (4x100Mb/s) drives were used.
Problem
Solved
During our implementation of a large-scale data warehouse for a
major automotive company, we ran into a number of problems. The
client had originally stipulated that we query directly against
their operation database from the ETL tool, including complex joins,
sorts, and filters. We were then to 'twist' the data using the ETL
tool to provide both the warehouse and a number of 'report ready'
de-normalized tables suitable for paper and web reporting.
The first problem
was one of speed. The extraction from source to ETL tool was taking
far too long and placing unacceptable strain on the operational
servers. The second issue was one of complexity - in that the ETL
tool, although well suited to simple transforms, soon required hand
scripting to do anything even modestly complex - and we would have
been forced to hard-code much of the logic due to the tools poor
parameterization capabilities.
By adopting
a radically streamlined method using CoSORT to process flat-file
dumps of the operational data (very quick to extract) we could drastically
cut our run-times. We could do all the joins and filters we needed
on a separate server, and due to CoSORT's ability to run in parallel
across all CPUs using all the RAM we got an excellent return on
our clients hardware investment.
Product
Functionality
As an additional bonus, we found that we could pipe CoSORT's output
straight through some simple Perl scripts to do our transformations
- a great solution as Perl gave us genericism across operating systems
and the ability to parameterize the business logic and avoid hard-coding.
The ETL tool simply became a method of defining the process flow,
with all the work being done under the hood by CoSORT.
LG01 Ltd. now
recommends CoSORT to all it's clients as a means of achieving high
speed, high volume Warehousing; it has become a highly prized and
reliable part of our toolset.
Strengths
The ability to perform flat-file 'SQL-like' processing, particularly
Joins is very useful indeed and the speed with which it does it
makes it an essential 'must have' tool in our opinion. The script
language is rich, and appears to carry no cross OS differences.
Weaknesses
CoSORT as used had no GUI to speak of - but I understand a Java
one is available or imminent. Joins across more than 2 tables requires
some extra work within the scripts, which whilst not hard makes
the process more fussy than it need be.
Selection
Criteria
Having evaluated a number of similar products none seem to carry
the set of functionality that CoSORT does. In the Data Warehousing
arena - where the big ETL vendors compete, a tool that encompasses
70% of their functionality for only 20% of the cost makes the decision
an easy one to make.
Deliverables
We delivered an architecture that the client was delighted with,
under budget and that ran well within the timeframes required.
Vendor
Support
IRI have proven to be very quick and reactive, and seem genuinely
eager to assist in any way they can. During evaluation they offered
excellent support and advice. A UK support base - if only to avoid
the time difference issues would be useful, but CoSORT is a straightforward
tool to use as it is.
Documentation
The PDF documentation we received was very good.
|
