Success Story
CoSORT Speeds Valdero Supply Chain Integration
(published July 2003, DM Review)
Reviewer
Krishanu Lahiri, integration manager for Valdero Corporation.
Background
Valdero Corporation is the leading provider of enterprise-class
software applications for real-time supply and demand chain execution.
Valdero creates high-velocity supply and demand chains through real-time
visibility, collaboration and execution. Applicable across manufacturing
companies in the high technology, medical device and consumer markets,
Valdero solutions are currently in use at industry-leading companies.
Valdero's management team has deep supply chain experience with
premier technology companies and a demonstrated track record in
building and managing customer-focused organizations. Valdero's
real-time supply chain execution applications give manufacturers
visibility across multitier supply and demand chains, enabling them
to rapidly respond to fluctuations in demand and supply as they
occur.
Platforms:
Solaris, Linux, Windows NT/2000.
Problem Solved
A major strength of the Valdero solution lies in its ability to
process large amounts of supply chain data in near-real time. This
data is sourced from both internal systems as well as from the various
nodes of the supply and demand chain. Typically, within the same
implementation, data is received over different transport mechanisms
varying from bulk file transfers to message queues. Integrating
these varied sets of data involves identifying changes, performing
data quality checks and transforming incoming data to match application
standards before loading the active repository. One of the main
constraints to this entire process is that it must occur with minimal
impact to online activity.
We use CoSORT's sort control language (SortCL) program to sort,
transform and join files outside the database to accomplish large
parts of our integration processing. We use our proprietary Application
Integration Manager (AIM) to load data into our Active Repository.
We also use CoSORT extensively within AIM to identify and filter
the data that has changed in the current ETL batch (internally known
as the Netchange) process.
Because the Valdero application sources data in different formats
from multiple locations, it is very critical to our daily batch
loads to narrow the records to only those that have changed since
the last successful load. CoSORT utilities improve the performance
of the Netchange process and provide an easily configurable framework
to perform the Netchange on different files.
Product Functionality
Among the CoSORT package's many user interfaces and API calls is
a 4GL for data definition and manipulation (integration, staging
and reporting) called SortCL. By wrapping Python script around CoSORT's
SortCL program, we can maintain multiplatform support for AIM. This
has been a very important factor in selling the Valdero AIM process
to our customer IT departments, as they can maintain the same code
base and performance levels across multiple platforms.
In the future, we plan to use CoSORT's native support of different
data types, as well as SortCL's filtering capabilities to enhance
the data transformations supported by the Valdero AIM process. We
can also count on CoSORT's superior sort-join performance for surrogate
key lookups within AIM.
Strengths
The primary strength of the CoSORT SortCL utility that we leverage
in the Valdero AIM process is the enhanced performance on multi-CPU
machines. The ability to process hundreds of thousands of input
rows and quickly filter them based on various criteria helps in
significantly improving the overall time it takes to refresh the
Valdero repository with minimum impact to the application users
that are online.
Weaknesses
SortCL does not allow more than one character per field to be used
as an internal separator. This has caused us to "escape"
special characters before processing them with CoSORT.
Selection Criteria
We required a product that has stood the test of time, is available
on all of our target platforms and has a price point that would
enable us to bundle it with our solution. After researching the
open systems sorting and ETL tool markets, CoSORT was the best choice.
Deliverables
We use CoSORT primarily as a data processing tool with minimal reporting
usage. CoSORT's SortCL program is invoked from within our application
on dynamically generated transform information.
Vendor Support
Vendor support has been excellent. We have had quick responses from
CoSORT's support team via e-mail and phone. The best part is that
our contacts are always abreast of the issues.
Documentation
While the documentation is excellent, the sample programs and examples
that are bundled with the product are an invaluable pointer to develop
programs using CoSORT. The product is very easy to install, and
the documents guide us through all the different functionality provided
by CoSORT.
|
