Data Loader Overview

Some systems obtain their data from other systems in the form of a periodic extract file. The task of the data loader software is to read each data record in the extract file and to then test, validate and re-format the data in such a manner that it "fits" comfortably and accurately into the appropriate buckets of the target system's data store.

Data loaders can be programmed to run automatically, and can often obtain the data file over the Internet from the source, rather than having to handle physical media.

Typical project plan

The typical project consists of three major phases: design, development/ test, and installation. For those customers that have "easy" jobs to do, remember that each of these stages can take a few hours or a few weeks to do, depending upon the complexity and scale of the project.

Stage 1: Design

The design stage is devoted to gathering, organizing, and documenting the following information:

The source of the data. The file or host computer from which the data will be obtained, and the format of the data elements to be loaded into the target system must be identified. If the data is obtained across a network, all the necessary security and authentication mechanisms must also be identified.

The transformations, edits, and tests that must be done prior to target-system load. This includes formulas or algorithms for converting data from one form or value to another, edit checks to make sure data values fall within the allowable range as required for the new system, and any tests that are required to meet data integrity, uniqueness, and referential integrity constraints imposed by the target system

The destination of the data. What location within the target system's data store will the each element be stored into once the transformations are completed successfully.

The Cutover Test Plan. What final acceptance tests will the customer and Redbridge perform to assure us that the tranfer of the data is done accurately and completely?

Design Stage Collaboration

The customer supplies all available documentation about the source, transformations and destination of the data. Once this information has been obtained, Redbridge consolidates all this material into a mapping document that spells out where each data element is coming from, where it's going, and any applicable transformations, edits, and tests that must be performed on the data prior to being loaded into the target system. A system architecture diagram is also included for those datafeeds that operate across networks. The diagram identifies all security and authentication protocols which must be utilized in order for the data source to be accessed.

The customer then reviews the mapping document, signs off on it, and that kicks off the development and testing work.

Stage 2: Development and Test

Redbridge uses the mapping information from the Desigh phase as blueprints to write all the required programs to invoke the loader, access the data source, transform and edit the data, and load the data into the target data store. As the coding is completed for each section of the data to be transferred, it is unit tested. Once all unit development and testing is done, a full system-test is performed. This testing and development is typically done as an iterative, incremental process until finally the entire loader can run automatically and correctly. Then we're ready for installation.

Stage 3: Installation

Once the loader software is built and tested, it's time to install it into the target production environment. The work in this phase is mainly to coordinate the timing of the installation, and do a post-installation spot-check to make sure everything is working as expected.