By Pat McCaslin, Mark Hei, and John Garrett
This article describes the logical functioning of the DIOnAS software developed by NSSDC's primary data management software development team plus the functioning of the associated modules developed by the NSSDC/NOST team. An architectural diagram is available.
This software creates and ingests Archive Information Packages (AIPs), writes these to a Digital Linear Tape (DLT) jukebox for long term archiving and writes the AIPs' constituent files (data file and attribute file) to magnetic disk for external access via ftp or http, and builds an Oracle data base of information about the locations and other attributes of the AIPs and their files. The data base (1) is used in setting up and controlling data ingest/migration jobs, (2) provides long term NSSDC data management capabilities and (3) offers NSSDC the option to eventually build user interfaces to data at higher levels of functionality than ftp and http.
The DIOnAS software was developed by the NSSDC data system development team. The major DIOnAS software elements are the ingest manager server and the ingest manager client. The server uses and populates the data base, invokes the NOST software (see below) and writes data to DLT and to magnetic disk. The client provides an operator GUI (graphical user interface) and performs some front end validation of operator-provided processing instructions.
The NOST team's Data Migrator Utility (DMU) software, called by DIOnAS, reads VMS files, converts them as needed into platform-independent canonical formats, creates the companion attribute files including information needed to support backward-migration of data to VMS, bundles the data and attribute file pairs into Archive Information Packages (AIP) and delivers these to DIOnAS.
The Package Splitter Utility (PSU) splits AIPs into canonical format data files and their accompanying attribute files.
DIOnAS pipeline processing operations are organized into discrete "jobs" defined by externally prepared "listfiles". Listfiles contain the information necessary for DIOnAS processing: names of the source data files to be processed; the destination AIPs, destination DLT, and destination canonical data files; processing flags; and other metadata not obtainable from the source file itself. All files in a given job are of the same data type and the resulting AIPs are bundled into one tar file by DIOnAS for writing to DLT.
For writing data AIP's to magnetic disk for customer access, DIOnAS calls NOST's Package Splitter Utility (PSU) software which splits the sensor data file and the attribute file out of the AIPs and delivers these back to DIOnAS for writing into the relevant directory on nssdcftp.gsfc.nasa.gov.
The following describes, at a high level, the DIOnAS processing pipeline for a single job:
1. The DIOnAs operator using the GUI selects a prepared listfile for processing;
2. The selected listfile is checked for format and syntax;
3. The validated listfile information is inserted into the data base - at this point a job has been created;
4. The necessary processing information (source, destination, metadata, etc.) for all entries in the job are obtained from the data base and passed to the DMU software;
5. The DMU software reads the files on the remote VMS machine and creates AIPs on the DIOnAS host machine;
6. Entries destined for DLT are identified and the AIPs (tarred into whole- job files) are written to DLT;
7. Entries destined to be placed in the ftp area are identified and the necessary information (source, destination, etc.) for those entries is passed to the PSU software;
8. The PSU software operates on the AIPs to split out the original data file in its canonical format and the attribute file and writes these to magnetic disk.
The existing DIOnAS/DMU subsystem is capable of accepting data submitted in
the form of individual files. A version of DIOnAS with the ability to
automatically accept and process data already packaged as AIPs is under
development with deployment anticipated early in 2001. Two new NOST
software utilities support this new capability. The Package Generator
Utility (PGU), an extension of the DMU software, provides
external data
suppliers (initially, IMAGE) the ability to create AIPs. The Extractor
Utility provides DIOnAS with the ability to extract from AIPs the
information needed by the DIOnAS data base.
NASA home page
GSFC home page
GSFC organizational page