GEWEX RFA Data Provider Instructions

Please read all instructions carefully before proceeding.

Table of Contents:

• New providers
• Selecting data
• Getting approval for data upload
• Packaging the data for RFA upload
• Checking the upload package
• Uploading and verifying data
• Instructions for resubmitting data
  • When upload fails
  • When you discover errors in previously submitted data
• md5sum for Windows users

New providers:

N1. Complete the data provider account application form.

N2. E-mail completed form to NASA Langley Atmospheric Science Data Center (ASDC) User and Data Services (UDS) (larc@eos.nasa.gov, 757-864-8656).

N3. If approved by the local RFA team, you will be contacted with the information needed to upload your data.

Selecting data:

The data you provide for this collaboration may be in any or all of the categories listed in the data file convention document. In particular, see the requested data types for satellite data and for ground-based measurements. These types are being given the highest priority in the initial assessment.

Getting approval for data upload:

A1. All of the data in a given upload must be from the same data product, as defined in point 1 of the file convention document. Specifically, the names of all files in the upload must begin with the same <ProductName>_<ProductVersion> tags.

A2. Complete and save the data description file, <ProductName>_<ProductVersion>.txt for this data product.
(NOTE: You will amend and resubmit this description file whenever new data from this product is uploaded. The new version will automatically be associated with all files for this data product.)

A3. Complete a data ingest form with information about this upload.

A4. Save an electronic copy of this form for use in step P6 below.

A5. E-mail the ingest form to UDS (larc@eos.nasa.gov).

A6. UDS will notify you by e-mail as to when you may begin your upload.

Packaging the data for RFA upload:

P1. Copy the data files to be uploaded to a single directory called RFAupload.

P2. Include at least one sample image file for each data type in the collection (i.e., ASWUP, CLWNET, etc). Each image file should have the same name as the corresponding data file, except that the file extension should be tif, tiff, jpg, png, or pdf, as appropriate.

P3. Copy the data description file from step A2 to the RFAupload directory.

P4. Verify that all file names and formats conform to the GEWEX RFA conventions.

P5. Select a tar file name, tarfilename, for your upload. This name must be unique and cannot be re-used. The ftp transfer will fail if a file with the same name already exists in the system, even if the previous file was rejected by the ingest system.

P6. Copy the saved data ingest form from step A3 above to tarfilename.txt in the RFAupload directory.

P7. Package the data. These instructions are for UNIX or Macintosh OS X computers. (Instructions for Windows users)

Steps M1 through M3 create a tar file and the corresponding MD5 file in the directory above RFAupload for your submittal. If you have trouble with this process, consult your system administrator or UDS for assistance.

Checking the upload package:

• tarfilename.tar: tar file containing the files in RFAupload:
  • data files (<ProductName>_<ProductVersion>*.asc)
  • data description file (<ProductName>_<ProductVersion>.txt)
  • ingest form (tarfilename.txt)
  • sample image(s) (same names as data files, with appropriate extension)
  • MD5 checksum of each file in the tar file (checksums.md5)
• tarfilename.tar.md5: MD5 checksum of the tar file

Uploading and verifying data:

U1. Follow the data upload instructions received from UDS.

U2. Note that the tar file (tarfilename.tar) MUST be uploaded before the tar MD5 file (tarfilename.tar.md5).

U3. You will receive an automated confirmation (or error report) indicating that the integrity of the transferred data has been verified.

U4. Once your data has been posted to the GEWEX-RFA website, you will receive a second confirmation via e-mail. You must reply to this message to verify that your data are properly displayed on the website or report any errors.
NOTE: Failure to reply within two weeks will result in this data being removed from the public site.

Instructions for resubmitting data:

When upload fails:

• Correct any indicated errors (e.g., file naming convention, file format).
• Repackage the corrected files following the procedure outlined in steps P1 through P7 above.
• Remember that a new name is required for the tar file MD5 file, the tar file itself, and the ingest form included in the tar file.
• Upload the data following steps U1 to U4 above.

When you discover errors in previously submitted data:

• Correct the errors you have discovered in your data files.
• Use the same file names as before but increase the RFA number by one (e.g., RFA01 becomes RFA02).
• Make a note in section 5 of the data description form explaining what corrections you made and what files are being replaced.
• Repackage the corrected files following the procedure in steps P1 through P7 above.
• Remember that a new name is required for the tar file MD5 file, the tar file itself, and the ingest form included in the tar file.
• Upload the data following steps U1 to U4 above.

NOTE: ALL files in the archive having the same <ProductName>, <ProductVersion>, and <DataDescriptors> (see the file convention document) as the resubmitted files will be assumed to be in error and will be deleted from the system. Thus even if only one file needs to be replaced, all files containing the same type of data (e.g., time series of ASWDN) should be resubmitted as a group, with their RFA number incremented.

md5sum for Windows users:

Two options are available for Windows users.

1. Install the UNIX emulator cygwin on your machine and follow the instructions listed above for UNIX users. (cygwin is available at http://www.cygwin.com/.)

2. Install the DOS command md5sum.exe and a tar utility on your machine and follow the instructions provided below. Windows md5sum.exe and download instructions are available from http://www.etree.org/md5com.html. A utility that performs tar functions using a graphical user interface, 7-zip.exe, is available at http://www.7-zip.org. After installing these applications, perform the following in place of steps M1 to M3 above:

   W1. In a DOS window, cd to the directory above "RFAupload" and run the command:

   md5sum RFAupload\?*.* > RFAupload\checksums.md5

   This will create a file called "checksums.md5" in the RFAupload directory.

   W2. Start 7-zip and use it to tar all of the files (data, data description, image, checksums.md5, and tarfilename.txt files) in RFAupload into one tar file.

   To do this,

   • go to the directory above "RFAupload,"
   • select "RFAupload"
   • select "Add"
   • enter the desired tar file name
   • check that "archive format" is set to "tar,"
   • select "OK"

   W3. Run md5sum on the finished tar file in the directory above RFAupload using the command

   md5sum tarfilename.tar > tarfilename.tar.md5

Steps W1 through W3 create a tar file and the corresponding MD5 file in the directory above RFAupload for your submittal.