RXTE GOF	RXTE Archive The RXTE Team is committed to making archival data available promptly and conveniently.	RXTE FAQ

When do data enter the archive?
When will the GO data I'm interested in go public?
How are the data organized in the Archive?
How do I find and retrieve the data I want?
How do I proceed with data reduction and analysis?
Cautions and caveats

When do data enter the archive?

RXTE data fall into four categories, with different policies for public release:

NRA data: The standard proprietary period for RXTE data obtained in response to accepted GO proposals is twelve months from the date of receipt by the PI. However, due to reprocessing activities, the proprietary period for AO-1 data taken before 1996 December 16 was extended until six months after receipt of the reprocessed data.

Non-NRA Targets of Opportunity: The data from TOO observations that were not anticipated in accepted proposals are publicly available as soon as they are processed by the RXTE Science Data Center.

ASM: All ASM data are made publicly available as soon as they are processed.
In-Orbit Checkout: These data were taken during the first month of the mission (1996 January) to calibrate and test the RXTE instruments. These data were all non-proprietary from the beginning, and are already present in the Archive.

The RXTE data archive is maintained by the High Energy Astrophysics Science Archive Research Center, HEASARC. and can be accessed using their online service.

When will the GO data I'm interested in go public?

The RXTE GOF has developed the RXTE Public Data Web tool to allow RXTE users to find out when proprietary data will go public. Users can search on target name, PI, proposal number, complete ObsID, and/or date. For example, to find out what data went public in March 2000, one would enter in the appropriate box "Mar-*-2000".

The user may also search the XTEPUBLIC catalogue using the HEASARC's Browse to find out which datasets are currently available.

By the beginning of 2000, more than 400 GBytes of RXTE GO data had been made public.

How are the data organized in the Archive?

Archival RXTE data occupy the xte/data/archive directory on the anonymous FTP server at legacy.gsfc.nasa.gov. Directories at the next lowest level are named AO0 for IOC data, AO1 for AO-1 data, etc. Note that you will see directories for data that are not yet public. The links are there, but the data files themselves will not be accessible to you until the proprietary period expires.

Below this level, RXTE data files are arranged in a hierarchical set of directories. First come directories named after the proposal number; the next level contains the individual observations ("ObsID's"), while the final level contains directories for each of the Subsystems (PCA, HEXTE, ACS etc.). In the catalogue of publicly available RXTE data, individual entries correspond to the ObsID level. GO data released through our Online Data Area are also grouped by ObsID.

Observation IDs

Since the ObsID is the link from the browsable catalogue to the data themselves, it's worth explaining its nomenclature. Each ObsID corresponds to a single observation, where "observation" refers to a temporally contiguous collection of data from a single pointing. The format for the ObsIDs is as follows:

    NNNNN-TT-VV-SSX

where:

NNNNN is the five-digit proposal number assigned by the GOF and is identical to the name of the parent directory.
TT is the two-digit target number assigned by the GOF. Note that for the case of only one target, the target number may be zero.
VV is the two-digit viewing number, assigned by GOF, which tracks the number of scheduled looks at the target. In particular the viewing number corresponds to:
- different (requested) observations of the same target (e.g., at different epochs for monitoring),
- different instrument configurations during the same pointing,
- different scans for scan-mapping of extended sources.
SS is the two-digit sequence number used for identifying different pointings that make up the same viewing if the SOF decided, for operational reasons, to split that viewing into more than one chunk or if it was split "by accident", by a TOO, for instance.

X, 15th character, when present, indicates:

    A       Slew before observation
    Z       Slew after observation
    S       Raster scan observation
    R       Raster grid observation
C, D, G, H, I, T, F, U ObsIDs that may contain data gaps
    0-9     Segments of a long observation (i.e., > 8 hours)
    b-r     (reserved for) Real-time configuration changes

When not present, it indicates a regular pointed observation (< 8 hours) or the last segment of a long observation.

Spacecraft Subsystems

Each ObsID directory contains a set of 15 subdirectories, each containing data files derived from a single spacecraft Subsystem. The typical observer will not need to know about most of these, but a complete listing is as follows:

ace - Attitude Control Electronics & star trackers
acs - Attitude Control System
cal - References to files in the Calibration Database
clock - Time delta correction data from Mission Operations Center
eds - Experiment Data System housekeeping
fds - Flight Data System
gsace - Gimbals and Solar Array Control Electronics
hexte - HEXTE science and housekeeping data
ifog - Interferometric Fibre Optics Gyroscope
ipsdu - Instrument Power Switching and Distribution Unit
orbit - Orbit ephemeris from the Flight Dynamics Facility
pca - PCA science and housekeeping data
pse - Power System Electronics
spsdu - Spacecraft Power Switching and Distribution Unit
stdprod - Standard Products - cleaned light curves, spectra etc.

Though they represent the lowest rung in the directory hierarchy, the Subsystem directories do not necessarily contain files of one type. In most cases, a further division is made based on Application, the term used for a distinct source of telemetry. In the case of the PCA, the six Event Analyzers are considered applications. Note that the ASM is not included.

Also in the ObsID directory are various index files used by XDF - the XTE Data Finder - to identify the underlying data files.

Example of Directory Structure

An example will clarify how XTEPUBLIC relates to the ObsID and manifold directory structure. Searching XTEPUBLIC for the IOC observation of 4U1907+09 yields the ObsIDs:

    10154-02-01-00
    10154-02-03-00
    10154-02-04-00
    10154-02-05-00
    10154-02-06-00
    ...

The directory containing PCA data from the first ObsID is:

    xte/data/archive/AO1/P10154/10154-02-01-00/pca

Please note that although you can work your way down the directory structure using only FTP, it is often more convenient to use Browse to find and retrieve the data, or retrieve a whole ObsID using anonymous FTP.

How do I find and retrieve the data I want?

The XTEPUBLIC catalogue in the HEASARC database lists all publicly available RXTE data. (Also, remember the RXTE Public Data tool mentioned above.) There two methods for retrieving public data: the web-based Browse, and anonymous FTP.

Using Browse

Browse is a web-based browser with a graphical user interface. Link to the main Browse page, and choose the "Advanced" version (to allow greater search flexibility). In the "Advanced" interface, check the RXTE mission box, choose to search either by object name/coordinates or by parameters, and then begin your search:

Choose the XTEPUBLIC catalogue.
Enter search criteria into appropriate boxes. If you chose to search by name, you might enter e.g. 4U1907+09 into the Object Name or coordinates box, and click Submit. To search by parameters, you'll click again to begin your search, and perhaps choose data by proposal number (called PRNB within Browse), ObsID etc. Then click Submit.
A successful search will display the corresponding catalogue entries, i.e. the ObsIDs, one per line. Check the box(es) to the left of the ObsID(s) you're interested in.
Below this listing, you'll find a choice of categories of data:
```
GIF Standard Products Only of a Single RXTE Observation (GIFS)
Standard Products (including GIFs) of a Single RXTE Observation (STDPRODS)
Complete Data Directory of a Single RXTE Observation (FULL OBSID)
```
(Note that the STDPRODS includes the GIFS in the category above, and the FULL OBSID includes the STDPRODS and GIFS.) Choose one by clicking on it to highlight it.
Then click on the radio button to Retrieve data products in selected categories (or List products in the selected categories and then retrieve the products as a subsequent command).
Click on Submit. Browse will now construct the appropriate tar command, and tar up the data you have requested. This may take a while.
Browse will bring up a new page when your tar job is complete. You may now click on Download TAR file to initiate the transfer of the tar file to your host machine. Make sure you have enough disk space! - the total size of the tar file will be indicated.
When untarred, your data files and index files will appear in the usual configuration for RXTE datasets, as described above

Anonymous FTP

Anonymous FTP can be a convenient alternative to Browse, if you don't wish to tie up your Web browser for the duration of the data retrieval. Here's an example of the steps you might follow:

ftp legacy.gsfc.nasa.gov, giving 'anonymous' as your name, and your Email address as the password.
cd xte/data/archive (stopping to read the information presented here)
cd AO1
get P10154.tar to obtain all the public data from this proposal. The FTP server will automatically create the tarfile on your home machine in the directory from which you initiated the FTP run - make sure you have sufficient space. You might want to cd P10154 and obtain a subset of the data (e.g. get 10154-02-04-00.tar).

There is no advantage to pulling over a gzipped file using e.g. get 10154-02-04-00.tar.gz. All the datafiles within the directory structure are already gzipped.

If you try to FTP data which is still proprietary, you'll end up with a tarfile containing clock and orbit information, calibration files and index files, but no science data.

Cautions and Caveats

RXTE data will be released into the archive on a weekly basis. Observations newly turning public before 12:01a.m. on a given Saturday will be available (i) the following Monday, for anonymous FTP access; (ii) Tuesday, for Browse access.
RXTE datasets may be larger than you are accustomed to, and response may be slow at peak times, particularly if multiple users are attempting to get their hands on the same dataset. Please ensure you have sufficient disk space for the transfers, and try to plan your data retrieval to avoid times of heavy network traffic. If you have problems, try again later.
RXTE data will go public on an ObsID basis, based on the date when the tape containing that ObsID was mailed to the original PI. Thus, not all the parts of a given proposal will necessarily go public at the same time, particularly in the case of large proposals or long-term monitoring campaigns.

How do I proceed with data reduction and analysis?

Install the FTOOLS package
Read the materials on the Data Analysis & Processing page
Consult RXTE Frequently Asked Questions.

If you have a question about RXTE, please send email to one of our help desks.

This page is maintained by the RXTE GOF and was last modified on Monday, 07-Apr-2003 14:53:31 EDT.