The ABC of XTE
Extracting Light Curves and Spectra
Extracting Light Curves and Spectra
N.B. This chapter is under revision
Postscript version of this chapter
Extracting clean, useable light curves and spectra is one of the main objectives of RXTE data reduction. In broad terms, it involves choosing:
Practically, extraction involves running one of two ftools, depending on the kind of data: saextrct for science array data and seextrct for science event data. How to run these ftools, known as the extractors, is described in detail below: each question asked by the two ftools is explained and examples are given.
One supplementary aspect of extraction deserves its own description and is covered in the the section that follows, namely, the various ways in which time selections can be made.
Finally, before extracting data from a given configuration, it's a good idea to review just what options the data format actually allows. PCA Standard 1 data, for example, come in only one spectral bin, so extracting a spectrum would only yield the mean count rate. The reduction options are described in the PCA Issues chapter, in the section on EDS configurations.
In practical terms, the final step of selecting data based on time entails entering a set or sets of start and stop times into the extractors. This simple matter is covered below in the two sections about saextrct and seextrct. Here, however, we describe the first step of creating these start and stop times.
Generally, creating start and stop times involves using some "other" information. How you do this falls into three categories:
This subsection presumes, of course, that you've already extracted a light curve. It appears here, rather than after the sections on running the extractors, for the sake of completeness.
Light curves created by the extractors are in FITS and usually have columns called TIME, RATE, ERROR and FRACEXP. This can be verified using the ftool flcol which prints to the screen the column names in a FITS file.
To plot a light curve, simply run the ftool fplot. The program prompts the user for the column containing the x-axis and then for the columns to be used on the y-axes. It calls the same PGPLOT subroutines as Xspec and Xronos. Here's an example:
The times displayed on the plot are relative to the value of the TSTART keyword in the header of the light curve. Since the extractors work in SCCS (spacecraft clock seconds), the start and stop times from the light curve have to be converted. The Perl script timetrans will do this. Simply type "timetrans" and then enter the ranges from the plot. The resulting output file containing the converted times can be entered into the extractors (with an @ sign) at the prompt Input time intervals t1-t2, t3-t4 in seconds.
Creating GTI files based on filter files is covered in the Screening chapter. Here's a briefer, alternative example: a GTI file generated from a light curve containing the start and stop times corresponding to when the count rate in the light curve is between 10,000 and 20,000 counts per second. As in the case of filter files, the ftool maketime is used to generate the GTI file:
The times in the GTI file, which you can examine with the ftool fdump, are relative to the value of TSTART in the header. The GTI file itself is applied by the extractors in response to the question Input GTI file to be AND'd with INFILE.
The best way to explain how to use saextrct is by
example. In what follows, a typical saextrct run is shown. Inputs are explained under the links which take you to a series of notes at the end of the example.
After the example and explanatory notes comes a list of various "reduction scenarios" motivated by scientific priorities and accompanied by the corresponding saextrct inputs - scenarios such as "creating a light curve with 0.125-second bins from Binned Burst Catcher data in the 2-6 keV band."
Here, we'll use saextrct to create a light curve with 32-second bins from PCA Standard-2 data. We have already run: a) XDF to create a list (sdt2.list) of Standard-2 data files, and b) maketime to create a GTI file excluding elevations greater than 10 degrees (elv_gt_10.gti). No other selections are applied. In addition:
The corresponding screen dialogue is as follows:
We recommend, however, if you want to combine more than one FITS GTI file, that you should use the ftool mgtime to merge the individual FITS GTI files into a single file.
Although the prompt refers to spectra, this option can be used to restrict the pass band for a light curve.
Although the prompt refers to spectra, this option can be used to restrict the pass band for a light curve.
The following "real-life" scenarios are intended to show the range of saextrct's data reduction capabilities. For the sake of brevity, only the inputs that differ from those in the above example are described.
Light curve: PCA Standard-2, 64-s bins, 2-6 keV
Creating light curves for specific energy bands is a common task in
X-ray astronomy. The key first step is to determine what channels correspond to the desired energy band. For the PCA, this is best done via an extracted spectrum, as the instrument has undergone several
high-voltage gain changes since launch, i.e. the energy-to-channel conversion depends on epoch:
Generate the corresponding response matrix with pcarsp.
Read the spectral file into xspec with the "data" command (pcarsp automatically writes the name of the response file into the header of the spectral file).
Issue the xspec command "ignore 0.0-2.0 6.0-**" which causes the channels corresponding to 0.0-2.0 keV and above 6.0 keV to be excluded from fitting (the ignore command interprets integers as channels, reals as energies in keV).
Issue the xspec command "show" and look in the screen output for a line like:
Remembering that saextrct requires original channels, we have to convert this channel range from the 129-channel Standard-2 binning to the full 256-channel binning. The script chantrans will do this. In this case, 6-19 corresponds to 10-23.
Finally, we can now run saextrct to extract the light curve. The channel range, 10-23, corresponding to 2-6 keV should be entered at the prompt:
Spectrum: pcabackest file, top layer only
The ftool pcabackest models the instrumental background of the PCA and produces output files that look in many respects like a Standard-2 data file. With these files as input, saextrct can be used to extract background light curves or, as in this example, a background spectrum from the top layer of all five PCU:
Run the ftool flcol on the file generated by pcabackest, (pca_back.fits, say) to list the columns.
From this list, create the file called, say, top_layer.cols, containing the lines:
Run saextrct, using pca_back.fits as the input file and entering top_layer.cols at the prompt:
Spectra, source and background: HEXTE Archive, all detectors
Before running HEXTE science array files through saextrct, the data have to be first filtered by cluster position, i.e. separated into on-source and off-source parts. As explained in the footnote, this is done using the ftool fselect. Then, assuming that you've produced two sets of files - one on-source, one off-source - proceed as follows:
Run saextrct on the on-source files, calling the output spectrum, e.g., herx1_on.pha.
Run saextrct on the off-source files, calling the output spectrum, e.g., herx1_off.pha.
Adjust the exposure of the off-source spectrum. This is necessary because the time spent by the cluster moving between on-source and off-source positions (2 seconds) is not accounted for in the data files, i.e. the structure of the files implies that on-source and
off-source rows each contain 16 seconds of data. To make the adjustment:
As described in the Data Files chapter, science event files contain irregularly spaced
time-stamped event words which, in the case of the PCA, are interspersed at regular time intervals with various EDS flags (also known as clock events). This means that extracting light curves and spectra from an event file involves removing the clock events, and, if desired, picking a subset of the valid events, e.g. data from PCU0 only. In practice, this entails defining a bitmask: a binary string (or set of strings) which the event words in the file must match in order pass through extraction.
Seextrct applies but does not define bitmask selections, so another program, sefilter, must be run before the extraction. This caveat applies to:
This section is split into two. The first, immediately following, covers running sefilter. The second covers running seextrct. As for the corresponding section on saextrct, a fully worked example is given along with some data reduction scenarios.
Before running sefilter, it's a very good idea to review the sections in the PCA Issues chapter that describe the various event-mode configurations, namely, Good Xenon and Generic event. Revise, too, the footnote on DDL. And it almost goes without saying that you should type fhelp sefilter.
Sefilter is a Perl script that helps you create an ASCII bitmask filter. The script then uses the bitmask to run either of the following:
Bitmasks contain a prefix (b for binary, h for hex, o for octal - all RXTE bitmasks are binary) followed by a string of zeroes, ones or x's. The x is a wild card: for binary bitmasks it means "0 or 1". When you run sefilter, not selecting a particular token is equivalent to leaving all its values as x's. Practically, this means that if, say, you want events from all five PCU, you should not make a selection based on the D-token which would otherwise enable you to single out a particular PCU.
By default, and if no other selections are specified, sefilter creates a bitmask for PCA data which will select only good science events, i.e. those with the M[1]{1} marker or M-token. It is an ASCII file containing a line like
Although, for some configurations, channel information is contained in the bitmask, selection by channel is not performed via sefilter. Seextrct is used instead.
Example of using sefilter
Here, we'll use sefilter to generate a bitmask corresponding to the top layers of all the PCU. The input file, E_1ms_128M_0_8s.list is a list of data files in the E_1ms_128M_0_8s configuration, hence the @ sign. In this example, user input is in boldface.
Here, we'll use seextrct to create a light curve with 10-ms bins from data in the PCA configuration E_1ms_128M_0_8s. This is a continuation of the above example where we used sefilter to create a bitmask corresponding to the top layer of the PCA which was then used by fselect to create the file top.fits. Apart from excluding events from the middle and bottom layers, top.fits also excludes clock events. Please assume we have already run: a) maketime to create a GTI file excluding elevations greater than 10 degrees (elv_gt_10.gti) and b) timetrans to create an ASCII time-interval file from a Standard-2 lighcurve (flare.timeint). No other selections are applied. In addition:
The corresponding screen dialogue is as follows:
Introduction
Time Selections
Creating Start and Stop Times with xTime
Creating Start and Stop Times from Light Curves
rufus [121] [day] x1728: fplot data4_soft.lc
Name of X Axis Parameter[error][] TIME
Name of Y Axis Parameter[error] up to 8 allowed[] RATE[ERROR]
Lists of rows[-]
Device: /XWindow, /XTerm, /TK, /PS, etc[/XW]
Any legal PLT command[]
At the PLT> prompt, you can enter various commands to change the appearance of the plot. After examining the plot, make a note of the start and stop times you're interested in and quit the program.
Creating GTI
rufus [123] [day] x1728: maketime
Name of FITS file and [ext#][] data7.lc
Name of output FITS file[] medium.gti
Selection Expression[] rate.gt.10000.and.rate.lt.20000
Column containing HK parameter names[NAME]
Column containing HK parameter values[VALUE]
Column containing HK parameter times[TIME] Time
Flag, yes if HK format is compact[yes] no
Extracting Science Array Data
Example: Creating a Light Curve from PCA Standard-2 Data
rufus [58] [day] redux: saextrct
Running SAEXTRCT version 3.5.2
==============================================
Input file name or @file-of-filenames:[] @std2.list
Input GTI files to be OR'd with INFILE (APPLY):[APPLY]
Input GTI file to be AND'd with INFILE:[-] elv_gt_10.gti
Root name for output file:[sac] pca_32s
Accumulate (ONE) or (MANY) Spectral/Light Curves:[ONE]
Name of TIME column:[TIME]
Name of COLUMNS to be accumulated (GOOD):[GOOD]
Input the binsize in seconds, use 0.1 etc. if nec (INDEF):[INDEF] 32
Minimum acceptable fractional exposure (INDEF):[INDEF]
Chose print option, LIGHTCURVE, SPECTRUM, or BOTH:[BOTH] lightcurve
Type of binning for LIGHTCURVE: (SUM, RATE, MEAN):[RATE]
Type of binning for SPECTRUM (SUM, RATE, MEAN):[SUM]
Maximum acceptable intensity for Light Curve (INDEF):[INDEF]
Maximum acceptable intensity for Spectrum (INDEF):[INDEF]
Column titles for SUM written as header Keywords:[-]
Column titles for MEAN written as header Keywords:[-]
Starting time for summation (INDEF):[INDEF]
Ending time for summation (INDEF):[INDEF]
Input time intervals t1-t2,t3-t4 in seconds (INDEF):[INDEF]
Minimum energy bin to include in Spectra (INDEF):[INDEF]
Maximum energy bin to include in Spectra (INDEF):[INDEF]
Input channels to be retained 1-2,3-4 (INDEF):[INDEF]
Input channels for each bin 1-5,6-256 (INDEF):[INDEF]
Perform a dryrun on files without processing any data? (Yes):[no]
Explanation of the Various Inputs in the Example above
/data/kaah5/xtegof/day/herx1/P00023/00023-02-01-00/pca/FS4a_3eb7080-3eb7fd0
/data/kaah5/xtegof/day/herx1/P00023/00023-02-01-00/pca/FS4a_3eb8860-3eb922d
At present, up to 100 files can be listed in this way. Please note that the list should end with a carriage return.
X1LSpecPcu0
X1RSpecPcu0
X1LSpecPcu1
X1RSpecPcu1
X1LSpecPcu2
X1RSpecPcu2
To see the column names in any FITS file, run the ftool flcol. Please note that the list should end with a carriage return.
63460012.89412300-63460012.89412400,63460012.89412412-63460012.89412512
63460012.89412300 63460012.89412400
63460012.89412412 63460012.89412512
where the last character is a carriage return.
5 25
45 70
76 180
where the last character is a carriage return. The Perl script chantrans can be used to generate such a list.
In addition, please note
5 25
45 70
76 180
where the last character is a carriage return. Or it can be in the form of the CPIX keyword (without the CONTINUE keywords, e.g., for the SpecDet0 (the ninth) column in HEXTE Archive data files:
1CPIX9 = '0~63;2,64~127;4,128~255;8'
Data Reduction Scenarios: Using Saextrct
First, use saextrct to extract a spectrum from your
Standard-2 data files (just one file will do). Defaults can be entered for all inputs, if you like, or you can use a "good" spectrum, i.e. one that has been extracted with all your desired GTI and other selections.
Noticed channels 6 to 19
which explains that for this particular Standard-2 spectral file, 2-6 keV corresponds to channels 6-19. Quit xspec.
Input channels to be retained 1-2,3-4 (INDEF):[INDEF] 10-23
First run pcabackest to generate an model background datafile, choosing the option of separating the detector layers.
X1LSpecPcu0
X1RSpecPcu0
X1LSpecPcu1
X1RSpecPcu1
X1LSpecPcu2
X1RSpecPcu2
X1LSpecPcu3
X1RSpecPcu3
X1LSpecPcu4
X1RSpecPcu4
Make sure the file ends with a carriage return.
Name of COLUMNS to be accumulated (GOOD):[GOOD] @top_layer.cols
Verify the names of the columns containing the spectral data by running the ftool flcol on one of the Archive mode data files. The columns, one per detector, are:
SpecDet0
SpecDet1
SpecDet2
SpecDet3
which should be entered at the prompt:
Name of COLUMNS to be accumulated (GOOD):[GOOD] SpecDet0 SpecDet1
SpecDet2 SpecDet3
rufus [78] [day] redux: fdump prhead=no
/data/herx1/P00023/00023-02-01-00/hexte/FS52_3eb6dcb-3eb7570
Name of optional output file[STDOUT]
Names of columns[] ClstrPosition
Lists of rows[-]
ClstrPosition
Count
1 5
2 0
3 1
4 0
5 5
6 0
. .
. .
. .
Here, we see that the cluster position changes from 0 (on-source for 1.5-degree rocking) to 1 or 5 (+1.5-degree & -1.5-degree offsets, respectively) every row, i.e. every 16 seconds. This means that each off-source row really contains 16 - 2 x 2 = 12 seconds of data, and, consequently, the total exposure for a spectrum extracted from
off-source data in this case is overestimated by a factor of 16/12 = 1.33.
rufus [80] [day] redux: fkeyprint herx1_off.pha EXPOSURE
# FILE: herx1_off.pha
# KEYNAME: EXPOSURE
# EXTENSION: 0
# EXTENSION: 1
EXPOSURE= 9.28000000000000E+02 / Exposure time
fmodhead herx1_off.pha modhead.txt
where modhead.txt is an ASCII file containing the line:
EXPOSURE 696.0
Extracting Science Event Data
Defining Bitmasks with Sefilter
An example of using sefilter follows, but please note the following:
Bitmask files are simple, containing only the coded selection criteria and no record of their origin. Moreover, neither fselect nor seextrct can check whether a given bitmask is sensible. Fortunately, sefilter contains built-in safeguards: after creating a valid bitmask, sefilter initiates either fselect or seextrct, whichever is appropriate, to apply the filter.
Event == b1xxxxxxxxxxxxxxx.
rufus [41] [day] test: sefilter
Enter SE FITS file for filtering:[]@E_1ms_128M_0_8s.list
Enter the M-token to be processed (M[1]{1}):[M[1]{1}]
Enter the column name to be operated upon (Event):[Event]
Enter the output bitfile name to contain processed boolean expression:[]top.bit
Defaults are entered for the
M-token and column name. We've named the filter file itself top.bit.
Beginning generation of filter description.
###########################################
This file contains a Z-token, specifically:
Z[D[0]&E[X1L],D[0]&E[X1R],D[0]&E[X2L],D[0]&E[X2R],D[0]&E[X3L],D[0]&E[X3R],D[1]&E
[X1L],D[1]&E[X1R],D[1]&E[X2L],D[1]&E[X2R],D[1]&E[X3L],D[1]&E[X3R],D[2]&E[X1L],D[
2]&E[X1R],D[2]&E[X2L],D[2]&E[X2R],D[2]&E[X3L],D[2]&E[X3R],D[3]&E[X1L],D[3]&E[X1R
],D[3]&E[X2L],D[3]&E[X2R],D[3]&E[X3L],D[3]&E[X3R],D[4]&E[X1L],D[4]&E[X1R],D[4]&E
[X2L],D[4]&E[X2R],D[4]&E[X3L],D[4]&E[X3R]].
We have 5 bits for this token.
So this token can take on values from 0 --> 31
The Z-token is SPECIAL. So we will deal with it in detail.
Number of Z-tokens is 30
D[0]&E[X1L] == 0 D[2]&E[X2R] == 15
D[0]&E[X1R] == 1 D[2]&E[X3L] == 16
D[0]&E[X2L] == 2 D[2]&E[X3R] == 17
D[0]&E[X2R] == 3 D[3]&E[X1L] == 18
D[0]&E[X3L] == 4 D[3]&E[X1R] == 19
D[0]&E[X3R] == 5 D[3]&E[X2L] == 20
D[1]&E[X1L] == 6 D[3]&E[X2R] == 21
D[1]&E[X1R] == 7 D[3]&E[X3L] == 22
D[1]&E[X2L] == 8 D[3]&E[X3R] == 23
D[1]&E[X2R] == 9 D[4]&E[X1L] == 24
D[1]&E[X3L] == 10 D[4]&E[X1R] == 25
D[1]&E[X3R] == 11 D[4]&E[X2L] == 26
D[2]&E[X1L] == 12 D[4]&E[X2R] == 27
D[2]&E[X1R] == 13 D[4]&E[X3L] == 28
D[2]&E[X2L] == 14 D[4]&E[X3R] == 29
After reading the datafile headers, sefilter prints on the screen the part of the TEVTB2 keyword describing the selectable parts of the template. For this particular configuration, Z-tokens enumerate the various combinations of PCU and anode. We'll be selecting the ones corresponding to the X1L and X1R anodes of PCU0-5.
Do you want to select any of these? [Yes, No]: yes
Enter the numbers associated with the items you want to select: [0-5,15,24-30]
The above syntax must be followed (no negative values!): 0-1,6-7,12-13,18-19,24-25
You entered 0-1,6-7,12-13,18-19,24-25.
Is this correct? [Yes, No]: yes
The Z-token expression built up was: Z[] <= 1 | Z[] >= 6 & Z[] <= 7 | Z[] >= 12
& Z[] <= 13 | Z[] >= 18 & Z[] <= 19 | Z[] >= 24 & Z[] <= 25
Z-Filter is Z[] <= 1 | Z[] >= 6 & Z[] <= 7 | Z[] >= 12 & Z[] <= 13 | Z[] >= 18
& Z[] <= 19 | Z[] >= 24 & Z[] <= 25
The constructed filter is:
( Z[] <= 1 | Z[] >= 6 & Z[] <= 7 | Z[] >= 12 & Z[] <= 13 | Z[] >= 18 & Z[] <= 1
9 | Z[] >= 24 & Z[] <= 25 )
Running sebitmask to create the bitmask pattern:
Completed running sebitmask:
Expression generated by SEFILTER stored in outfile_368
Do you want to save this file? [Yes,No]no
Deleting outfile_368
The file we've elected to delete, outfile_368, contains the lines ( Z[] <- 1... etc. It is not essential. The more important file, top.bit, contains the bitmask itself, as shown below (sebitmask is the actual ftool that creates it).
Updating parfile for sebitmask:
Final bitmask generated by SEBITMASK is:
Event == b1xxxxxxxxxxxxxxx &&
(Event <= bxxxxxxxx00001xxx ||
Event >= bxxxxxxxx00110xxx &&
Event <= bxxxxxxxx00111xxx ||
Event >= bxxxxxxxx01100xxx &&
Event <= bxxxxxxxx01101xxx ||
Event >= bxxxxxxxx10010xxx &&
Event <= bxxxxxxxx10011xxx ||
Event >= bxxxxxxxx11000xxx &&
Event <= bxxxxxxxx11001xxx )
After creating the bitmask, sefilter then prompts us for an output file name for the filtered events (top.fits). Since our bitmask contains alternatives, fselect, not seextrct, will be run.
Enter the output filename to contain the processed data:[]top.fits
Using output of sebitmask as input and running Fselect:
Completed running fselect and created top.fits
We can now input top.fits into seextrct to extract light curves and spectra.
Example: Creating a Light Curve from PCA E_1ms_128M_0_8s Data
rufus [93] [day] test: seextrct
Running SEEXTRCT version 3.6
==============================================
Input file name or @file-of-filenames:[] top.fits
Input GTI files to be OR'd with INFILE (APPLY):[APPLY]
Input GTI file to be AND'd with INFILE:[-] elv_gt_10.gti
Root name for output file:[sac] flare_10ms
Name of TIME column:[TIME]
Name of COLUMN to be accumulated:[Event]
Input the binsize in seconds, use 0.1 etc. if nec (INDEF):[INDEF] 0.01
Minimum acceptable fractional exposure (INDEF):[INDEF]
Chose print option, LIGHTCURVE, SPECTRUM, or BOTH:[BOTH] lightcurve
Type of binning for LIGHTCURVE: (SUM, RATE, MEAN):[RATE]
Type of binning for SPECTRUM (SUM, RATE, MEAN):[SUM]
Maximum acceptable intensity for Light Curve (INDEF):[INDEF]
Maximum acceptable intensity for Spectrum (INDEF):[INDEF]
Column titles for SUM written as header Keywords:[-]
Column titles for MEAN written as header Keywords:[-]
Starting time for summation (INDEF):[INDEF]
Ending time for summation (INDEF):[INDEF]
Input time intervals t1-t2,t3-t4 in seconds (INDEF):[INDEF] @flare.timeint
Minimum energy bin to include in Spectra (INDEF):[INDEF]
Maximum energy bin to include in Spectra (INDEF):[INDEF]
Input channels to be retained 1-2,3-4 (INDEF):[INDEF]
Input channels for each bin 1-5,6-256 (INDEF):[INDEF]
Explanation of the Various Inputs in the Example above
The ABC of XTE is written and maintained by the RXTE GOF. Please email xtehelp@athena.gsfc.nasa.gov if you have any questions or comments. This particular page was last modified on Thursday, 16-Sep-1999 08:53:21 EDT.