Files:
CONF
Description
This important file provides all parameters used
in the calibration process. It contains references to
calibration matrices (such as flat fields, instrument gains,
etc...), label keywords and parameter constants used in the
radiometric calibration process.
The format of this PVL file is highly flexible for managing
HiRISE calibration data. This configuration file contains
numerous module profiles that govern the behavior of each phase
in the calibration process. These modules each have parameters
that are specific to its requirements. There are 6 different
modules that are used in the application. However, you will
typically see many other profiles in the configuration file.
The technique employed here is that you can take advantage of
the ability to merge other profiles into one or more of the
other main profiles, simply by the content of the labels or
observing conditions of the image.
The file must contain a top level Hical object. WIthin
this object is numerous profiles. The keywords that are
contained in the Object section of the file are always used in
every profile and can be superceded by any subsequent profile
loaded after the initial one. Optional profile names are
contructed by the combination of the OptionKeywords and
ProfileOptions keywords. The OptionKeywords
lists keywords in the configuration file and or label that can
be used to textually replace the patterns surrounded by the {}
in the ProfileOptions keyword. The kewords FILTER, CCD,
CHANNEL, TDI and BIN are always generated after the initial
module and label is loaded in. After the initial profile is
loaded the FROM file label is loaded using the
LabelGroups to determine which keywords are loaded. Note
that the groups must exist or an error is issued. Other than
that you an specify any defined keyword in the
OptionsKeywords to apply to profile names.
The process by which a file name is determined also utilizes the
OptionKeywords to replace patterns that exist in
retrieval of all filename references in the configuration file.
This includes virtually all files including matrix and CSV
files.
Matrices are currently multi-band ISIS cube files that contain
one line and up to 1024 samples. There are expected to be
28 bands in each one of thes calibration matrices, one for each
CCD channel. There are 14 different CCDs with 2 channels each
for a total of 28 channels. (Note that channels are the basic
HiRISE image product where calibration is applied.) There will
always be one line for these cubes since HiRISE detectors are
line scan instruments. There will be a minimum of 128 (for
summing mode of 16) and a maximum of 1024 (summing mode of
1) samples in these cubes. The content of these matrix files are
dependant upon summing mode and the time delay integration
(TDI) mode used during image acquisition. TDI allows varying
number of line scans, in conjunction with exposure time, to pass
over the same point on the surface of Mars to increase the
signal-to-noise ration (SNR) as well as resolution. There are 4
selectable modes of TDI: 128, 64, 32 and 8. Coupled with 6
difference summing modes (1, 2, 3, 4, 8, and 16), there are at
least 24 calibration cubes per set of matrices. These matrices
are referenced as file paths of the form:
A = $mro/calibration/matrices/A_TDI{TDI}_BIN{BIN}_????.cub
Here, {TD} and {BIN} correspond to the TDI and summming mode
used during image acquisition, respectively. (More on how these
values are determined is provided in the follow text.) Matrix variables
are equated to file references using the pattern substituion of
keywords enclosed in {} in the configuration file. Matrix file
paths are also subjected to pattern replacement in the same
fashion as profiles. This will minimize the content management
aspect of the configuration file and encourage consistant file
naming schemes.
The real power of the configuration file is its use of named Profiles.
Profiles are groups of keywords that can be associated to a
unique definition. The hical PVL configuration file consists of a
single Hical object with numerous named Profile
groups. Each of the Profile groups must contain a Name
keyword that uniquely identifies it within the Hical object.
This allows us to create Profiles that pertain to particular
combinations of filter (RED, IR, BG), CCD (RED0-9, IR10/11,
or BG12/13), TDI (128, 64, 32, 8), BIN (1, 2, 3, 4, 8, 16) or
CHANNEL (1 or 2). These values are determined from the
content of the HiRISE label. Combinations of profiles that are
added after the initial default are specified in the
ProfileOptions keyword. Profile combinations can
consist of any combination or use of these defined values.
These defined values are specified as or with the Name
keyword in Profile groups delimited by curly braces. Given
this definition one can specify a particular group of calibration
parameters for a specific CCD channel with the pattern
{FILTER}{CCD}_{CHANNEL}. Then, one can define a special
collection of calibration matrices, keywords or scalars for any
one (or none) of the filters. So, for the problematic BG13, you
can have a named profile called BG13_1 whose keywords in the
profile are loaded when calibrating a BG13_1 channel, thus
overriding any defaulted keywords. Should named
profiles using this option not exist, they are ignored. Also,
Profiles specified in this manner are loaded in the
order specified in the ProfileOptions keyword, thus
creating a hierarchy of calibration specification configurations.
Below is an example of a complete PVL configuration file that
demonstrates some of the features described:
# HiRISE Calibration Matricies configuration file
# See documentation for the hical application on the content and form of
# this file.
Object = Hical
Program = "hicalbeta"
Name = "HiMatrices"
DefaultProfile = "HiMatrices"
/* If you want to rerun hical, you must set PropagateTables to True. Use */
/* this in conjuction with Debug::SkipPhase = True option for each module. */
PropagateTables = False
/* Define label groups that are loaded for each profile reference */
/* Note all keywords in these groups become available to all profiles. */
/* Thus, you can use them in the OptionKeywords and ProfileOptions keywords */
/* to create very specialized profiles for special needs. */
LabelGroups = ( "Dimensions", "Instrument", "Archive")
/* These keywords are used in ProfileOptions mapping. Note that order and */
/* case matter! WARNING: You can easily break file lookups if these keys */
/* are deleted or modified improperly!!! */
OptionKeywords = ("FILTER", "CCD", "CHANNEL", "TDI", "BIN", "ProductId",
"Program", "Module", "OPATH")
/* Additional profile combinations and order load hierarchy. These keywords */
/* are defined when the LabelGroups are loaded. */
ProfileOptions = ("{FILTER}", "TDI{TDI}", "BIN{BIN}", "TDI{TDI}/BIN{BIN}",
"{FILTER}{CCD}_{CHANNEL}",
"{FILTER}{CCD}_{CHANNEL}/TDI{TDI}/BIN{BIN}", "Debug")
/* This profile contains parameters pertinent to processing the buffer */
/* pixels for subsequent using in the drift correction, Zd module */
Group = Profile
Name = Zf
Module = Zf
ZfFirstSample = 5
ZfLastSample = 11
ZfFilterWidth = 201
ZfFilterIterations = 2
End_Group
/* This profile contains parameters pertinent to processing drift correction */
Group = Profile
Name = Zd
Module = Zd
/* Turns Levenberg-Marquardt fitting off */
ZdSkipFit = True
/* Maximum number of iterations for the algorithm to converge and */
/* other limits */
MaximumIterations = 100
MaximumLog = 709.0
/* Convergence parameters for Levenberg-Marquardt algorithm */
/* Equation is solved when |dx_i| < AbsoluteError + RelativeError * |x_i|
*/ /* where dx is the last step and x is the current step for each i-th */
/* value */
AbsoluteError = 1.0E-4
RelativeError = 1.0E-4
/* Filtering of the guestimate buffer */
GuessFilterWidth = 17
GuessFilterIterations = 1
DumpModuleFile = "{OPATH}/{ProductId}_{Module}.log"
End_Group
/* This profile contains parameters pertinent to processing the offset */
Group = Profile
Name = Zz
Module = Zz
/* Set calibration parameters for hiclean operations. Indexes are all 0-based */
ZzFirstLine = 1
ZzLastLine = 19
DumpModuleFile = "{OPATH}/{ProductId}_{Module}.log"
End_Group
/* This profile contains parameters pertinent to processing dark current. */
/* Required label keywords: Summing, Tdi, FpaPositiveYTemperature, */
/* and FpaNegativeYTemperature, Lines */
/* Also needs LineTime which is computed. */
Group = Profile
Name = Zb
Module = Zb
/* Define the B matrix file reference */
B = "$mro/calibration/matrices/B_TDI{TDI}_BIN{BIN}_????.cub"
SkipLines = 1
Slope = "$mro/calibration/matrices/t_slope_????.csv"
Intercept = "$mro/calibration/matrices/t_intercept_????.csv"
ReferenceTemperature = 21.0
/* Do filtering? */
ZbFilterWidth = 3
ZbFilterIterations = 1
DumpModuleFile = "{OPATH}/{ProductId}_{Module}.log"
End_Group
/* This profile contains parameters pertinent to processing gain correction. */
/* Required label keywords: CpmmNumber, ChannelNumber, Lines */
Group = Profile
Name = Zg
Module = Zg
SkipLines = 1
GainLineCoefficients = "$mro/calibration/matrices/line_correct_{BIN}_????.csv"
DumpModuleFile = "{OPATH}/{ProductId}_{Module}.log"
End_Group
/* This profile contains parameters pertinent to processing gain correction. */
/* Required label keywords: CpmmNumber, ChannelNumber, Lines */
Group = Profile
Name = Zgg
Module = Zgg
/* Define the G matrix file reference */
G = "$mro/calibration/matrices/G_TDI{TDI}_BIN{BIN}_????.cub"
End_Group
/* This profile contains parameters pertinent to processing gain correction. */
/* Required label keywords: CpmmNumber, ChannelNumber, Lines */
Group = Profile
Name = Za
Module = Za
/* Define the A matrix file reference */
A = "$mro/calibration/matrices/A_TDI{TDI}_BIN{BIN}_????.cub"
End_Group
/* This profile contains parameters pertinent to processing I/F conversion. */
/* Required label keywords: ScanExposureDuration */
Group = Profile
Name = Ziof
Module = Ziof
/* Debug::SkipModule = True */
/* I/F correction for tdi/bin - currently set at 1.0 for all tdi/bin */h
/* combinations. */
ZiofBinFactor = 1.0
End_Group
/* Here are the filter profiles. All keywords that pertain to a filter set */
/* should be specified here. FilterGainCorrection are I/F corrections in */
/* units of DN/s. */
Group = Profile
Name = RED
FilterGainCorrection = 175613029.0
End_Group
Group = Profile
Name = IR
FilterGainCorrection = 58571990.0
End_Group
Group = Profile
Name = BG
FilterGainCorrection = 93575312.0
End_Group
Group = Profile
Name = IR10_1
# LastGoodLine = 3100
End_Group
Group = Profile
Name = Debug
/** Current disables writting to label history due to bug in keyword formatter in ISIS **/
/* The bug has the following error signature: */
/* terminate called after throwing an instance of 'std::out_of_range' */
/* what(): basic_string::substr */
/* Abort */
/* You must set this to false when this occurs as a workaround and use the */
/* DumpHistoryFile parameter to see the parameter history. */
LogParameterHistory = False
/* Uncomment this line to write parameter history to the ProductId log */
DumpHistoryFile = "{OPATH}/{ProductId}.{Program}.log"
End_Group
End_Object
The filter profiles each contain a scalar constant of the same name
for each of the three filter sets. The
ProfileOrder keyword
contains the
{FILTER} pattern that will select the appropriate
filter gleened from the label. Other profiles that satisfy the remaining
patterns are excluded but can be added when necessary.
The final resutling matrices, constants and keywords used in
the calibration equation are recorded in the
RadiometricCalibration group of the output label.
Application Control Parameters
Description of main hicalbeta configuration
parameters. These parameters govern profile loading,
filename pattern replacement and debugging operations.
| Parameter |
Description |
| Program |
This keyword is set to the name of the application and can
be used to create unique filenames as described previously.
|
| Name |
The name of the current profile. This keyword is
required and is present in the Object keyword section as
well as in all other Profile groups. This uniquely
identifies the final comingled profile.
|
| DefaultProifile |
This names the default profile that is loaded when none
are specifically called for in the application. It can
be any profile (Debug might be an interesting
alternative) but is generally the Object profile. This
will cause no other profiles to be added when a generic
one is retrieved.
|
| ProgagateTables |
Specifies the behavior of the table progation of the FROM
file to the TO file when the file is completed. This has
some interesting implications. When False, all Table
objects in the FROM file are removed in the TO file, which
in effect prevents hicalbeta from being able to run again.
However, because of debugging capabilities, one may want ot
select which calibration modules are run. Setting this to
True will propagate all Table objects (BLOBs) in the FROM
file to be copied to the TO file.
|
| LabelGroups |
This keyword identifies which Groups in the FROM ISIS
cube label are included in profiles. This allows any
label keyword in the ISIS label to become available for
profiling and filename generation. Its handy for
creating very specific profiles for problem images and
other debugging purposes.
|
| OptionKeywords |
This keyword selects other keywords in the configuration
file or the FROM label groups as defined in the
LabelGroups keyword. This list is used to specify
additional profiles that can be loaded (via
ProfileOptions) and variable substitution within file
names (such as B, G and A matrix files).
|
| Debug::SkipModule |
This special keyword exists to provide the ability
to complete bypass processing of a module. If this
configuration keyword resolves to True in any module
configuration profile, then the module is not
invoked and the resulting contribution is set
appropriately such that it does not contribute to
the calibration process. This is very useful for
debugging and seeing how each module contributes.
Be sure to set ProgagateTables to True if you
intend to perform subsequent runs of hicalbeta.
|
| OPATH |
This keyword is the value of the OPATH parameter
entered when the program is executed. It can be
used to specify a path where log files are written
when that option is invoked. If the user did not
specify a value for this keyword, the current path
is supplied by default.
|
| DumpModuleFile |
This special keyword exists to provide the ability
to dump data from each Module. Each Module
implements a data dump that will be written to this
file. The file name can be made up of a combination
of any label or configuration keyword. The example
provided in the config file section uses the
ProductId and the name of the Module with a .log
extension. This keyword can be included in
individual Module Groups or at the top level which
will effectively dump all Modules. Note that the
Ziof Module is excluded from this feature as its
data is contained entirely in the history report.
|
| DumpHistoryFile |
This special keyword exists to provide the ability
to dump the Module history from each Module. Each
Module maintains a processing history that will be
written to this file. The file name can be made up
of a combination of any label or configuration
keyword. The example provided in the config file
section uses the ProductId and the name of the
Program with a .log extension.
|
Zf Module Parameters
This module is responsible for reading and processing of
the Buffer pixels for input into the DriftCorrect module.
It averages lines, runs smoothing filters and ensures
there are no gaps by resorting to a Spline Fill of any
remaining missing data.
| Parameter |
Description |
| Name |
Must be set to Zf as it names this Profile Module. The
program will fail without this Group/Name/Module.
|
| Module |
The name of the current processing module. It is
intended to be used as an identifer. Name is such a
generic name that it has the potential to be superceded
by some other profile loading operation (such as label
activities).
|
| ZfFirstSample |
Specifies the first (0-based) Buffer sample to start the
average.
|
| ZfLastSample |
Specifies the last (0-based) Buffer sample to end the
average.
|
| ZfFilterWidth |
Specifies the width of the filter to smooth and fill
gaps in the resulting Buffer averages. [Default: 201]
|
| ZfFIlterIterations |
Specifies the number of sequential filters to apply to
the averaged buffer pixels before finally filling with a
spline. [Default: 2]
|
Zd Module Parameters
The DriftCorrect Module. This module uses the Zf results
and computes a non-linear equation that removes the drift
component. It uses a Levenberg-Marquardt algorithm to
solve the equation f(x) = a0 + a1 * linetime + a2 * exp(a3
* linetime).
| Parameter |
Description |
| Name |
Must be set to Zd as it names this Profile Module. The
program will fail without this Group/Name/Module.
|
| Module |
The name of the current processing module. It is
intended to be used as an identifer. Name is such a
generic name that it has the potential to be superceded
by some other profile loading operation (such as label
activities).
|
| ZdSkipFit |
This parameter can is used to turn on/off the
non-linear fitting process described above. When
missing or set to TRUE, the result of the Zf module is
used. If FALSE, the DriftCorrect fitting is applied
using the parameters provided.
|
| ZdOnFailUseLinear |
This parameter can be used to select the behavior of
the data produced by the Zd module when the non-linear
fitting process has failed (typically due to
non-convergence). If set to TRUE, a linear fit to the
latter half of the Zf (buffer pixel) data is computed.
If FALSE or non-existant, the result of the Zf module
is used as is.
|
| ZdMinimumLines |
Minimum number lines required to apply the non-linear
fit processing. The actual number of lines used is the
total lines less TrimLines/Summing. If there are not
enough lines, then the result of the Zf module (filtered
Buffer pixel data) is simply passed through as is. This
results in the same behaviour when the ZdSkipFit
parameter is invoked.
|
| MaximumIterations |
The maximum number of iterations to allow the equation
to converge.
|
| MaximumLog |
Constrains the last term in the equation such that the
exponenent will not cause an overflow.
|
| AbsoluteError |
Specifies the absolute maximum error to determine
convergence.
|
| RelativeError |
Specifed the relative maximum error to determine
convergence.
|
| GuessFilterWidth |
The data used to fit the non-linear equation can be
optionally filtered before fitting. This controls the
width of the filter [Default: 17].
|
| GuessFilterIterations |
This parameter controls the number of times the fit
buffer is filtered prior to non-linear fitting. Set
this to 0 to turn off this processing and use the data
as is [Default: 1].
|
Zz Module Parameters
Computes the Offset component. This module uses the
Reverse Clock region to compute the offset that is then
subtracted from the result of the Zd module.
| Parameter |
Description |
| Name |
Must be set to Zz as it names this Profile Module. The
program will fail without this Group/Name/Module.
|
| Module |
The name of the current processing module. It is
intended to be used as an identifer. Name is such a
generic name that it has the potential to be superceded
by some other profile loading operation (such as label
activities).
|
| ZzFirstLine |
Specifies the first of the Reverse clock region in the
input FROM file to average for dark current correction.
|
| ZzLastLine |
Specifies the last line in the Reverse clock region in
the input FROM file to average for dark current
correction.
|
Zb Module Parameters
The Dark Subtract module. This module uses the
Slope/Intercept Temperature profile.
| Parameter |
Description |
| Name |
Must be set to Zb as it names this Profile Module. The
program will fail without this Group/Name/Module.
|
| Module |
The name of the current processing module. It is
intended to be used as an identifer. Name is such a
generic name that it has the potential to be superceded
by some other profile loading operation (such as label
activities).
|
| B |
This the B matrix cube that is derived to remove the
dark current. In addition, it uses an elaborate
temperature model for this component.
|
| SkipLines |
Specifies the number of lines to skip in the Slope and
Intercept temperature profile CSV files. [Default: 0]
|
| Slope |
Specifies the file pattern of the slope component of the
Temperature profile.
|
| Intercept |
Specifies the file pattern for the intecept component of
the Temperature profile.
|
| ReferenceTemperature |
Specifies the reference temperature to normalize the
temperature profile equation. [Default: 21C]
|
| ZbFilterWidth |
Width of the smooth filter component for the temperature
profile.
|
| ZbFilterIterations |
Specifies the number of sequential filters to apply to
the resulting temperature profile. [Default: 1]
|
Zg Module Parameters
Computes the Gain versus Line correction. This will read
in the line correction gain coefficients file and apply
the same non-linear equation used in the Zd module.
| Parameter |
Description |
| Name |
Must be set to Zg as it names this Profile Module. The
program will fail without this Group/Name/Module.
|
| Module |
The name of the current processing module. It is
intended to be used as an identifer. Name is such a
generic name that it has the potential to be superceded
by some other profile loading operation (such as label
activities).
|
| SkipLines |
Specifies the number of lines to skip in the gain
coefficients CSV files. [Default: 0]
|
| GainLineCoefficients |
Specifies the file pattern of the gain/line
coefficients.
|
Zgg Module Parameters
This module loads the Gain correction matrix file.
| Parameter |
Description |
| Name |
Must be set to Zgg as it names this Profile Module. The
program will fail without this Group/Name/Module.
|
| Module |
The name of the current processing module. It is
intended to be used as an identifer. Name is such a
generic name that it has the potential to be superceded
by some other profile loading operation (such as label
activities).
|
| G |
This the G matrix cube that is derived to remove the
effects of image gain.
|
Za Module Parameters
This module loads the flat field correction matrix file.
| Parameter |
Description |
| Name |
Must be set to Za as it names this Profile Module. The
program will fail without this Group/Name/Module.
|
| Module |
The name of the current processing module. It is
intended to be used as an identifer. Name is such a
generic name that it has the potential to be superceded
by some other profile loading operation (such as label
activities).
|
| A |
This the A matrix cube that is derived to remove the
flat field effects of image.
|
Ziof Module Parameters
This module computes parameters necessary to convert DNs
to I/O values.
| Parameter |
Description |
| Name |
Must be set to Ziof as it names this Profile Module. The
program will fail without this Group/Name/Module.
|
| Module |
The name of the current processing module. It is
intended to be used as an identifer. Name is such a
generic name that it has the potential to be superceded
by some other profile loading operation (such as label
activities).
|
| ZiofBinFactor |
Specifies the factor that takes into account binning of
the image data. It is currently set to 1.0.
|
| FilterGainCorrectionr |
Specifies the filter dependant gain correction factor.
Note that these values are found in the RED, BG, IR
profiles and relies on the profile options to load the
proper parameter in at runtime.
|
|
Type
| filename |
|
File Mode
| input |
|
Default Path
| $mro/calibration |
|
Default
| $mro/calibration/hicalbeta.????.conf |
|
Filter
| *.conf |
