Release Date
October 13, 2005
Functional Changes
-
Find Data Smart Polling - Smart polling functionality has been added to Find Data. When this option is invoked (via the -smart option), the frequency of Find Data polling of the file system will decrease over time at a prescribed step function rate when no data are found. Once a data file is found, the rate of polling is reset to its highest starting value (the theory being that once a data file is found, the likelihood of finding other data files is increased). The rate at which polling decreases over time (when no data are found) is configurable although there is a reasonable default if not specified. See the s4pm_stringmaker_string.cfg file for information on setting and defining smart polling in the Find Data station. Note that even as the frequency of polling decreases, the rate at which signal files are polled for remains the same. Thus, an operator can still issue a Ignore Option manual override and expect a fairly quick response regardless of what the current file system polling rate is.
-
Allocate Disk Bundled Allocation - The requests for disk pool space in the Allocate Disk station are handled more intelligently in this release. Rather than making a request for each data file as the input work order (PCF) is scanned, requests are first bundled by data type. After the scanning is complete, requests are made in bulk for each data type found. In this way, a PCF that has multiple outputs of the same data type results in a single, aggregated request for disk space rather than multiple individual requests. This should prevent a situation where more than one algorithm run, each having multiple outputs of the same data type, compete for space with none of them actually getting all the space required (even though there is enough space for at least one run).
-
PH Export Consistency - A bug was fixed whereby a string with none of the algorithms configured for Production History (PH) generation could nonetheless be configured to export those same PH files. Checking has been implemented so that there needs to be at least one algorithm configured for PH generation in order for the string as a whole to be configured to export PH.
-
Auto Request Bug Fix - The Auto Request tool has been fixed so that it doesn't grow too wide when many data types are included. An additional row is added for every 10 data types.
-
Command-line S4PM Administration - The S4PM Administration tool can be run on the command line (without a GUI) with the -task option. The -force option was intended to allow the tool to be run non-interactively from the command line (for example, by a script rather than a person). But the check it did for unrecognized files was still interactive. In this release, if the -force option is used, it is assumed that you don't want it to run interactively. Hence, the checking of unrecognized files in this case is disabled in this release.
-
S4P::read_config_file Deprecation - The function S4P::read_config_file() has been replaced by using the Safe module in almost all scripts.
-
Time Tools Daylight Savings Time bug - The Add_Delta_DHMS() in TimeTools.pm had a bug that produced incorrect results when times spanned a switch to or from daylight savings time. This bug has been fixed.
-
Leapsec Correction Bug - A fix was made to S4PM::leapsec_correction() which caused it to not handle leap seconds properly.
-
Stringmaster Deprecation - Note that this is the last release that contains Stringmaster. In subsequent releases, Stringmaster will be dropped from the baseline. Use Stringmaker instead.
-
External Product Dispatcher - This release includes support for the ECS External Product Dispatcher (EPD) system in on-demand processing strings. This support includes code changes to s4pm_track_requests.pl, s4pm_export.pl, s4pm_receive_pan.pl, in order of decreasing lines of code. It has also necessitated changes to a number of configuration files to handle a new job type (EXPORT_EPD) between the Track Requests and Export stations, and a new datatype (SBSTDATA) and destination (EPD) in the Export station. The .procmailrc mail filter also requires changes to intercept emailed PANs. The EPD mechanism is the default for all requests coming from the V0 Data Gateway. For such requests, the Ship Data station is not used. For requests not coming from the V0 Data Gateway, the old Distribution Command Line Interface (DCLI) is still used in the Ship Data station.
-
Optional $stats_index_datatype - In previous releases, the $stats_index_datatype parameter set in the Stringmaker algorithm configuration file was mandatory. In 5.8.0, it is optional. The default value is the first element of the @stats_datatypes array (i.e. $stats_datatypes[0]).
-
s4pm_algorithm_stats.cfg Bug - A bug was fixed in the generation of the s4pm_algorithm_stats.cfg file which is used to generate volume and performance statistics for algorithms. The bug basically broke this configuration file and hence, the statistics generated. This bug has been fixed.
-
Automatic Determination of %output_uses - In previous releases, the %output_uses parameter hash was required in all Stringmaker algorithm configuration files. In 5.8.0, the need for this parameter has been removed and output uses are determined automatically. The %output_uses parameter hash is now ignored. Note that this change addresses a bug that occurred when two or more algorithms producing the same output data type resulted in an incorrect output uses setting in the s4pm_register_data.cfg file in the Register Local Data station. Since the output uses are determined automatically, this bug no longer occurs.
-
Performance Improvements in On-Demand - A number of changes should increase performance in On-Demand strings: (1) The Find Data work order recycling frequency was shortened from 300 seconds to 1 second. This should alleviate the problem of jobs building up in the Find Data station. (2) The disk pools in On-Demand strings has been condensed to just two: INPUT and OUTPUT. All output data types are assigned to the OUTPUT disk pool. This should free up more space for processing or at least prevent space from being squandered. (3) The sort order in the Run Algorithm station was changed to FIFO which should increase performance when under heavy load.
-
Symbolicly linked input files now accounted for correctly - In strings that take advantage of symbolicly linking to the input data (e.g. to the ftp pull area on Datapool), the space allocations are automatically adjusted for the fact that the links themselves occupy almost no disk space. In previous releases, a lot of space was allocated for each data file without regard to whether it was an actual file or only a symbolic link.
-
Fix to the Punt Failure Handler - A bug that caused the Punt failure handler to fail was fixed. This involved a minor fix to s4pm_failed_pge_handler.pl as well as a fix to the Stringmaker code itself in how it generated a supporting configuration file.
-
New $toolkit_root Stringmaker Parameter - A new parameter, $toolkit_root, has been added to Stringmaker. It is in the host configuration file by default. This new parameter sets the location of the ECS Toolkit. The default location is $ecs_root/TOOLKIT/toolkit.
-
Adjustable Log Levels - S4P::logger will now consult the environment variable S4P_LOG_LEVEL before outputting a message. This log level is the minimum severity to log, but is set as text. The order of severities, from lowest to highest is:
-
DEBUG
-
INFO
-
WARNING
-
ERROR
- FATAL
Thus, if the S4P_LOG_LEVEL is set to "WARNING", only WARNING, ERROR and FATAL messages will be logged. The OUTPUT_DEBUG environment variable is deprecated, though the code is backward compatible.
Detailed File Changes
.procmailrc
- Modified to handle emailed PANs such as those generated by EPD, e.g.:
:0 b
* ^Subject:.*PDR Acceptance Notice: *\/S4PM.*PAN $HOME/g0dus02/on_demand/neutral/receive_pan/$MATCH
PDR.pm, FileGroup.pm, FileSpec.pm
S4P.pm
S4PM.pm
-
Modified the function S4PM::parse_patterned_filename() to return the file name extension as the last parameter.
-
Fixed a bug in S4PM::leapsec_correction() that produced an erroneous time correction.
-
Changed some INFO messages to DEBUG messages.
s4pm_algorithm_stats.pl
s4pm_allocate_disk.pl
-
Modified call to S4PM::parse_patterned_filename due to the change to this function.
-
Modified so that multiple requests for a single data type are bundled when calling S4P::ResPool::update_pool(). This should help prevent disk pool lock-outs when more than one such algorithm is competing for the same space.
s4pm_compose_request.pl
s4pm_create_string.pl
s4pm_delete_expired_data.pl
s4pm_export.cfg
s4pm_export.pl
-
Applied more uniform approach to log and failure messages (S4P::logger() and S4P::perish()).
-
Replaced S4P::read_config_file() with Safe.
-
Modified to process the EPD destination slightly differently, tweaking the PDR to conform to the External Subsetter ICD.
s4pm_failed_pge_handler.pl
s4pm_failed_service_handler.pl
s4pm_find_data.cfg
-
Modified with two additional parameters: @cfg_intervals and @cfg_freqs. These new parameters support the new smart polling feature of Find Data.
-
In keeping with tradition, the parameter name $Polling_Interval was changed to $cfg_polling_interval.
s4pm_find_data.pl
-
Implemented smart polling which is invoked with the -smart option. Adding this capability required some refactoring of the exiting code, particularly in fill_in_optional_file_groups() and fill_in_required_file_groups(). The code for snoozing between polling cycles was removed and put into a new subroutine, doze(). In addition, the subroutine get_polling() was added to return the polling frequency at any time.
s4pm_poll_data.pl
s4pm_prepare_run.pl
s4pm_preprocess_order.pl
s4pm_preselect_data.pl
s4pm_receive_pan.pl
-
Modified to guess the file_type if not specified in the PDR (as for EPD PDRs).
-
Tries to find PDRs in Limbo only if cfg_pdr_limbo variable is set. Configuration should be to NOT set it for EPD instances.
s4pm_request_data.pl
-
Generalized the way that the UR is modified so that it will work in any ECD DAAC.
-
Replaced S4P::read_config_file() with Safe.
-
Applied more uniform approach to log and failure messages (S4P::logger() and S4P::perish()).
s4pm_resubmit_pdr.pl
s4pm_run_algorithm.pl
s4pm_run_easy.pl
s4pm_select_data.pl
s4pm_ship_data.pl
s4pm_show_pge.pl
s4pm_split_order.pl
s4pm_split_services.pl
s4pm_stringmaker.pl
-
Changed Stringmaster to Stringmaker in several locations including what is written out in a Trouble Ticket.
-
Fixed a bug in how the s4pm_algorithm_stat.cfg file is generated.
-
Removed the reading of output uses and their summing as output uses determination has been moved to the s4pm_stringmaker_derived.cfg file.
-
Addressed the bug of the s4pm_failed_pge_handler.cfg file %Uses hash not getting constructed automatically. The problem was due to a substantial amount of code somehow getting dropped from this script. That code was added back in. (TT 16011)
-
Modified alloc_disk_setup() and generate_pool_capacities() subroutines so that in any strings employing symbolic links in the INPUT disk pool would force input data type file sizes to 2048 bytes (just enough to support space for the link) rather than the actual file size.
-
Modified code so that for on-demand strings, only two disk pools are created: INPUT and OUTPUT with all output data types being assigned to the OUTPUT disk pool. This should increase performance.
s4pm_stringmaker_algorithm.cfg
s4pm_stringmaker_datatypes.cfg
s4pm_stringmaker_derived.cfg
-
Removed references to variable $make_ph as it is not used in Stringmaker. Instead, the hash %all_ph_algorithms is used. Code was modified to properly use this hash and to not configure for export of PH unless at least one algorithm is configured to produce it.
-
Added functionality to support Find Data smart polling.
-
Fixed bug preventing sub_notify from being included in the station list in the s4pm_start.ksh script.
-
Changed default priority for on-demand in the ACQParmfile from HIGH to NORMAL.
-
Added some very gratuitous code so that for AIRS production, a work order of type REQUEST_DATA_AIRS is included in the station.cfg file of the Request Data station. Although gratuitous, it isn't harmful to any other type of processing.
-
Set $cfg_root in Auto Request station section correctly. It had been set to the default "../.." which is incorrect since this station is under the S4PM global root.
-
Made a number of changes to support ECS PDR Distribution (EPD) in On-Demand strings affecting the Track Requests, Export, and Receive PAN stations.
-
Fixed a bug in how the s4pm_algorithm_stat.cfg file is generated.
-
Added some lines to include automatically the output uses. The determination of output uses was removed from s4pm_stringmaker.pl and placed here.
-
Modified to automatically create the DATA/REQUESTS directory in on-demand strings to support EPD.
-
Replaced the $gear parameter with $instance as it affects data mining edition (DME) strings.
-
Changed the recycling frequency of work orders in the Find Data station for on-demand strings from 300 seconds to 1 second. This appears to increase overall performance.
-
Changed sort order to FIFO in Run Algorithm station for on-demand strings. This should help performance under heavy load.
-
Modified to use the new $toolkit_root parameter as set (by default) in the Stringmaker host configuration file. It defaults to $ecs_root/TOOLKIT/toolkit.
s4pm_stringmaker_host.cfg
s4pm_stringmaker_string.cfg
s4pm_sweep_data.pl
-
Modified call to S4PM::parse_patterned_filename due to the change to this function.
-
Applied more uniform approach to log and failure messages (S4P::logger() and S4P::perish()).
-
Replaced S4P::read_config_file() with Safe.
s4pm_tk_admin.pl
-
Disabled the running of ExtraFiles() if the -force option is set. The ExtraFiles() checks for unrecognized files and if found, prompts the user whether or not to delete them. When using the -force option, however, it is assumed that a script rather than a user is running it. Therefore having prompts is a problem.
s4pm_tk_auto_request.pl
s4pm_tk_compose_request.pl
s4pm_tk_compose_single_request.pl
s4pm_tk_disk_alloc.pl
s4pm_track_requests.pl
-
Got rid of redundant S4P::read_config_file() call. The configuration file is already being read using Safe.
-
If request was received through V0 Gateway (identified by the JOB_ID), it outputs an EXPORT_EPD PDR instead of SHIP.
-
Processes PDRs from Receive PAN to close orders.
-
Out of order processing: If work order is received from Find Data before Request Data, assumes that everything is fine and updates the status anyway. Also, if Request Data status cannot be updated (because it is already past that point), exits gracefully instead of failing.
stationmaster.pl
-
Added $cfg_rename_retries and $cfg_rename_retry_interval to work around the problem when Windows holds onto a file too long causing a directory rename to fail.
-
Added restart_station functionality to restart itself.
TimeTools.pm
tkstat.pl
|
 |
Atmos Composition
