Evaluation Findings

This evaluation examined the technical and financial performance of several promising technologies for increasing the security of HAZMAT shipments. The intent was to determine what levels of security, safety, and operational efficiency benefits can be attained through deployment of the technologies. The evaluation also examined the levels of investment required to equip fleets with the technologies. Based on the evaluation of the test technologies, conclusions can be drawn regarding: technical performance; security benefits; efficiency benefits; safety benefits; public sector reporting center concept; deployment potential; consolidated benefits and costs; and policy options for consideration, described as follows.

TECHNOLOGY PERFORMANCE

Technical performance of the FOT technologies was demonstrated through initial Beta Testing, then during the operational test through technology exercises and "staged events" defined by the Evaluation Team. The staged events tested the systems in the field under near real-world conditions. Additionally, day-to-day performance of the test technologies' performance was captured via participant interviews and analysis of archived event transaction logs.

This evaluation approach resulted in findings that overall, technology performance for the technologies was good, with most technologies performing well under operational conditions. The exceptions included Biometric Login, and to a lesser extent, the E-seals, and the ESCM, which were deemed as requiring additional technical development for the HAZMAT trucking environment.

The "core enabling technology" for the test suites, Wireless Communications with GPS Tracking Capabilities, has been deployed commercially for several years and performed per expectations during the FOT. The technology also demonstrated its ability to integrate additional security functions with the established communications network providing a reliable data transfer mechanism. Additionally, all eight participant carriers that have previously and continued to use this Wireless Communications technology attest to the positive efficiency impact it has had on their operations, and all showed robust technology utilization.

Positioning frequency ranged from 17 to 70 minutes for FOT participants that depended on operational conditions, such as desired customer reporting frequency, type of commodity being hauled, and length of route.

Global Login proved to be a reliable form of driver identification at the four carriers who were assigned the technology during the FOT. Several other carriers used Global Login as a backup to Biometric Login when it failed. The time required for Global Login was consistent across FOT participants. Global Login events were completed successfully several times at each site in about 30 seconds. Incorrect Global Login events were also conducted to show the ability of the system to reliably detect incorrect login attempts under operational conditions. Incorrect Global Logins also took approximately 30 seconds for the system to detect. Global Login was generally well received by drivers who found that training was brief and simple, especially when compared to the Biometric Login.

The actual experience that test participants had with the Biometric Login device used in this FOT was that it was often unreliable in the field. The unreliability was due to the difficulty in finger placement to introduce consistent fingerprints into the Biometric Reader. Correct finger placement would allow either the vehicle to properly start or for employees to log into the ESCM system to work with manifest files. Driver complaints were high for this technology in regard to usability in the field, suggesting a need for overall design improvement, such as a finger guide to properly align drivers' fingers on the reader. In addition to difficulty in finger placement location for participants, if a driver's finger were too hot or too cold, the Biometric Reader would often fail to obtain a successful login event.

Use of the Electronic Supply Chain Manifest was disappointing and statistically irrelevant during the course of the FOT, with only 55 manifests being successfully created, and of these, only 12 were utilized through delivery of the HAZMAT load. Some of this poor usage resulted from problems encountered with the Biometric Login, similar to the problems encountered with the driver login procedure. As a stand-alone system, participant usage also suffered due to the redundancy of having to use both the paper and electronic manifest processes, and also due to user difficulties with the Web interface.

Both in-dash Panic Buttons and driver remote Panic Buttons were well accepted by the motor carriers in this test, and 118 test events were conducted to validate their successful use.³ Recorded panic alert notifications from the technology exercise site visits took between 25 seconds to about 1 minute from the moment the Panic Button was pressed to the point when the dispatcher was alerted at the motor carrier facility. Panic Buttons were seen as a viable technology method to alert the motor carrier or law enforcement from remote regions of the nation. Panic Buttons were viewed as having excellent security potential; in fact, several of the participants already had in-dash Panic Buttons installed prior to this FOT and expressed excellent satisfaction with the technology. The only issue associated with the wireless Panic Button was that some drivers felt the key fob (security token) design could cause an alert to be issued by a driver by accidentallyy bumping the trigger device.

Geofencing was tested in two operational uses for alerting a trucking company when one of its vehicles leaves its designated route (mapped on computer by the dispatcher), and for when one of its vehicles enters a restricted area. Both functionalities involved frequent vehicle positioning via Wireless Communications. Geofencing was viewed by the FOT participant who used it as an excellent technology to locate a vehicle that was off route or in an area where management did not want that truck to be positioned. The FOT participant thought it would useful as a tool not only to improve security, but it might keep drivers from stopping for excessive periods of time at unauthorized locations.

The Electronic Seals testing in this FOT demonstrated that this technology might not be realistic in a HAZMAT trucking environment at this time. For example, the initial deployment of the E-seals had significant problems in communicating with the on-board communications system with trucks having heavy steel doors. In addition, the cycle of operations for a driver to operate the seal took several minutes, and at times confused the drivers. By the end of the FOT, E-seal operations had improved somewhat.

The Remote Door Look system testing, while inconclusive due to only 16 data points, showed promise of being another cargo security solution that could be considered for HAZMAT loads along with E-seals. This sequence of events was demonstrated during on-site testing at the FOT participant carrier using this functionality on one truck. Additionally, the single participant using this technology concluded that it had worked excellent in daily operations, it had merit as a security device, and it might be a cost-effective means of security for a HAZMAT load.

Both the Tethered and Untethered Trailer Tracking technologies were well received by the FOT participant using these technologies under test conditions. Dispatch found useful the ability to detect trailer connects and disconnects with the Tethered Trailer Tracking system, and the ability to track an unconnected trailer as another authorized carrier moved it. Both technologies were used on a consistent basis during the FOT.

Vehicle Disabling was accomplished by integrating Intelligent On-Board Computers (OBC) with the Wireless Communications/vehicle operating systems. The OBC permitted the motor vehicle to be disabled in the event of a security breach. These remote disabling techniques included blocking fuel or sending instructions via the Wireless Communications system directly to the vehicle's data bus, which caused loss of throttle power to the motor vehicle. The OBC also was configured to shut down the vehicle whenever there was a loss of satellite signal strength, such as when cables are tampered with or the receiver unit is covered. One variant of the vehicle disabling capability that did not require the use of the OBC was local vehicle disabling. By the driver depressing the panic button of his key fob, a signal was sent directly to the vehicle to initiate the disablement. The wireless panic button with local disabling capability is carried by the driver and has a range of up to 250 feet. This latter application does not require the OBC to perform the local vehicle disablement.

SECURITY BENEFITS

To assess security benefits for these technologies, the Evaluation Team devised a groundbreaking process and corresponding methodologies to assess the security benefits of transportation technologies. Building on previous work conducted by Battelle, Total Security.US, and the Volpe Center, the Evaluation Team combined expert panel review information to assess the potential benefits. As part of this process, a "Security Expert Panel" was established specifically for this project to provide input from TSA and other recognized experts.

The Security Expert Panel was co-chaired by the Deputy Director for Land and Maritime Security at TSA. This Panel also included representatives from major trade associations, the insurance industry, and other security and counter-terrorism experts. This Panel provided input into staged and controlled events testing the technologies. In addition, the Security Expert Panel provided oversight for the Delphi Panel, which included the selection of a wider group of experts in predicting potential security vulnerabilities relating to terrorist attacks.

The Delphi Panel used an iterative process of reviewing and responding to several surveys used to quantify the security impacts of the technologies to identify benefits in the form of vulnerability reductions. These vulnerabilities in truck-based HAZMAT shipping were defined by the Deployment Team and used with the Delphi Panel assessments to derive an overall vulnerability reduction. The Delphi Panel also reviewed and provided comments to the Security Expert Panel on the final analysis.

In implementing this process, the detailed daily operational and staged event data was collected to assess the technical efficacy of the technology suites and participant perceptions regarding the value of the technologies to their operations vis-à-vis reducing operational vulnerabilities to terrorist activity. This information was summarized and presented to the Delphi Panel⁴ of 26 nationally recognized experts in HAZMAT transportation security and risk assessment. The Delphi Panelists were requested to render educated expert opinions regarding the ability of the technologies to reduce vulnerabilities in truck-based HAZMAT shipping, as related to three primary areas of security threats to HAZMAT trucking operations: theft, diversion, and interception and their contributing vulnerability factors: chain of custody, access, and response time.

Using the relative weightings for these factors provided by the Delphi Panel and their informed opinions regarding the technologies' ability to address critical vulnerabilities, overall reductions in vulnerabilities were calculated for each technology combination for each of the test load types. Table 3 presents a sample of the technology-enabled vulnerability reductions.

Table 3. Percent Reduction in Overall Vulnerability by Load Type and Technology Scenario⁵

The overall vulnerability reductions were then applied to the dollar figures developed as cost estimates for a terrorist attack involving hazardous materials to determine the overall security benefits of the technologies to determine the estimated security benefits. The estimated "worstcase" attack consequences were established through a separate study sponsored by FMCSA. Sample benefit calculations are shown in Table 4.

Table 4. Vulnerability Reductions for Bulk Fuel Scenario and Security Benefits

In all load cases the overall security benefit cost ratio was positive, except for the LTL environment. In this case, the potential consequence and attractiveness of the LTL loads for use as a weapon of mass effect is relatively low and the number of trucks that would require being equipped is relatively high. As discussed above, the primary benefits in both the security and efficiency impact categories were derived from the use of Wireless Communications with GPS positioning. It should be noted that partial deployment might not necessarily result in a directly proportional security benefit. In other words, 50 percent deployment may not yield 50 percent of achievable security benefits. This may occur because while the technology-equipped fleet may not be attacked, a non-equipped fleet would possibly be targeted instead. The deterrent effect of the technologies, if partly deployed, could simply shift terrorist targeting from one fleet to another, with no net change in overall security. Under this aassumption, then full deployment is required to realize the security benefits.

Realizing that threat can be unpredictable and vary over time, breakeven numbers of successful attacks that needed to be reduced via the technologies to equal the costs of deploying the technologies was calculated. These breakeven values were calculated using the following formula:

Breakeven Number of Attacks =
(Total Deployment Cost for the Technology / Consequence per Attack)

The breakeven number of attacks is presented as a decision tool - if one believes that the probability of an attack (threat) is greater than the breakeven for a technology combination for a load type, and then to society, the investment in the technology combination can be considered sound. For example, preventing one attack over the 3-year period would easily surpass the breakeven point for all scenarios other than the LTL operations.

For context, the highest breakeven numbers for each load type were compared to prognostications made by the Delphi Panel as to the number of attack attempts on/using truck-based HAZMAT shipments and the proportion of those attempts that are likely to be successful within the next 3 years. The Delphi Panel, at the low end, indicated the number of successful attacks as being likely exceed the breakeven attack numbers for all load types, except LTL-High Hazard loads. In summary, for all load types, except LTL-High Hazard, the Panelists feel there is at least a 5 times greater probability of successful attack than is required for equating security benefits with deployment costs.

Footnotes

³ Panic Buttons were not tested during normal operations due to the sensitive nature of the technology and not creating "false alarms".

⁴ In the Delphi process, the experts were asked to provide estimates of vulnerability and the beneficiary effect of the FOT-considered technologies via surveys. Both numerical and linguistic responses were developed over a series of group interrogations. Outputs that had linguistic values were then processed using Soft Computing Methods in order to provide input values that support conventional Multi-Attribute Decision Making Methods. The benefit levels, in terms of the vulnerability reduction potential for each shipment type, were evaluated using a weighted sum method that considers the probability of each particular threat. Finally, the process included a computational element to allow economic analysis, such as Cost/Benefit or Net Benefit analysis for each technology application strategy.

⁵ Vulnerability reductions from 0-10 percent are considered nil; reductions from 11-25 percent are considered low; reductions from 26-50 percent are considered medium; and greater than 50 percent are considered a high reduction.

Continue to: Table of Contents > Part 1 > 2 > Part 3 > 4 > 5