This is the accessible text file for GAO report number GAO-06-612 
entitled 'Homeland Security: Guidance and Standards are Needed for 
Measuring the Effectiveness of Agencies' Facility Protection Efforts' 
which was released on July 7, 2006. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Report to the Chairman, Committee on Government Reform, House of 
Representatives: 

United States Government Accountability Office: 

GAO: 

May 2006: 

Homeland Security: 

Guidance and Standards Are Needed for Measuring the Effectiveness of 
Agencies' Facility Protection Efforts: 

GAO-06-612: 

GAO Highlights: 

Highlights of GAO-06-612, a report to the Chairman, Committee on 
Government Reform, House of Representatives 

Why GAO Did This Study: 

The protection of U.S. federal facilities has become an important 
concern due to the ongoing threat of terrorism. The General Services 
Administration (GSA), U.S. Postal Service (USPS), and the Departments 
of Veterans Affairs (VA) and Interior (Interior) hold the most 
domestic, nonmilitary property. Additionally, the Department of 
Homeland Security (DHS) is responsible for the protection of GSA 
facilities. DHS chairs the Interagency Security Committee (ISC), which 
is tasked with coordinating federal agencies’ facility protection 
efforts. The need to better protect federal facilities, as well as 
federal budget constraints, have prompted the need for these agencies 
to measure the performance of their facility protection efforts. GAO’s 
objectives were (1) to identify examples of performance measures for 
facility protection being used by selected organizations outside of the 
federal government; and (2) to determine the status of U.S. federal 
agencies’ efforts to develop and use performance measures as a part of 
their facility protection programs. 

What GAO Found: 

GAO found a range of examples of performance measures that 
organizations outside the U.S. government—including private-sector 
entities, state and local governments, and foreign government 
agencies—have developed that, collectively, indicate whether facility 
protection efforts are achieving results (see figure below). These 
organizations use security-related performance measures to help improve 
security, make decisions about risk management and resource allocation, 
and hold employees accountable for whether a program meets its security 
goals and objectives. However, many of the organizations said that 
developing and using these measures can be challenging and that they 
look to the U.S. government for assistance and leadership in developing 
standards and guidance for facility protection. 

Figure: Performance Measurement Types, Examples, Uses, and Results: 

[See PDF for Image] 

Source: GAO. 

Note: Output measures focus on the direct product/services delivered by 
a program. Outcome measures provide information on the results of 
products/services. 

[End of Figure] 

We found that some bureaus and services within DHS (for GSA 
properties), USPS, and Interior are using security performance 
measures, while the VA and other bureaus and services within the three 
agencies collect data that could be used to measure security 
performance. Agencies that have performance measures use them to ensure 
adequate protection at individual facilities, make risk management 
decisions, and evaluate program effectiveness. However, agencies face 
challenges—similar to those cited by nonfederal entities—in further 
developing and using security performance measures. Currently, there is 
no governmentwide guidance or standards on measuring facility 
protection performance to help federal agencies address these 
challenges. This differs from information technology security, where 
agencies have detailed, governmentwide guidance for developing and 
using performance measures. Without effective performance measurement 
data, decision makers may have insufficient information to evaluate 
whether their investments have improved security or reduced federal 
facilities’ vulnerability to acts of terrorism or other forms of 
violence. ISC is uniquely positioned to develop and disseminate 
guidance and standards for measuring the performance of federal 
government facility protection efforts. 

What GAO Recommends: 

GAO is recommending that the Secretary of DHS direct ISC to establish 
guidance and standards for measuring performance in federal government 
facility protection. DHS agreed with the findings and recommendations 
in this report. 

[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-612]. 

To view the full product, including the scope and methodology, click on 
the link above. For more information, contact Mark Goldstein at (202) 
512-2834 or goldsteinm@gao.gov. 

[End of Section] 

Contents: 

Letter: 

Results in Brief: 

Background: 

Organizations outside of the U.S. Government Use Security Performance 
Measures to Enhance Decision Making and Help Ensure Accountability: 

U.S. Agencies Have Made Some Progress in Developing and Using 
Performance Measures for Facility Protection Programs, but Lack 
Guidance and Standards: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendix I: Objectives, Scope, and Methodology: 

Appendix II: Examples of Performance Measures Used by Selected 
Organizations outside of the Federal Government: 

Appendix III: Comments from the Department of Homeland Security: 

Appendix IV: Comments from the Department of the Interior: 

GAO Comments: 

Appendix V: GAO Contact and Staff Acknowledgments: 

Tables: 

Table 1: Examples of Performance Measures for Facility Protection: 

Table 2: FPS's Performance Measures for Facility Protection: 

Table 3: BOR's Performance Measures for Facility Protection: 

Table 4: Inspection Service's Performance Measure for Facility 
Protection: 

Table 5: Types of Information Technology Security Performance Measures 
Described by NIST: 

Table 6: U.S. State and Local Governments Contacted: 

Table 7: Foreign Government Agencies and Organizations Visited: 

Figures: 

Figure 1: Smart Card Access Portals at a Federal Building Entrance: 

Figure 2: Linkages between District of Columbia Strategic Goals and 
Performance Measures for Facility Protection: 

Figure 3: Linkages between DHS Mission and FPS Performance Measures for 
Facility Protection: 

Figure 4: Linkages between USPS Inspection Service Strategic Goals and 
Performance Measure for Facility Protection: 

Figure 5: Sample Standardized Performance Measurement Data Form: 

Abbreviations: 

BOR: Bureau of Reclamation: 

DHS: Department of Homeland Security: 

DSO: departmental security officer: 

FPS: Federal Protective Service: 

GPRA: Government Performance and Results Act of 1993: 

GSA: General Services Administration: 

HSPD-7: Homeland Security Presidential Directive Number 7: 

ICE: Immigration and Customs Enforcement: 

ISC: Interagency Security Committee: 

IT: information technology: 

NIPP: National Infrastructure Protection Plan: 

NIST: National Institute of Standards and Technology: 

NM&I: National Monuments and Icons Assessment Methodology: 

OLES: Office of Law Enforcement and Security: 

OMB: Office of Management and Budget: 

PART: Program Assessment Rating Tool: 

USPS: United States Postal Service: 

VA: Department of Veterans Affairs: 

United States Government Accountability Office: 
Washington, DC 20548: 

May 31, 2006: 

The Honorable Tom Davis: 
Chairman: 
Committee on Government Reform: 
House of Representatives: 

Dear Mr. Chairman: 

The threat of terrorism has increased the emphasis on physical security 
for federal real property assets since the 1995 bombing of the Alfred 
P. Murrah Federal Building in Oklahoma City; the 1998 embassy bombings 
in Africa; the September 11, 2001, attacks on the World Trade Center 
and the Pentagon; and the anthrax attacks in the fall of 2001. The 
federal government owns or leases an estimated 3.2 billion square feet 
of space within the United States in more than 450,000 buildings, which 
are regularly accessed by millions of federal employees, contractors, 
and citizens. Approximately 42 percent of this square footage is 
nonmilitary property, and a majority of this is under the control or 
custody of the General Services Administration (GSA), the United States 
Postal Service (USPS), the Department of Veterans Affairs (VA), and the 
Department of the Interior (Interior).[Footnote 1] Under the Homeland 
Security Act of 2002, the Federal Protective Service (FPS), which 
protects GSA properties, was transferred to the Department of Homeland 
Security (DHS). For agencies that aim to ensure public access to their 
assets, protecting nonmilitary real property assets can be complex and 
contentious because of the need to strike a balance between public 
access and security.[Footnote 2] Federal agencies face additional 
security-related challenges, such as securing federally leased space 
and addressing conflicts with state, local, or private entities that 
also have jurisdiction over, or input regarding, physical security 
enhancements. The challenge of protecting federal facilities against 
the threat of terrorism was a major reason GAO designated federal real 
property as a high-risk area in January 2003.[Footnote 3] 

Although FPS is primarily responsible for protecting GSA properties, it 
also has responsibility for broader efforts across the federal 
government to enhance the protection of critical facilities and works 
closely with the Interagency Security Committee (ISC) on these issues. 
The ISC, which DHS chairs, is tasked with coordinating federal 
agencies' facility protection efforts, developing protection standards, 
and overseeing implementation of those standards.[Footnote 4] In 
November 2004, we recommended that ISC develop an action plan for 
fulfilling its responsibilities and establish a set of key practices 
for facility protection.[Footnote 5] We identified several key 
practices in facility protection, which included using risk management 
to allocate resources;[Footnote 6] leveraging security technology; 
coordinating protection efforts and sharing information; realigning 
real property assets to an agency's mission, thereby reducing 
vulnerabilities; strategically managing human capital; and measuring 
program performance and testing security initiatives.[Footnote 7] With 
regard to measuring performance, performance measures can be classified 
as output measures, which focus on the quantity of direct products and 
services a program delivers; outcome measures, which offer information 
on the results of the direct products and services a program has 
delivered; or process/input measures, which address the type or level 
of program activity an organization conducts and the resources used by 
the program. Outcome measures are particularly useful because they 
indicate what program activities are accomplishing. At the time of our 
November 2004 report, agencies were only in the early stages of 
implementing security performance measures. 

The need to better protect federal facilities, coupled with federal 
budget constraints and the increased scrutiny of homeland security 
funding and programs, has prompted the need for U.S. agencies to 
measure the performance of their facility protection efforts. In this 
environment, it is important for these agencies to ensure that 
investments in facility protection are providing adequate returns in 
terms of better protecting real property assets against terrorism. In 
addition, the U.S. government's national strategy, Presidential 
directive, and guidance on protecting critical infrastructures-- 
including facilities--have identified the use of performance 
measurement as a key means of assessing the effectiveness of protection 
programs. Given that protection of critical infrastructures is an 
important issue for organizations outside of the federal government as 
well, it is beneficial to look to the experiences of these 
organizations to identify lessons learned. As such, our objectives for 
this review were (1) to identify examples of performance measures for 
facility protection being used by selected organizations outside of the 
federal government--including private-sector entities, state and local 
governments, and foreign governments, and (2) to determine the status 
of U.S. federal agencies' efforts to develop and use performance 
measures as part of their facility protection programs. To address the 
first objective, we interviewed private-sector representatives from 
four entities in the gaming industry and from five major financial 
services entities, because these industries were identified as having 
invested in security and likely to have developed performance measures. 
We also interviewed officials from 17 of the 20 state and local 
governments that received the most funding from two security-related 
DHS grant programs in fiscal year 2005.[Footnote 8] Finally, we 
interviewed government officials from multiple agencies in Australia, 
Canada, and the United Kingdom, because these countries have experience 
with threats of terrorism and have performance measurement initiatives. 
We also reviewed relevant documents we obtained from these 
organizations, related GAO reports, and literature on performance 
measurement. To address the second objective, we interviewed federal 
officials from DHS, GSA, USPS, VA, and Interior--the agencies that 
hold, or are responsible for the security of, the majority of the 
domestic, nonmilitary property. We also reviewed pertinent documents 
and policies obtained from these agencies, in addition to related laws 
and directives. A detailed discussion of our scope and methodology, 
including more information on how we selected the organizations we 
contacted, is contained in appendix I. We conducted our work between 
June 2005 and April 2006 in accordance with generally accepted 
government auditing standards. 

Results in Brief: 

We found a range of examples of performance measures that organizations 
outside the U.S. government, including private-sector firms, state and 
local governments, and foreign government agencies, use to help improve 
the security of facilities, inform risk-management and resource- 
allocation decisions, and hold security officials and others in their 
organizations accountable for security performance. These included 
output measures, such as the average time to process background 
screenings, and outcome measures, such as the change in the total 
number of security incidents relating to thefts, vandalism, and acts of 
terrorism. For example, an agency in Australia monitors an outcome 
measure concerning the impact of additional security expenditures on a 
facility's risk rating, while controlling for existing security 
enhancements that mitigate the risk, such as the number of guard 
patrols and the adequacy of access control systems (e.g., electronic 
locks). In another example, each business line in one financial 
services organization conducts security compliance reviews of its 
facilities, including confirming the presence of required key security 
equipment and determining whether staff are following security 
policies. Senior security officials review the results to determine 
where problems exist and hold each business manager accountable for 
addressing them. Despite some organizations' use of these measures, 
less than one-quarter of the organizations we contacted had developed 
performance measures for facility protection, and there was widespread 
acknowledgement among the organizations that effectiveness in facility 
protection is challenging to measure. For example, security officials 
do not necessarily know whether a potential security threat or incident 
has been prevented, even after perceived security weaknesses have been 
addressed. Since security is so challenging to measure, some of the 
organizations that we interviewed told us that they rely on U.S. 
federal agencies for support and leadership in developing security 
standards and performance measures, and one foreign government agency 
said it was interested in developing guidance for security performance 
measurement but was looking to U.S. federal agencies for assistance in 
this area. 

We found that some bureaus and services within three of the agencies we 
reviewed--DHS (for GSA properties), USPS, and Interior--are using 
output measures, and, to a lesser extent, outcome measures, while VA 
and some bureaus and services within the other three agencies are not. 
The agencies that have developed performance measures use them to 
evaluate and improve program effectiveness, make risk management 
decisions, and help ensure adequate protection at individual 
facilities. For example, within DHS, FPS has established an output- 
oriented performance measure to monitor the timely deployment of 
security enhancements such as x-ray machines. Such a measure provides a 
basis for FPS to compare planned versus actual performance. Several 
bureaus and services within USPS and Interior have developed 
methodologies to rank and monitor the relative risk ratings of their 
respective facilities over time--these ratings are then used as outcome 
measures for determining the change in the effectiveness of facility 
protection efforts. VA and the bureaus and services that did not have 
security performance measures generate data on ongoing protection 
activities, such as monitoring the numbers and types of security 
breaches at a given facility. This information could provide useful 
feedback about the agency's effectiveness in mitigating building 
security risks and therefore could be used for measuring performance. 
Although agencies have placed an emphasis on performance measurement 
and initiatives are under way, agency security officials said it has 
been challenging to measure the actual impact of various approaches on 
improving security and that resources for measurement initiatives have 
been scarce. Furthermore, while importance has been placed on 
performance measures in national homeland security policies and broad 
guidance exists for measuring the performance of critical 
infrastructure protection programs, agencies have not established 
specific guidance and standards for developing and using performance 
measures for facility protection programs in particular. This differs 
from the information technology security area, where agencies not only 
are required to measure performance, but also have detailed guidance 
and standards for developing and implementing performance measures. 
Without effective performance measurement data, especially data on 
program outcomes, decision makers may have insufficient information to 
evaluate whether the benefits of security investments justify their 
costs, to determine the effectiveness of security activities, to know 
the extent to which security enhancements have improved security or 
reduced federal facilities' vulnerability to acts of terrorism or other 
forms of violence, or to determine funding priorities within and across 
agencies. 

Because ISC was established to enhance the quality and effectiveness of 
security in buildings and facilities in the United States and to 
provide a permanent body to address continuing governmentwide security 
in federal facilities, we are recommending that the Secretary of DHS 
direct ISC to (1) establish guidance and standards for measuring the 
performance of facility protection efforts, particularly for program 
outcomes; (2) communicate the established guidance and standards to 
relevant federal agencies; and (3) ensure that the guidance and 
standards are regularly reviewed and updated. In commenting on a draft 
of this report, DHS, USPS, VA, and Interior generally concurred with 
the findings, and DHS concurred with the recommendations. DHS, USPS, 
and Interior also provided comments, which were incorporated as 
appropriate to ensure accuracy. GSA said they did not have any comments 
on the draft report. 

Background: 

The protection of federal facilities gained importance after the 1995 
bombing of the Alfred P. Murrah Federal Building in Oklahoma City, and 
this issue became even more critical after the 1998 embassy bombings in 
Africa; the September 11, 2001, attacks on the World Trade Center and 
the Pentagon; and the anthrax attacks in the fall of 2001. Shortly 
after the 1995 bombing, the President signed Executive Order 12977, 
establishing the Interagency Security Committee (ISC). ISC--which has 
representation from all major federal departments, agencies, and key 
offices--was charged with enhancing the quality and effectiveness of 
security in, and protection of, nonmilitary facilities occupied by 
federal employees in the United States.[Footnote 9] Furthermore, ISC 
was tasked to serve as a permanent body to address continuing 
governmentwide security issues for federal facilities. Under the order, 
ISC became responsible for developing policies and standards, ensuring 
compliance and overseeing implementation, and sharing and maintaining 
information. Around the same time that ISC was created, the Department 
of Justice categorized all federal facilities into security levels I 
through V based on factors such as facility size and number of 
employees, and it established recommended minimum security standards 
for each of the five levels. These standards covered perimeter, entry, 
and interior security and security planning.[Footnote 10] 

The 2001 terrorist attacks prompted additional policies concerning 
facility protection and a variety of security enhancements at federal 
facilities. The Homeland Security Act of 2002 and a number of national 
strategies, including the National Strategy for Homeland 
Security,[Footnote 11] assigned DHS specific duties associated with 
coordinating the nation's efforts to protect critical infrastructures 
and key assets. Government facilities (at the federal, state, and local 
levels) were identified as key assets and therefore were included in 
this effort.[Footnote 12] Furthermore, the 2002 Act transferred FPS 
from GSA to DHS and, as a result, made DHS responsible for 
ISC.[Footnote 13] A related directive, the Homeland Security 
Presidential Directive Number 7 (HSPD-7), stated that DHS's Secretary 
was responsible for coordinating the overall national effort to 
identify, prioritize, and protect critical infrastructures and key 
assets.[Footnote 14] To meet this responsibility, DHS developed a 
National Infrastructure Protection Plan (NIPP), which is currently in 
draft form. FPS is responsible for implementing the NIPP for the 
government facilities sector. HSPD-7 also required each federal agency 
to develop plans to address identification, prioritization, protection, 
and contingency planning for physical and cyber critical 
infrastructures, along with key assets that they hold or operate. As 
the governmentwide emphasis on protecting critical infrastructures 
mounted, the federal agencies' facility protection efforts continued to 
intensify. In addition to implementing such activities as searching 
vehicles that enter federal facilities, restricting parking, and 
installing concrete bollards, federal agencies also implemented various 
security technologies, such as smart cards for access control. Figure 1 
shows smart card technologies that are utilized at a federal building. 

Figure 1: Smart Card Access Portals at a Federal Building Entrance: 

[See PDF for image] 

Source: GAO. 

[End of figure] 

While it is evident from the policies and strategies outlined above 
that the protection of key assets, including federal facilities, has 
become an important issue for the U.S. government, the protection of 
such assets has also gained attention in state, local, and foreign 
governments, as well as the private sector. State and local governments 
in the United States, for instance, have taken steps to ensure the 
protection of critical infrastructures and key assets within their 
jurisdictions, often receiving resources for such efforts from the 
federal government. For example, DHS's Homeland Security Grant Program 
provides funding to state and local governments to prevent, deter, 
respond to, and recover from acts of terrorism. Funding from this grant 
program can be used for, among other things, critical infrastructure 
protection activities. The protection of critical infrastructures and 
key assets has also gained momentum in foreign governments, 
particularly in countries like the United Kingdom that have recently 
faced terrorist attacks. Furthermore, because many U.S. critical 
infrastructures are owned and operated by the private sector, and 
because some of these infrastructures have been targeted by terrorists 
in the past, many private-sector entities have increased their 
investments in security efforts. 

Due in part to the growing attention to facility protection, we 
designated federal real property as a high-risk area in January 2003 
and have since published a number of reports on this issue.[Footnote 
15] In a November 2004 report, we identified six key practices in 
protecting federal facilities, one of which was measuring performance 
to help achieve broad program goals and to improve security at 
individual facilities. We reported that, for broader program goals, 
performance measures could indicate whether organizations establish 
timelines and adhere to budgets. And, at the individual facility level, 
on-site security assessments and other active testing could provide 
data on the effectiveness of efforts to reduce a facility's 
vulnerability to attack. Training exercises and drills are also useful 
in assessing preparedness.[Footnote 16] 

The need for agencies to measure performance stemmed from the 
Government Performance and Results Act of 1993 (GPRA),[Footnote 17] 
which was intended to improve federal program effectiveness, 
accountability, and service delivery. This act required federal 
agencies to develop strategic plans, link them with outcome-oriented 
goals, and measure agency performance in achieving these goals. 
Likewise, in the security context, a number of national strategies 
called for federal agencies to use performance measures to, among other 
things, assist in the planning and budgeting of protection activities 
for critical infrastructures and key assets. 

We have previously reported that successful performance measures should 
(1) be linked to an agency's mission and goals; (2) be clearly stated; 
(3) have quantifiable targets or other measurable values; (4) be 
reasonably free of significant bias or manipulation that would distort 
the accurate assessment of performance; (5) provide a reliable way to 
assess progress; (6) sufficiently cover a program's core activities; 
(7) have limited overlap with other measures; (8) have balance, or not 
emphasize one or two priorities at the expense of others; and (9) 
address governmentwide priorities.[Footnote 18] 

Managers can use performance measures in a number of ways to improve 
programs and allocate resources more efficiently and effectively. 
Decision makers can use results from performance measurement to 
identify problems or weaknesses in programs, identify factors causing 
the problems, and modify services or processes to try to address 
problems. Conversely, results from performance measurement can be used 
to identify and increase the use of program approaches that are working 
well and to consider alternative processes in areas where goals are not 
met. Separately, performance measures can also be used to identify 
priorities and allocate resources. Decision makers can compare 
performance measure results with program goals and subsequently 
determine where to target resources to improve performance. 
Furthermore, in a risk management process, agencies can use performance 
measurement to assess progress towards meeting homeland security goals. 
The intended effect of assessing such progress, when coupled with other 
aspects of the risk management process, is the reduction of 
risk.[Footnote 19] Finally, when performance information is used to 
reward individuals, these measures can hold individuals accountable for 
certain work activities and related goals and, as a result, create an 
incentive for achieving results. A greater focus on performance results 
can be achieved by creating a cascade from an organization's goals and 
objectives down to the individual performance level. Such alignment 
facilitates the linking of individual performance to organizational 
performance.[Footnote 20] 

Organizations outside of the U.S. Government Use Security Performance 
Measures to Enhance Decision Making and Help Ensure Accountability: 

We found a range of examples of performance measures that organizations 
outside the U.S. government--including private-sector firms, state and 
local governments, and foreign government agencies--used to track the 
number and types of security activities conducted, the quantity of 
security equipment and services delivered, and the outcomes of these 
security efforts.[Footnote 21] Security officials within these 
organizations recognized that performance measures helped them better 
assess how effective they were in protecting against threats to and 
vulnerabilities of their facilities. Organizations then used the 
results of these performance measures to improve security, inform the 
risk management process, make resource allocation decisions, and hold 
security officials and others in the organization accountable for 
security performance. Despite efforts by some organizations to use 
performance measures as an additional decision-making tool, some 
security officials told us that they faced some challenges in 
developing and implementing performance measures. The challenges 
include limited guidance and expertise in the performance measurement 
area. 

Selected Organizations Use a Range of Output, Outcome, and Process/ 
Input Measures to Assess the Effectiveness of Facility Protection 
Efforts: 

Security officials recognized that performance measurement is important 
for improving facility protection and ensuring accountability. They 
also acknowledged that performance measures would allow them to take a 
more strategic, outcome-based approach to managing their security 
programs and to better prepare their facilities against terrorism and 
other threats. However, less than a quarter of the organizations we 
interviewed told us that they have developed and used various 
performance measures for their security programs, and several of those 
that did have performance measures said that the measures are still a 
work in progress. Table 1 provides examples of the output, outcome, and 
process/input measures these organizations have developed. Appendix II 
provides additional examples of performance measures. 

Table 1: Examples of Performance Measures for Facility Protection: 

Type of measure: Output; 
Example: 
* Number of risk assessments performed; 
* Average time to process background screenings; 
* Compliance with security policies; 
* Client/customer satisfaction with security services. 

Type of measure: Outcome; 
Example: 
* Evidence of damage to buildings and facilities; 
* Change in risk rating resulting from countermeasures deployed; 
* Change in the total number of security-related incidents. 

Type of measure: Process/Input; 
Example: 
* Number of security clearances undertaken; 
* Number of training courses and drills conducted; 
* Number of security guards. 

Source: GAO. 

Note: GAO analysis of data from selected state, local, and foreign 
government agencies and private-sector organizations. 

[End of table] 

In some of the organizations we interviewed, some security officials 
use output measures to monitor the direct products and services 
delivered by a program and the characteristics of those outputs, 
including efficiency, cost-effectiveness, timeliness, quality, and 
customer service. Some security officials use outcome measures to 
compare the results of those products and services with the goals 
security officials are trying to achieve, such as reducing the total 
number of security incidents relating to thefts, vandalism, and acts of 
terrorism. In addition, some security officials use outcome measures to 
assess whether their security program is operating efficiently and to 
determine the quality of the services and products they are trying to 
provide. Separately, security officials use various process/input 
measures to provide a descriptive overview of the program activities 
and the resources of their security program, including the types and 
numbers of facilities they manage and the level of 
countermeasures,[Footnote 22] such as entry control security systems, 
they have installed. Input measures are used for resource allocation 
and monitoring and do little to reflect the effectiveness of the 
security program. 

As an additional output measure, some of the organizations we 
interviewed determine whether their security efforts comply with their 
security policies, standards, and guidance. For example, some of the 
government agencies in the three foreign countries we visited use 
performance measures to evaluate whether their security activities are 
compliant with their government's protective security policies. Several 
security officials in these agencies told us that they use this measure 
to demonstrate compliance with established government standards. Some 
of these foreign government agencies indicated that they measure 
compliance based on the results of security audits completed 
internally--by the security department or other departments within the 
organization--or externally. Some of these security officials then use 
the results of the audits to identify security weaknesses and make 
corrections to improve security. Other foreign government agencies use 
surveys to measure the degree of security policy compliance. For 
example, Australian government agencies are required to adhere to the 
minimum protective security standards contained in the Australian 
government's Protective Security Manual.[Footnote 23] Ministers and 
agency heads are accountable for their agency's compliance with these 
standards. Agencies are surveyed annually for compliance with the 
security manual standards. The survey results are assessed and reported 
to the central government. 

Some of the nonfederal organizations we interviewed also measure the 
effectiveness of their countermeasures by determining whether the 
services and security equipment they provide are adequate under both 
real and simulated conditions. Some of the organizations we interviewed 
stated that they test security equipment, such as perimeter alarms and 
x-ray machines, and conduct simulated attacks and penetration exercises 
on a periodic basis. One official from the gaming industry said that it 
is important to test equipment to ensure it is being used properly, 
because the technology itself is not as important as how it is used. 
For example, a facility could have a sophisticated card access system 
but still be vulnerable if someone props the door open. To help 
government agencies select effective security equipment, a central 
agency in the United Kingdom tests security equipment and provides 
those in the security community with information to help the user match 
the appropriate equipment to the requirement. Similarly, an agency in 
Australia conducts tests on security equipment and provides agencies 
with a catalog of approved products. Security officials from the gaming 
industry also told us that they are members of an external group that 
tests security equipment and shares the results of the testing with 
security officials in other industries, such as the chemical, 
petrochemical, and pharmaceutical industries. 

In some organizations, the selection of useful performance measures has 
evolved through a trial-and-error process. For example, one financial 
services organization went through several iterations of its security 
performance measures over a 1-1/2 year period in order to determine 
which performance measures were important to monitor and would provide 
them the right information needed to achieve the organization's 
security objectives. For example, they initially reported on the number 
of security alarms, and then changed the measure to a more useful 
measure--the number of alarms with unique responses (i.e., alarms that 
required a guard to respond in person)--so that they could better 
understand how security staff were interacting with the security 
equipment. One security official acknowledged that, although they were 
satisfied with their current performance measures, it would still be 
helpful to measure performance in other areas, such as employee 
satisfaction with security services. 

Case Example: A Financial Service Organization's Performance Measures: 

Security officials at a large, well-known financial services 
organization use a number of output and outcome measures to regularly 
monitor the performance of their security program. In addition, they 
use process/input measures to assist them with resource allocation 
decisions. The security officials emphasized that there is a constant 
need to measure and evaluate what their security program does in order 
to educate business professionals on the importance of a security 
investment. While the organization assesses all of its facilities using 
a baseline set of security standards and risk assessments, performance 
measures provide security officials with information to understand 
whether these standards and risk assessments are actually improving 
their security situation. The security officials told us that they use 
the following performance measures: 

* Outputs--Security officials use output measures relating to their 
operational readiness (i.e., how prepared the security program is 
against potential security threats), which includes the number of risk 
assessments performed. They also measure the number of non-security 
related incidents such as false alarms or broken security cameras. In 
addition, security officials monitor the number of policy exceptions 
that exist when a business line or facility cannot comply with the 
standards set forth in their security policy manual. If many exceptions 
to a particular section of the policy manual occur in a given month, a 
policy working group reviews the issue and determines whether 
additional assistance will be required to bring the facilities into 
compliance. 

* Outcomes--One outcome measure is the monetary savings resulting from 
less costly, more efficient security processes and new technologies. 
Security officials use this outcome measure to demonstrate savings from 
the security program's budget as well as from the budgets external to 
the security division, such as operations. Officials are also able to 
prorate contract-related savings over the lifetime of the contract to 
better understand how the savings affect the organization over time. To 
understand the effectiveness of their security efforts, security 
officials use data on the responses to security incidents, which are 
classified by type (e.g., assault, burglary, terrorism). Security 
officials then analyze the data to help them make recommendations for 
additional security improvements. 

* Process/Input--The financial organization tracks guard levels, 
security expenditures, and security activities across all its 
facilities. Security officials use these measures to compare the 
different levels of service required, given the risk associated at each 
facility or region. In a given month, they also measure the number of 
training sessions and drills conducted. The performance measure for 
training identifies the specialized fields in which the security staff 
are being trained and the type of training the security staff are 
providing to others. 

Security officials at this financial services organization told us that 
they monitor their performance measures on a monthly basis, and that 
the data are aggregated for the entire organization and also broken out 
by region. They developed, and have continued to modify, their 
performance measures based on the analysis of incidents and other 
activities in a particular region as well as trends across regional 
facilities. They also obtained feedback from regional offices and from 
their own security staff. Security officials noted that they tried to 
select performance measures that represented common threads and were 
not biased in favor of one particular region. They also continuously 
evaluate the usefulness of their performance measures, adding a measure 
if they determine that information is needed on a particular subject or 
dropping a measure if it does not seem to be informative. 

Security Officials Use Performance Measure Results for Risk Management 
and Resource Allocation: 

We have previously reported that organizations can use the results of 
performance measures to make various types of management decisions to 
improve programs.[Footnote 24] Security professionals also recognize 
the benefits of using performance measurement within the security 
industry. At a major security industry conference in 2005, a conference 
presenter indicated that the ability to compare past performance and 
the performance of others contributes to the goal of continuous 
improvement, the result of which is a stronger, more mature security 
program with security processes that can better protect facilities and 
staff from harm. Performance measures also provide management with the 
tools to verify that the organization's resources are used responsibly 
and security risks are managed appropriately. 

In some of the organizations we interviewed, security officials and 
other decision makers use performance measures to manage risk, allocate 
resources, and improve the quality of the security services they 
provide for their facilities. For example, at one financial services 
organization, security officials installed protective security 
equipment at some of their facilities and then compared the number of 
security incidents and the level of customer satisfaction before and 
after the equipment was installed. In this particular case, security 
officials used this performance measurement data to demonstrate the 
value of that security investment to their corporate management and the 
business lines they supported. The performance measures also allowed 
security officials to compare individual facility performance to the 
average within the industry, which they use to demonstrate the risk 
level of a particular facility and take appropriate action to address 
the risk. 

Where security goals and objectives were not achieved, some security 
officials also used performance measurement results to identify problem 
areas and take corrective action. Several organizations mentioned that 
they measure the quality of their security efforts through an output 
measure by soliciting feedback from employees and clients through 
customer satisfaction surveys. For instance, one Canadian organization 
periodically surveys clients about their satisfaction with the security 
services the organization provides to government agencies. The survey 
questions range from how often the client saw security managers to how 
satisfied they were with the services they received. The responses to 
the surveys provide feedback that allows security officials to improve 
their provision of security services to both private and public sector 
clients. 

Case Example: An Australian Agency's Risk Model: 

Performance measures helped security officials in one government agency 
in Australia become better risk managers and allocate resources more 
efficiently across facilities. The agency uses a security plan that 
includes security objectives that are linked to its strategic goals. 
The plan also lists strategies and actions for achieving these 
objectives, along with performance measures that assess the extent to 
which objectives are being achieved. For example, the performance 
measures monitor the extent to which security practices are in 
accordance with the agency's security policies, any evidence of harm to 
agency staff or facilities, and the extent to which agency stakeholders 
view the agency's facilities as safe for their resources and assets. To 
monitor performance, security officials use two different review 
processes. First, security officials can access the audit function of a 
computer-based risk assessment model to monitor the outcomes of the 
performance measures contained in their security plan and to understand 
how well their security efforts are performing within individual 
facilities. For example, the risk-assessment model allows security 
officials to monitor the impact of additional security expenditures on 
a facility's risk rating while controlling for existing security 
enhancements that mitigate the risk, such as the number of guard 
patrols and the adequacy of access control systems (e.g., electronic 
locks). Security officials can then use the results to justify spending 
decisions and prioritize security investments. For example, one 
facility requested a perimeter fence, and security officials were able 
to use the risk-assessment model to demonstrate that the facility's 
risk was adequately managed without the fence since there were no known 
risks in that location and since the facility already had guards and an 
alarm system. Second, the agency's audit unit also conducts its own 
independent measurement of the security activities so that security 
officials can compare across facilities to guide them in determining 
where they need to make adjustments. Together, these two security 
reviews provide the security program with enough information to assess 
their security position, according to one agency security official. 

Performance Measures Can Be Used to Hold Security Officials Accountable 
for Achieving Goals and Results: 

Security officials recognized the value of performance measures to help 
ensure the accountability of security officials, management, and other 
employees throughout the organization. Many of the organizations we 
interviewed had security policies and procedures in place, and some of 
these organizations were able to link these plans directly to 
performance measures that demonstrated achievement of both the security-
related strategic goals and the organization's broader strategic goals. 
We have previously reported that aligning the goals at the executive 
level with the goals and objectives at each operational level 
reinforces the connection between strategic goals and the day-to- day 
activities of managers and staff.[Footnote 25] For example, an annual 
goal that is linked to a program and also to a long-term goal can be 
used to hold agencies and program offices accountable for achieving 
those goals.[Footnote 26] Furthermore, we reported that such alignment 
increases the usefulness of performance information to decision makers 
at each level.[Footnote 27] 

Case Example: The District of Columbia's Alignment of Security Goals 
and Measures: 

One agency within the District of Columbia (D.C.) government uses 
performance measures and targets to hold agency management and security 
officials responsible for its security-related activities. D.C.'s 
Office of Property Management is responsible for D.C. government 
buildings, and the Protective Services Division, which falls under 
Property Management, is responsible for security at these buildings. 
Protective Services faces a unique environment in protecting the 
facilities that it is responsible for because of the proximity of these 
assets to federal facilities, which are considered to be attractive 
targets for terrorist attacks. To help ensure that their security 
concerns are addressed, security officials in Protective Services noted 
that they have linked their security goals and related performance 
measures with the Property Management's goals and citywide strategic 
goals (see fig. 2). Specifically, Protective Services' goals, 
performance measures, and related targets support the goal of Property 
Management to provide a high-quality work environment and user-friendly 
facilities, and also support the broader citywide strategic goal of 
making government work. The security officials pointed out that this 
alignment is very deliberate and can help hold officials accountable 
for a security-related activity. For example, the Director of Property 
Management can use security-related performance measures and 
corresponding targets to hold the Protective Services Division 
accountable for its activity. If Protective Services does not meet the 
targets, it is required to submit justifications to senior management 
as to why they were not met. The officials explained, however, that in 
situations where there are unforeseen circumstances, their targets can 
be realigned, with the consent of senior management. For example, 
following Hurricane Katrina, Protective Services was required to 
provide security services for Katrina victims housed at a D.C. arena. 
The human resources required for this task made it impossible for 
Protective Services to meet all the targets, and the D.C. mayor's 
office allowed for adjustments to the target for that time. Separately, 
the mayor's office can also use the security-related performance 
measures and targets in conjunction with other Property Management 
performance measures and targets to monitor the work of the entire 
agency and hold the Director of Property Management accountable for 
agencywide activity. 

Figure 2: Linkages between District of Columbia Strategic Goals and 
Performance Measures for Facility Protection: 

[See PDF for image] 

Source: GAO analysis of District of Columbia data. 

[End of figure] 

Departmental Security Officers and Individual Accountability: 

We also recognized in a previous report that the establishment of a 
chief security officer position is essential in organizations that own 
and operate large numbers of mission-critical facilities.[Footnote 28] 
Such a high-level position is important for coordinating security 
responsibilities across facilities and ensuring accountability for 
security results, including establishing linkages between security 
performance and outcomes. We found that government agencies in all 
three countries we visited are required to designate a departmental 
security officer (DSO) or an agency security executive to oversee 
security matters across all agency facilities and implement government 
security policies. For example, in the United Kingdom, security 
officials told us that the DSOs are sufficiently senior within each 
agency department to have an effective voice and to put security issues 
on the management agenda. These security officials also told us that 
the DSOs are playing a greater role in coordinating with other agency 
departments to enhance their security. The financial services and 
gaming organizations we interviewed also have directors or vice- 
presidents of security who have a direct line of communication to their 
corporate management. They said that this arrangement promotes a good 
working relationship with management and allows them to identify and 
fix security problems efficiently. 

Some of the organizations we interviewed also used performance measures 
to hold security officials accountable for program performance. For 
example, some organizations hold their security officials accountable 
for results through the use of customer satisfaction surveys. Security 
officials at one financial services organization indicated that they 
conduct quality surveys with their business-line clients, which allows 
clients to provide input to security officials on whether the security 
program is effective and whether the security program met the client's 
expectations. 

Case Example: Individual Accountability in Two Financial Services 
Organizations: 

Two major financial services organizations we interviewed use 
performance measures to help ensure accountability for investments in 
security improvements and compliance with security policies and 
regulations. Security officials in one financial services organization 
told us that they work in a security culture that is very performance 
driven. While their security budget is fully separate from other 
corporate expenditures, regional security directors are responsible for 
determining how to spend security funds. Regional security directors 
use performance measures to justify security expenditures to all of the 
individual business lines they support and to demonstrate a return on 
investment for their security expenditures. For example, the 
organization uses output and outcome performance measures to monitor 
monetary savings, the number of security incidents, and the impact of 
new technologies and processes. When security officials want to invest 
in a new security technology, they use these performance measures to 
demonstrate to the business lines that they have investigated all of 
the alternatives and determined the cost and potential savings of the 
purchase. For example, they used past data on the cost and performance 
of security equipment and guards to calculate the cost of installing 
some security equipment versus hiring a security guard to protect one 
of its facilities. They were able to demonstrate that the security 
equipment would be more cost-efficient over time and be more effective 
in deterring certain crimes. 

Another financial services organization uses performance measures to 
help ensure that all facilities are complying with its security 
policies and regulations. The security policies for each of the 
organization's business lines differ based on their level of risk. As a 
form of quality control for its security operations, each business line 
is expected to conduct compliance reviews of all of its facilities, 
including confirming the presence of required key security equipment 
and determining whether staff members are following security policies. 
Each business manager is held accountable for the results of these 
reviews: senior security officials receive and review monthly 
compliance reports, and the financial services organization's central 
audit department ensures that the reviews were properly conducted. 
According to security officials, the data in the monthly reports are 
used to determine where problems exist and look for emerging security 
trends. 

Case Example: An Australian Agency's Security Certification Process: 

One Australian government agency uses performance measures to hold its 
security executives accountable for identifying and addressing security 
risks. Officials from the agency noted that they have historically had 
a strong security and risk management culture that emphasizes executive 
accountability for performance. The agency holds its security 
executives accountable by requiring them to produce a certificate of 
assurance that includes physical and personal security. The purpose of 
the certificate, which is signed by a senior agency executive, is to 
assure the chief executive that the agency is meeting its security 
obligations, and that action plans are in place to address any 
problems. It covers compliance with external requirements, including 
government regulations, and internal conformance with corporate 
security policies. The assurances given must be underpinned by 
evidence, which includes the results of physical security reviews that 
are conducted periodically at each facility. These reviews measure and 
report on the standard of physical security, including perimeter 
security, access control, alarm systems, and surveillance and 
communication systems. The certificate uses a color code to indicate 
the overall status of the security function--red, amber, or green. 
Certificates rated red or amber are reviewed and resubmitted every 6 
months. Green certificates are reviewed annually. If the certificate 
identifies a security problem, it must be accompanied with an action 
plan for addressing the risks involved. 

Organizations Cited Challenges in Developing and Using Performance 
Measures: 

Although performance measurement is seen as an important tool for 
assessing the effectiveness of security programs, developing and using 
performance measures can be challenging, according to security 
officials we interviewed at selected organizations. A difficulty with 
developing performance measures is determining whether the measures 
that are used are suitable, given a constantly changing threat 
environment. Some security officials said that it was difficult to know 
what to measure because security is intangible or difficult to 
quantify. Others also acknowledged that it is difficult to determine 
whether a potential security threat or incident has been prevented, 
even after additional countermeasures or security staff are introduced 
to address perceived security weaknesses, because deterrence is 
immeasurable. Several security officials cited the difficulty in 
determining a causal relationship between security efforts and changes 
in the number of security incidents. For example, a security official 
from an Australian government agency indicated that an increase in the 
number of breaches in a particular facility may result because an 
organization is being targeted at that particular point in time rather 
than because it lacks adequate security measures. Organizations also 
find it hard to measure the impact of some security actions, such as 
the potential financial savings resulting from attacks that have been 
discouraged. Organizations told us that they recognize the need to draw 
linkages between security incidents and security investments, but some 
organizations find it difficult to measure the benefit of a particular 
security process or piece of equipment in the absence of a security 
breach. 

A number of organizations also told us that other priorities and 
considerations might hinder their ability to effectively use 
performance measures for making security decisions. Some security 
officials pointed out that the ultimate decision on how to allocate 
security resources can be based on priorities other than performance. 
For example, several private sector and foreign government agencies we 
interviewed noted that they have to balance their security needs with 
their goals of maintaining sufficient public access to their 
facilities. Some security officials are also reluctant to use 
performance measures because they do not want to be held accountable 
for not meeting their performance targets. Several organizations 
mentioned that potential liability could be seen as a disincentive for 
using performance measurement data, because an organization may be seen 
as negligent if the performance data were to show that an organization 
could have done something to prevent an incident but chose not to. One 
security official told us that having established performance targets 
could also discourage organizations from accurately collecting data 
because security officials may be reluctant to report an incident if a 
decline in the number of incidents is one of the performance goals. 

Some organizations we interviewed cited the lack of knowledge and 
expertise available to collect and analyze security data as a 
limitation to overcoming some of the challenges of using performance 
measures. One financial services organization indicated that some of 
its security officials did not see the benefits of using performance 
measures until after they saw that their business line managers 
responded favorably to the use of performance measures to demonstrate a 
return on investment for security expenditures. Several state, local, 
and foreign government agency officials noted that they had limited 
management staff available to develop and monitor performance measures 
for physical security. According to one state government agency 
official, without staff expertise in this area, security staff tend to 
approach security initiatives like a project--they monitor the 
initiative's progress to make sure that it is delivered on time and on 
budget, but they do not necessarily measure the effectiveness of the 
security equipment once it is installed. 

Many organizations we interviewed said that they face the 
aforementioned challenges, and we noted that some of the entities 
outside the U.S. government rely on U.S. agencies for support and 
leadership in developing security standards and performance measures. 
One state government agency we interviewed expressed an interest in 
developing performance measures in the future and mentioned that it 
often looks to the federal government for guidance on security efforts. 
DHS officials told us that their agency was providing assistance to 
several foreign government agencies in the United Kingdom in measuring 
performance and allocating security resources. One foreign government 
agency said that it was interested in developing governmentwide 
guidance for measuring security performance but was looking to U.S. 
agencies for assistance in this area. 

U.S. Agencies Have Made Some Progress in Developing and Using 
Performance Measures for Facility Protection Programs, but Lack 
Guidance and Standards: 

Responding to the requirements in 2002 by the National Homeland 
Security Strategy and subsequent federal policies, agencies have paid 
greater attention to facility protection and have begun using key 
practices--such as performance measurement--to varying degrees. Agency 
officials noted that developing performance measures for facility 
protection was a difficult undertaking, since the results are not 
always readily observable. We found that some bureaus and services 
within three of the agencies we reviewed--DHS, USPS, and Interior--are 
using output measures and, to a lesser extent, outcome measures, while 
the VA and some bureaus and services within the other three agencies 
are not. Despite the lack of security performance measures, we found 
that ongoing protection activities within these bureaus and services 
and the VA, such as monitoring the numbers and types of security 
breaches at a given facility, generate a wealth of data that could 
provide useful feedback about the agency's effectiveness in mitigating 
building security risks, and therefore could be used as measures of 
performance. While the agencies have demonstrated some progress in 
applying performance measurement to facility protection, with limited 
precedent for how to do this, more work remains to identify measures-- 
particularly outcome measures--that assess the impact of facility 
protection efforts. Output measures do not provide an indication of 
what security activities are accomplishing, while outcome measures that 
are clearly tied to results indicate the extent of progress made and 
help identify the security gaps that still remain. Officials expressed 
concerns about the lack of resources and the limitations of existing 
guidance in providing direction about how to measure progress and 
evaluate the effectiveness of physical security programs. 

Agencies Use Output Measures and Some Outcome Measures to Inform Risk 
Management, Help Ensure Adequate Protection, and Assess Effectiveness 
of Facility Protection Efforts: 

In general, the agencies we reviewed have made some progress in 
collecting and using performance-related data for their facility 
protection program activities, but many of the measures are of program 
outputs rather than outcomes. While output measures are an important 
part of performance measurement, outcome measures could provide 
information to evaluate whether the benefits of security investments 
outweigh their costs and to determine the effectiveness of security 
activities. The agencies we reviewed use output measures, such as the 
timely completion of risk assessments and whether countermeasures work 
as intended once deployed, to inform risk management decisions and to 
help ensure adequate protection at the individual facility. 
Additionally, several bureaus and services within DHS, USPS, and 
Interior have developed outcome measures to rank and monitor the 
relative risk ratings of their respective facilities over time or to 
otherwise assess the effectiveness of their facility protection 
efforts. 

Case Example: DHS's Federal Protective Service in GSA Facilities: 

The effectiveness of security programs at GSA facilities is evaluated 
using performance measures developed by the Federal Protective Service 
(FPS) and a physical security testing program developed by GSA. FPS has 
identified four performance measures--both output and outcome measures-
-to assess its efforts to reduce or mitigate building security risks. 
These four performance measures, detailed in table 2, are at varying 
stages of implementation and are still evolving. Under the Homeland 
Security Act of 2002, DHS, through FPS, is directly responsible for law 
enforcement and security-related functions at facilities under GSA's 
control or custody. FPS delivers security and law enforcement services 
for approximately 8,000 facilities that fall under GSA's jurisdiction. 

Table 2: FPS's Performance Measures for Facility Protection: 

Type of measure: Output; 
Performance measure: Timely deployment of countermeasures; 
Purpose: To compare actual deployment dates with planned deployment 
dates. 

Type of measure: Output;
 Performance measure: Countermeasure functionality (e.g., surveillance 
cameras, x-ray machines); 
Purpose: To gauge whether those security countermeasures for which FPS 
is contractually responsible are working as intended, once deployed. 

Type of measure: Output; 
Performance measure: Patrol and response time; 
Purpose: To assess FPS's ability to respond to calls for service within 
certain time limit goals. 

Type of measure: Outcome; 
Performance measure: Facility security index; 
Purpose: To calculate FPS's average success rate for the above three 
performance measures. 

Source: GAO. 

Note: GAO analysis of FPS data. 

[End of table] 

The first measure--monitoring the deployment of countermeasures-- 
focuses on the timeliness of implementation and serves as a measure of 
program output. Once approval and funding have been obtained to 
implement a recommended countermeasure, FPS personnel record planned 
deployment dates so that they can compare them with actual 
implementation dates. An FPS working group decided that the initial 
baseline for this measure, developed in fiscal year 2005, would be 90- 
percent success, which is calculated as the number of countermeasures 
actually deployed by the scheduled due date, divided by the number 
planned. FPS officials noted that they will not know how well they are 
progressing on this measure until the end of fiscal year 2006 because 
they are still automating the process and training regional staff. For 
fiscal year 2007 and subsequent years, FPS expects the annual goal to 
be some increment above the preceding year's results until the long- 
term goal of 98 percent is achieved and maintained. 

Countermeasure functionality, FPS's second measure, gauges whether a 
countermeasure works as intended once it is deployed. Specifically, it 
assesses the operational capability of five major groups of 
countermeasures for which FPS is contractually responsible: closed 
circuit television surveillance, contract security guards, intrusion 
detection systems, magnetometers, and x-ray machines. In some 
instances, contract guards are routinely evaluated to determine whether 
they are performing effectively. Performance includes the guards' 
knowledge of and compliance with relevant operations for their security 
post. Based on FPS testing results in fiscal year 2005, the baseline 
for this measure is 90-percent success, which is calculated as the 
number of countermeasures working and performing as intended divided by 
the number tested. According to FPS officials, FPS currently has about 
a 92-percent success rate for this measure. The long-term goal for this 
measure is 100-percent effectiveness. Related to facility protection, 
this output measure reflects the functionality of a program element, 
but not its effect. 

Patrol and response, the third measure, assesses FPS's ability to 
respond to calls for service within certain time limit goals. The 
initial baseline for this measure was established in October 2005 and 
was about 17.5 minutes. This baseline represents an average response 
time for all of FPS's 11 regions, and is calculated using dispatch and 
arrival time information from FPS's online incident reporting system. 
The time parameters for data collection fell between FPS's core duty 
hours of 6:00 a.m. and 6:00 p.m. The goal for this measure is to reduce 
response times by 10 percent, although FPS noted that this goal could 
increase or decrease depending on staffing levels or deployments. At 
the time of this report, FPS noted that they have collected statistics 
on response times for this measure and are in the process of evaluating 
whether they have achieved their goal. 

Finally, the facility security index--an outcome measure[Footnote 29]-
-calculates the overall effectiveness of FPS operations in meeting the 
performance goals of the three output measures described above (timely 
deployment of countermeasures, countermeasure functionality, and patrol 
and response time). An index score of 1 indicates that FPS has met its 
performance goals, a score of greater than 1 indicates that FPS has 
exceeded the goals, and a score of less than 1 indicates that it has 
not met the goals. 

Taken together, these four FPS performance measures provide insight 
into activities designed to support FPS's efforts to prevent and 
respond to security and criminal incidents, including terrorist 
threats. In addition to assessing FPS's performance in fulfilling its 
facility protection responsibilities, the measures also serve as a 
baseline for making decisions about deploying existing resources or 
requesting additional resources. FPS officials told us that these 
measures are derived from strategic objectives established by DHS's 
Immigration and Customs Enforcement (ICE), of which FPS is a component. 
These objectives include implementing appropriate countermeasures to 
reduce vulnerabilities facing buildings under GSA's jurisdiction (see 
fig. 3). Aligning facility protection performance measures and targets 
with broader DHS and ICE mission, goals, and objectives helps hold 
employees accountable for security activity and allows them to observe 
how day-to-day security activities contribute to the broader mission, 
goals, and objectives. Similar to organizations outside the federal 
government, FPS provides its financial management staff with quarterly 
and annual reports that document the accomplishments for each measure 
in order to support planning and budgeting efforts included in DHS's 
Future Years Homeland Security Program document.[Footnote 30] 

Figure 3: Linkages between DHS Mission and FPS Performance Measures for 
Facility Protection: 

[See PDF for image] 

Source: GAO analysis of DHS data. 

[End of figure] 

It is important to note that when FPS was a part of GSA, we reported on 
GSA's lack of performance goals and measures for its building security 
program. In June 1998, we testified that GSA had not established key 
program evaluation mechanisms for its building security program that 
could help determine how effective its security program has been in 
reducing or mitigating building security risks or in shaping new 
security programs.[Footnote 31] At the time, we reported on features 
that would support program evaluation, including: (1) developing 
specific goals, outcomes, and performance indicators for the security 
program, such as reducing the number of unauthorized entries; (2) 
establishing and implementing systematic security program evaluations 
that provide feedback on how well the security program is achieving its 
objectives and contributing to GSA's strategic goals; and (3) ensuring 
that a reliable performance data information system is in place. While 
we found that GSA had established goals and measures for its security 
program both apart from and in connection with GPRA, we noted that 
these goals and measures were output oriented and did not address the 
outcomes or results the building security program was expected to 
achieve. Consequently, we recommended that GSA develop outcome-oriented 
goals and measures for its building security program. As previously 
noted, FPS has demonstrated some progress in moving beyond the use of 
output measures that monitor program activity in carrying out its 
responsibilities within the Department of Homeland Security (DHS). 

In addition to FPS's performance measures for assessing the security of 
properties under GSA's control, GSA's Office of the Chief Architect 
also has a program for testing the physical security of GSA buildings. 
Under this program, GSA performs explosive testing of various window 
systems; identifies gaps in protective design and security 
technologies; and provides criteria and tools for blast resistant 
design, progressive collapse in new and existing facilities, and 
upgrading walls to reduce fragmentation and hazards resulting from an 
explosion, among other things. The program team is also developing a 
tool to identify gaps in security planning, ensure consistency with GSA 
policies and ISC's security design criteria, and provide a consistent 
foundation and methodology for making security design decisions. 

Case Example: Interior's Bureau of Reclamation and National Park 
Service: 

One bureau within Interior--the Bureau of Reclamation (BOR)--has 
identified performance measures for its facility protection programs, 
while the National Park Service (Park Service) generates information 
that could be used to monitor the effectiveness of its physical 
security efforts. Each of Interior's eight bureaus independently 
manages the protection program for the facilities that fall under its 
respective purview, and each bureau has developed broad security goals 
derived from the agency's overall mission.[Footnote 32] In general, 
Interior's program evaluation methods are based on GPRA and the Office 
of Management and Budget's (OMB) Program Assessment Rating Tool 
(PART).[Footnote 33] Several of the bureaus have had their programs 
reviewed under the PART system, and some security performance measures 
were identified as part of this effort. Over time, Interior intends to 
have all of its law enforcement programs assessed under the PART 
system. However, an agency official from the Park Service reported 
difficulty in developing formal performance measures because GPRA is 
directed toward evaluating federal programs and does not provide 
guidance on developing goals and measures specifically for security 
activities. 

Within Interior, BOR has an important role in protecting critical 
infrastructures because of its responsibilities related to dams. BOR is 
responsible for managing and protecting well-known assets such as 
Hoover Dam in Arizona and Nevada, which receives approximately 1 
million paying visitors each year. In 2005, the security program 
administered by BOR was selected for review under the PART system. To 
demonstrate its progress in meeting the long-term goal of reducing 
security-related risks at its critical facilities, BOR developed 
several output and outcome performance measures, including (1) timely 
completion of risk assessments, (2) the cost per active background 
investigation, (3) the percentage of recommendations that have been 
implemented based on the results of risk assessments, (4) the number of 
updated regional threat assessments, and (5) changes in the risk 
ratings as countermeasures are implemented for an individual asset (see 
table 3). Although these measures were developed for the protection of 
dams and related facilities, they could be applied to building security 
because there is some similarity in the protection activities. In all 
but one instance, BOR had achieved or exceeded its performance target 
for each measure established for fiscal year 2005. According to OMB's 
PART assessment, BOR's facility protection program was rated moderately 
effective and its performance measures were described as creative and 
useful measures that will help monitor program accomplishments and 
efficiency.[Footnote 34] 

Table 3: BOR's Performance Measures for Facility Protection: 

Type of measure: Output; 
Performance measure: Timely completion of risk assessments; 
Purpose: To compare actual completion dates with planned completion 
dates. 

Type of measure: Output; 
Performance measure: Cost per active background investigation file; 
Purpose: To monitor the cost efficiency of the personnel security 
program, including processing of background investigations, issuance 
and verification of clearances, and case file maintenance. 

Type of measure: Output; 
Performance measure: Status of recommendations designed to mitigate 
risk; 
Purpose: To indicate the percentage of recommended security 
enhancements that have been funded and implemented, and are 
operational. 

Type of measure: Output; 
Performance measure: Number of updated regional threat assessments; 
Purpose: To assess the frequency with which assessments are conducted 
and help ensure that current threat intelligence is incorporated as 
part of risk assessments and risk- reduction strategies. 

Type of measure: Outcome; 
Performance measure: Change in risk ratings; 
Purpose: To assess the risk-reduction benefits associated with 
implementing countermeasures at an individual asset. 

Source: GAO. 

Note: GAO analysis of BOR data. 

[End of table] 

The Park Service is responsible for managing and protecting some of the 
nation's most treasured icons, including the Washington Monument, the 
Lincoln and Jefferson Memorials, and the Statue of Liberty. The Park 
Service manages more than 375 park units, covering more than 84 million 
acres, which provide recreational and educational opportunities and 
numerous other benefits to millions of visitors each year. From 2001 to 
2005, park units averaged a total of about 274 million recreation 
visits per year. While a Park Service official stated that they did not 
have any formal performance measures for facility protection, we found 
that their risk management methodology provides useful feedback about 
the bureau's effectiveness in reducing or mitigating security risks for 
facilities under its jurisdiction. In June 2005, we reported that 
Interior had made significant progress in the risk assessment area, in 
large part due to its new National Monuments and Icons Assessment 
Methodology (NM&I).[Footnote 35] NM&I--a uniform risk assessment and 
ranking methodology--is specifically designed to quantify risk, 
identify needed countermeasures, and measure risk-reduction benefits at 
icon and monument assets. According to an Interior official, Interior's 
Office of Law Enforcement and Security (OLES) developed NM&I to assist 
bureaus in quantifying risk levels and identifying needed security 
enhancements, initially at critical infrastructures and key assets, but 
eventually at all departmental facilities. The NM&I methodology has a 
consequence assessment phase and a risk assessment phase. First, during 
the consequence assessment phase, senior officials from the Park 
Service and OLES determine which icons are considered nationally 
significant.[Footnote 36] Specific attack scenarios--such as chemical/ 
biological, aircraft, or improvised explosive device--are used to 
evaluate security at each asset and score attack consequences.[Footnote 
37] During the risk assessment phase, a group of security professionals 
from the Park Service and OLES, assisted by the site security 
supervisor and the site manager, collectively determine the 
effectiveness of existing security systems using DHS guidelines. Using 
risk values calculated from this evaluation, OLES assigns asset risk 
ratings of high, medium, or low, and specific mitigation 
recommendations are formulated. As part of its annual review, OLES 
routinely monitors the security enhancements that have been implemented 
to reduce the risk rating designations. OLES has not had formal 
performance measures and targets for reducing risk ratings in the past. 
However, in April 2006, according to Interior officials, OLES developed 
and submitted for inclusion in the departmental strategic plan 
performance measures related to the reduction in the percentage of 
physical security vulnerabilities identified at departmental 
facilities. If adopted, such outcome measures could provide valuable 
feedback about the Park Service's progress and overall effectiveness in 
protecting its physical assets. 

Case Example: USPS Inspection Service: 

The USPS Inspection Service utilizes an outcome-oriented performance 
measure to help ensure that it is progressing towards its strategic 
goal. USPS has over 38,000 facilities nationwide that collectively 
handle about 700 million pieces of mail every day, and the agency 
serves over 7.5 million customers daily in its post offices. Postal 
facilities are a compelling target for criminal and terrorist attacks, 
as evidenced by the anthrax attacks in 2001, which put at risk the 
integrity of the mail and the safety of USPS's employees, customers, 
and assets. Within USPS, the Inspection Service--an investigative 
branch whose mission is to protect the nation's mail system and its 
critical assets (i.e., employees, customers, and facilities)-- 
established its first performance measure related to facility 
protection: the percentage of facilities that have high-risk ratings 
(see table 4).[Footnote 38] This outcome measure allows the Inspection 
Service to monitor progress toward achieving its strategic goal of 
ensuring a safe, secure, and drug-free environment. 

Table 4: Inspection Service's Performance Measure for Facility 
Protection: 

Type of measure: Outcome; 
Performance measure: Percentage of USPS facilities with high-risk 
ratings; 
Purpose: To monitor the effectiveness of countermeasures through the 
percentage of USPS facilities that score more than 800 points. 

Source: GAO. 

Note: GAO analysis of USPS data. 

[End of table] 

Specifically, this effort involves annual security surveys of 
facilities conducted by facility protection control officers, as well 
as periodic comprehensive reviews of larger core postal facilities 
performed by the Inspection Service. The data from these surveys and 
reviews are maintained in a database and used by the Inspection Service 
to tabulate a risk score based on USPS's Facility Risk Rating Model. 
Several data elements are considered to compute the composite risk 
score for a given facility, including: 

* crime statistics; 

* building characteristics (e.g., the absence or presence of customer 
parking, whether the facility is attached to an adjoining structure); 

* location information (e.g., the number of federal buildings within a 
1-mile radius of the post office); 

* operational policies and procedures (e.g., the absence or presence of 
policies related to visitors, the timely completion of the facility 
security survey within the last 12 months); and: 

* countermeasures (e.g., the absence or presence of closed circuit 
television surveillance cameras). 

Using these data elements, the maximum risk score that can be computed 
for a facility is 2,854 points. After each element at a particular 
facility is assigned a risk score, the system ranks the facilities 
according to the designated composite risk score. The scoring and 
ranking system is national and is applied to all USPS facilities, which 
allows officials to compare facilities across the country using 
standardized data to identify which buildings are at the highest risk 
level. Facilities with scores at or above the threshold score of 800 
are considered to be high-risk.[Footnote 39] The Inspection Service 
reassesses its facilities every 3 years or when a facility undergoes 
any major renovations or expansions. However, if a facility receives a 
high-risk score, the facility can be reassessed more often to help 
ensure that countermeasures are effective and that USPS has lowered the 
security risks. For example, if a facility received a high-risk score 
in fiscal year 2005, the Inspection Service will revisit that facility 
again in fiscal year 2006 to try to lower the risk score. The target is 
to reduce facility risk scores for 90 percent of the facilities that 
have a high-risk designation. At the time of our review, USPS was 
successful in meeting its performance target, according to Inspection 
Service officials. 

The Inspection Service's outcome performance measure, outlined above, 
is closely aligned with its strategic goal--to ensure a safe, secure, 
and drug-free environment--and with its strategic objective--to enhance 
the security of USPS facilities. Linking their performance measures and 
targets with their strategic goals and objectives in this way provides 
managers and staff in the Inspection Service with a roadmap that shows 
how their day-to-day activities contribute to achieving broader 
Inspection Service goals (see fig. 4). Inspection Service officials 
told us that they designed their security-related strategic goal and 
objective to support USPS's broader strategic goal of improving 
services, which includes activities that protect mail, employees, and 
customers in order to improve services.[Footnote 40] 

Figure 4: Linkages between USPS Inspection Service Strategic Goals and 
Performance Measure for Facility Protection: 

[See PDF for image] 

Source: GAO analysis of USPS data. 

[End of figure] 

Case Example: Department of Veterans Affairs: 

Although it does not use security performance measures, VA collects 
data that could be used to assess the effectiveness of the agency's 
facility protection program. VA manages a large health system for 
veterans that now includes 154 medical centers, 875 ambulatory care and 
community-based outpatient clinics, and 136 nursing homes. In 2005, 
more than 5 million people received care in VA health care facilities, 
and VA's outpatient clinics registered nearly 58 million visits. VA 
also operates 61 major veterans' benefits facilities, including 57 
regional offices, 3 records centers, and headquarters.[Footnote 41] 
While VA officials noted the absence of performance measures for 
facility protection, we found that the Veterans Health Administration 
and the Veterans Benefit Administration rely on physical security 
assessments to inform risk-management and resource-allocation 
decisions, just as other federal agencies and nonfederal entities do. 
The phases of the physical security assessment include defining the 
criticality of VA facilities, identifying and analyzing the 
vulnerabilities of VA's critical facilities, and identifying 
appropriate countermeasures. VA determines vulnerability based on 
factors such as facility population, building characteristics (e.g., 
the number of floors in the facility), and the presence or absence of 
armed officers and surveillance cameras. VA's assessment includes a 
procedure for scoring and prioritizing identified vulnerabilities at 
each assessed site. The objective of the security assessment is to 
identify shortcomings in the physical security of functional areas 
within critical facilities and to estimate the cost of mitigating the 
risk of disruption or termination of the facility's ability to provide 
services to veterans. For example, they assess the vulnerability of a 
facility's air system to a criminal attack. For each assessed 
functional area, a composite score and corresponding risk rating is 
assigned. The risk-rating system is based on a color-coded "traffic 
light" scheme to designate low-, medium-, and high-risk functional 
areas. The results from the security assessment--in particular, the 
risk-rating designation--are used to develop recommendations to 
mitigate the security risk and to prioritize and justify resource- 
allocation decisions. VA officials said that they had conducted full 
assessments at 18 critical facilities and revisited these facilities a 
year later to determine progress since the assessment. At the time, 
approximately 16 percent of recommended mitigation items had been 
completed, were in progress, or had been planned for. VA officials said 
they are finalizing a database and software that would facilitate the 
tracking of facilities' responses to assessment recommendations. The 
officials said that they expect to roll out the database and software 
within a few months. 

Besides conducting security assessments, organizations can mitigate 
risk by testing their facility protection countermeasures. Like FPS, VA 
conducts inspections and tests to evaluate compliance with security 
policies and procedures and to help ensure that adequate levels of 
protection are employed. In some instances, such as in the VA 
headquarters building, inspections can include simulated attempts to 
gain unauthorized access to a building or to smuggle fake weapons into 
a building.[Footnote 42] For example, within VA, scenario-based tests 
that are derived from emerging security threats are commonly used to 
assess police officers' knowledge of, and compliance with, policies and 
procedures and to evaluate preparedness in the event of an attack. 
Earlier in this report, we noted that FPS has developed a performance 
measure using similar tests in order to assess the effectiveness of 
security countermeasures, such as contract security guards, in 
mitigating risk. In addition, both VA and FPS conduct biannual 
inspections of compliance with standards and policies, including for 
physical security. 

Such measurable activity could enable the measurement of program 
outcomes, including changes in the number of unauthorized building 
entries or the number of weapons and other prohibited items detected as 
part of facility intrusion tests. Although VA officials told us they 
had not developed performance measures, we believe they have valuable 
data that can be used to measure the overall effectiveness of the 
agency's facility protection program. For VA, security assessments and 
testing activity provide useful feedback on how well security measures 
have operated and whether they continue to be appropriate for the 
future. Further, these evaluations could form the basis for overall 
evaluations of VA's building security program and could provide data 
for performance measurement initiatives. 

Federal Guidance for Developing and Using Performance Measures Exists 
for IT Security, but Not for Physical Security: 

While performance measures have been used to monitor many federal 
programs, little has been done to apply performance measurement to 
physical security programs--a complex and challenging undertaking, 
since outcomes may not be quickly achieved or readily observable. 
Although we found that physical security performance measurement is a 
challenge for many organizations in the public and private sector, we 
found that the information technology (IT) security area has 
performance measurement initiatives under way. Similar to facility 
protection, IT security has been a considerable concern in large part 
because computer systems are vital to many of our nation's critical 
operations and infrastructure. The dependency on these systems prompted 
a number of congressional actions, including various mandates for 
agencies to implement security controls to protect information systems 
within the federal government. In compliance with these federal 
requirements, agencies must demonstrate their progress in meeting 
requisite information security requirements and report on their actual 
level of performance based on the results of annual program reviews. 

In its role as a leader on technology issues, the National Institute of 
Standards and Technology (NIST), a subagency within the Department of 
Commerce, issued a report in 2003--Security Metrics Guide for 
Information Technology Systems--to provide guidance on how an 
organization can use performance measures to determine the adequacy of 
in-place security controls, policies, and procedures intended to 
mitigate security risks.[Footnote 43] More specifically, the report 
provides an approach that helps managers decide where to invest 
additional security protection resources or how to identify and 
evaluate controls that are not effective. The guidance is the 
culmination of several efforts to identify a suitable method for 
measuring security and supplemented ongoing initiatives by OMB to help 
agencies develop workable measures of job and program performance that 
would hold federal employees accountable for their IT security 
responsibilities. In addition to providing practical examples of 
security performance measures that can be readily used or modified to 
meet agency-specific needs, the report provides a detailed description 
of how performance measurement is being approached in the IT security 
area and addresses the following areas: (1) the roles and 
responsibilities of agency staff at all levels, (2) the benefits of 
using performance measures, and (3) an overview of the performance 
measures development and implementation process. 

The NIST report advocates the use of measurable performance measures 
based on IT security performance goals and objectives. In turn, the 
report describes performance measures as tools designed to facilitate 
decision making and improve performance and accountability through the 
collection, analysis, and reporting of relevant performance-related 
data. NIST describes three types of performance measures-- 
implementation, efficiency and effectiveness, and impact--that can be 
used to measure progress (see table 5). Although NIST uses different 
terminology to describe the three types of performance measures, they 
are similar to the output and outcome measures that we have advocated 
for use in monitoring and reporting program accomplishments. The NIST 
report cautions that the type of performance measures that can 
realistically be obtained and used for performance improvement depends 
on the maturity of the security program. According to NIST, in the 
early stages of establishing a security program, the focus tends to be 
on developing security policies and procedures, and beginning to ensure 
that security controls are implemented. In such an environment, an 
appropriate performance measure would be one that focuses on 
implementation, such as the percentage of information systems with 
approved security plans. In contrast, a more mature security program 
may evolve to measure the efficiency and effectiveness of security 
controls and the impact of these controls on the organization's 
mission. In such cases, the performance measures may concentrate on the 
evidence and results of testing. 

Table 5: Types of Information Technology Security Performance Measures 
Described by NIST: 

Type of measure: Implementation; 
Performance measure: Percentage of systems with approved security plans 
and the percentage of systems with password policies configured as 
required; 
Purpose: Assess the extent to which security plans and password 
policies have been documented and implemented to support the security 
program. 

Type of measure: Efficiency and effectiveness; 
Performance measure: Percentage of crackable passwords within a 
predefined time threshold; 
Purpose: Evaluate the results of security controls that have been 
implemented; validate whether security controls, as described in the 
security plan, are effective in protecting the organization's assets. 

Type of measure: Impact; 
Performance measure: Quantify incidents by type (e.g., root compromise, 
password compromise, malicious code, denial of service) and correlate 
incident data with the percentage of trained users and system 
administrators; 
Purpose: Measure the impact of training on security.  

Source: NIST. 

[End of table] 

The guidance goes beyond extolling the virtues of using performance 
measures and illustrates the place of IT security within a larger 
organizational context, provides a roadmap for how to develop and 
implement a performance measurement program, and includes practical 
examples of performance measures. According to NIST, the performance 
measures that are ultimately selected can be useful not only for 
measuring performance, identifying causes of unsatisfactory 
measurements, and pinpointing improvement areas, but also for 
facilitating continuous policy implementation, effecting security 
policy changes, and redefining goals and objectives. NIST notes that 
successful implementation of a security performance measurement program 
can also assist agencies in meeting OMB's annual requirements to report 
the status of agency IT security programs. In addition to providing 
examples of performance measures, some of which are required by OMB, 
the report also includes a standardized template that describes the 
various data elements that should be documented (see fig. 5). The data 
elements include: 

* Performance goal: States the desired results of implementing security 
control objectives that are measured by the metric. 

* Performance objective: States the actions that are required to 
accomplish the performance goal. 

* Metric: Defines the metric by describing the quantitative 
measurements it provides. 

* Purpose: Describes the overall functionality obtained by collecting 
the metric; includes whether a metric will be used for internal 
performance measurement or for external reporting, what insights are 
hoped to be gained from the metric, and whether regulatory or legal 
lessons exist for collecting a specific metric if applicable. 

* Implementation evidence: Includes indirect indicators that validate 
that the activity is being performed and causation factors that may 
point to the causes of unsatisfactory results for a specific metric. 

* Frequency: Establishes time periods for collecting data that is used 
for measuring changes over time. 

* Formula: Describes the calculation to be performed that results in a 
numeric expression of the metric. 

* Data source: Identifies the location of the data to be used in 
calculating the metric (e.g., databases, tracking tools, organizations, 
or specific roles within the organization that can provide required 
information). 

* Indicators: Provide information about the meaning of the metric and 
its performance trend; state the performance target and indicate what 
trends would be considered positive in relation to the performance 
target. 

The NIST report notes that the universe of possible performance 
measures, based on policies and procedures in place in the 
organization, will be quite substantial and that the final performance 
measurement set selected for initial implementation should relate to 
high-priority areas, use data that can be realistically obtained, and 
measure processes that already exist and are relatively stable. The 
guidance further states that performance measures can be developed and 
selected using a phased approach. This approach identifies short-, mid- 
, and long-term measures where the time frame in which these measures 
are implemented depends on a combination of system-level effectiveness, 
performance measure priority, data availability, and process stability. 
The NIST report also notes that, once applicable performance measures 
have been identified, they should be documented using a standardized 
template (see figure 5). Standardizing the reporting process is 
particularly useful in cases where the reporting process within an 
organization is inconsistent. Such practices, among others, can help 
ensure the success of a performance measurement program. 

Figure 5: Sample Standardized Performance Measurement Data Form: 

[See PDF for image] 

Source: NIST. 

[End of figure] 

Federal Agencies Have Received Minimal Guidance on Using Performance 
Measurement for Facility Protection Programs: 

We have previously reported that, at the agencywide level, agencies 
face obstacles in developing meaningful, outcome-oriented performance 
goals and in collecting data that can be used to assess the true impact 
of facility protection efforts. GPRA emphasizes measuring the results 
of products and services delivered by a federal program (i.e., 
outcomes). For programs that have readily observable results or 
outcomes, performance measurement may provide sufficient information to 
evaluate the effectiveness of facility protection efforts. Yet in some 
programs, such as facility protection, outcomes are not quickly 
achieved or readily observable, or their relationship to the program is 
often not clearly defined. In such cases, more in-depth program 
evaluations, in addition to performance measurement, may be needed to 
examine the extent to which a program is achieving its objectives. 

While federal agencies have made some progress developing performance 
measures for facility protection, we noted that the emphasis is on 
using output measures that monitor program activity rather than outcome 
measures that assess the overall impact of program activity. This lack 
of outcome measures leaves agencies with insufficient information to 
determine whether security activities are effective and to evaluate 
whether the benefits of security investments justify their costs. We 
have previously reported that various security program outputs--such as 
conducting patrols--may have contributed to improved security, but that 
using them as performance measures may not systematically target areas 
of higher risk and may not result in the most effective use of 
resources, because these measures are not pointed toward outcomes. Such 
output measures do not provide an indication of what these activities 
are accomplishing. By contrast, outcome measures that are clearly tied 
to results would indicate the extent of progress made and help identify 
the security gaps that still remain.[Footnote 44] Without more 
information on security program outcomes, agencies do not know the 
extent to which security enhancements have improved security or reduced 
federal facilities' vulnerability to acts of terrorism or other forms 
of violence. In addition, there is some inconsistency in the types of 
activities that are being monitored and used as indicators of an 
agency's progress in fulfilling its facility protection 
responsibilities. If agencies use inconsistent approaches to 
performance measurement, decision makers could be at risk of having 
incomparable performance information to determine funding priorities 
within and across agencies. 

Echoing what organizations outside the U.S. federal government told us, 
some agency security officials said it was challenging to measure the 
impact that various approaches have on actually improving security. 
Some agency officials also noted that resources for performance 
measurement initiatives were scarce. Additionally, the availability of 
information needed for applying performance measurement to facility 
protection is somewhat limited. More generally, with the exception of 
DHS, the agencies that we reviewed do not view security as their 
primary mission, and some agencies are faced with competing demands for 
limited resources to accomplish broader agency goals. In such an 
environment, security must be integrated using scarce resources. 

In spite of the inherent difficulty in measuring facility protection 
performance, and the considerable emphasis on doing so, agencies have 
minimal guidance on how to accomplish this. There is, however, broad 
guidance for the protection of critical infrastructures, which includes 
government facilities. Using a risk-based approach, the Draft National 
Infrastructure Protection Plan (NIPP) was developed to provide an 
integrated, comprehensive approach to addressing physical, cyber, and 
human threats and vulnerabilities.[Footnote 45] As part of the NIPP, 
DHS officials have provided guidance and collected information on core 
performance measures--which are common measures that can be broadly 
applied to all protection programs for critical infrastructures and key 
assets. These measures are mostly process/input and output oriented, 
and DHS officials noted that they hope to develop outcome measures as 
the program matures. The NIPP, however, does not provide or collect 
information on specific performance measures related to the protection 
of federal facilities. Rather, it notes that FPS--the agency assigned 
responsibility for implementing the NIPP framework and guidance in the 
government facilities sector--will develop such performance measures. 
Separately, OMB issued a memorandum in June 2004 that reported it was 
working with agencies on initiatives related to physical security 
reporting requirements noted in Homeland Security Presidential 
Directive Number 7 (HSPD-7).[Footnote 46] The memorandum instructed 
each agency to disclose the performance measures it had designed and 
implemented to measure outputs and outcomes. However, OMB did not 
provide specific guidance or standards and instead directed agencies to 
use DHS guidance--related to the NIPP--that does not specify measures 
for facility protection. 

By contrast, the IT security performance measurement guidance issued by 
NIST includes information on: (1) clearly defining roles and 
responsibilities for relevant stakeholders; (2) establishing security 
goals and objectives; (3) identifying and implementing performance 
measures and performance targets; and (4) using measures that are 
unique to IT security to assess the impact of IT security efforts. One 
security official from the gaming industry said that IT security 
performance was somewhat easier to evaluate than physical security 
performance because it is possible to directly monitor the number of 
attempted IT security breaches. A foreign government agency we 
interviewed is farther along in developing standards and performance 
measures for IT security than for physical security. In general, IT 
security approaches are slightly more standardized than physical 
security because the field is newer than physical security and because 
organizations had to work together to prepare for possible 
complications in the year 2000 (Y2K). Despite such differences between 
IT and physical security performance measurement, some of the 
performance measurement guidance could be applicable to physical 
security situations. 

ISC is a body that addresses governmentwide security policy issues and, 
like NIST, is well positioned to develop guidance and promote 
performance measurement. Executive Order 12977 calls for ISC to play an 
oversight role in implementing appropriate security measures in federal 
facilities and taking actions that would enhance the quality and 
effectiveness of security in federal facilities. As we reported in 
November 2004, ISC has already made progress in coordinating the 
federal government's facility protection efforts through activities 
such as developing security policies and standards for leased space, 
improving information sharing, and coordinating the development of a 
security database of all federal facilities.[Footnote 47] The ISC Chair 
told us that he supports the use of performance measurement as a means 
of strengthening federal facility protection efforts. 

Conclusions: 

Given their competing priorities and limited security resources, U.S. 
federal agencies could benefit from specific performance measurement 
guidance and standards for facility protection to help them address the 
challenges they face and help ensure that their physical security 
efforts are achieving the desired results. While some of these agencies 
have implemented performance measures to monitor their security 
programs' outputs, fewer have developed outcome measures to assess the 
extent to which security enhancements have improved security or reduced 
their facilities' vulnerability to acts of terrorism or other forms of 
violence. Without a means of comparing security effectiveness across 
facilities, particularly program outcomes, the U.S. government is open 
to the risk of either spending more money for less effective physical 
security or investing in the wrong areas. The output measures that 
federal agencies have developed provide an indication of what their 
security activities are accomplishing but do not indicate the extent of 
progress made or help identify the security gaps that still remain, as 
outcome measures would. Fundamentally, performance measurement helps 
ensure accountability, since it enables decision makers to isolate 
certain activities that are hindering an agency's ability to achieve 
its strategic goals. Performance measurement can also be used to 
prioritize security needs and justify investment decisions so that an 
agency can maximize available resources. Over time, a thorough 
performance measurement approach could allow the federal government to 
manage the risks to federal facilities both within and across agencies. 
Recognizing the unique nature of U.S. federal agencies' missions, some 
uniformity in measuring performance in facility protection efforts 
could facilitate comparisons across agencies. 

Organizations outside of the U.S. government--including private-sector 
entities as well as state, local, and foreign government agencies--have 
developed and are using performance measures for facility protection, 
and their knowledge and experience could be helpful to U.S. federal 
agencies in developing and refining their own performance measures. 
Likewise, because the application of performance measures to facility 
protection can be challenging, many nonfederal organizations are 
looking to U.S. government agencies for assistance and leadership. Some 
U.S. federal agencies are already collecting data that could be used 
for measuring security performance, and they currently have guidance 
for measuring information technology security, but not physical 
security. The U.S. federal government has provided guidance and 
collected information on a set of common measures that can be broadly 
applied to all protection programs for critical infrastructures and key 
assets, and agencies will be required to report on additional security 
performance measures that are sector-specific. With regard to federal 
facilities, the ISC, in serving as the central coordinator for U.S. 
agencies' federal facility protection efforts, is well positioned to 
develop and promote performance measurement guidance and standards for 
physical security, and could look to information technology security as 
a model to follow. In turn, it could draw from examples of performance 
measurement we identified in the private sector and foreign government 
agencies. Federal agencies could subsequently follow the guidance and 
standards to evaluate their actions, identify lessons learned, and 
develop strategies for overcoming any challenges in developing and 
using performance measures for facility protection. Because of the ever-
changing nature of security threats and new security technologies and 
countermeasures, such guidance and standards would need to be 
periodically reviewed and updated. The development of guidance and 
standards for facility protection could help ensure uniform application 
of performance measurement so that the U.S. federal government, 
particularly its largest real-property-holding agencies, would be 
accountable for its facility protection programs and would be able to 
demonstrate that security investments are producing a return, both 
within and across agencies, in terms of better-protected facilities. 

Recommendations for Executive Action: 

To ensure that useful information is available for making decisions 
about the allocation of resources for, and the effectiveness of 
investments in, the protection of federal facilities, we recommend that 
the Secretary of Homeland Security direct the Chair of ISC to do the 
following: 

* as part of ISC's efforts to support DHS in developing sector-specific 
performance measures for the security of federal government facilities, 
establish guidance and standards, with input from ISC member agencies, 
for measuring performance in facility protection--with a particular 
focus on developing outcome measures; 

* communicate the established guidance and standards to the relevant 
federal agencies; and: 

* ensure that the guidance and standards are regularly reviewed and 
updated. 

Agency Comments and Our Evaluation: 

We provided a draft of this report to DHS, GSA, USPS, VA, and Interior 
for their official review and comment. DHS concurred with the report's 
overall findings and recommendations. DHS comments are contained in 
appendix III. USPS and VA concurred with the report's findings. In 
addition, DHS and USPS provided separate technical comments, which we 
incorporated into the final report where appropriate. GSA notified us 
that they had no comments on this report. 

Interior, while generally agreeing with the report's findings, 
suggested that an agency-by-agency assessment of each federal agency's 
facility vulnerabilities would be more effective than a cross-agency 
facility protection performance measure. We agree that identifying and 
monitoring vulnerabilities is important, but believe that it is also 
important for decision makers to have comparable information about the 
relative security performance of facilities within an agency as well as 
across the federal government. Interior also expressed concern that a 
more public viewing of agency facility protection performance could 
reveal weaknesses or vulnerabilities that could be exploited. We agree 
that this could be a concern but leave the development of guidelines 
for using and protecting this information to ISC and its member 
agencies. Interior also provided technical comments, which we 
incorporated. Comments from Interior and our evaluation can be found in 
appendix IV. 

As agreed with your office, unless you publicly announce the contents 
of this report earlier, we plan no further distribution until 30 days 
from the report date. At that time, we will send copies of this report 
to other interested congressional committees and the Secretaries of the 
Interior, Homeland Security, and Veterans Affairs; the Administrator of 
GSA; and the Postmaster General of the U.S. Postal Service. We will 
also make copies available to others upon request. In addition, the 
report will be available at no charge on the GAO Web site at 
[Hyperlink, http://www.gao.gov]. 

If you have any questions regarding this report, please contact me on 
(202) 512-2834 or at goldsteinm@gao.gov. Contact points for our Offices 
of Congressional Relations and Public Affairs may be found on the last 
page of this report. GAO staff who made major contributions to this 
report are listed in appendix V. 

Sincerely yours, 

Signed by: 

Mark L. Goldstein: 
Director, Physical Infrastructure Issues: 

[End of section] 

Appendix I: Objectives, Scope, and Methodology: 

The objectives of our report were (1) to identify examples of 
performance measures for facility protection being used by selected 
organizations outside of the federal government--including private- 
sector entities, state and local governments, and foreign governments; 
and (2) to determine the status of U.S. federal agencies' efforts to 
develop and use performance measures as part of their facility 
protection programs. 

To identify examples of performance measures for facility protection 
being used by selected organizations outside the federal government, we 
interviewed representatives from the private sector, U.S. state and 
local governments, and foreign governments. With respect to the private 
sector, we asked a number of umbrella organizations to identify 
industries that are likely to utilize performance measures for facility 
protection and known leaders in the security performance measurement 
area. These umbrella organizations included ASIS International, Real 
Estate Roundtable, Financial Services Roundtable, Financial Services 
Information Sharing and Analysis Committee, International Facility 
Management Association, and National Association of Industrial and 
Office Properties. GAO staff also attended the annual ASIS 
International Conference in 2005. Some of these entities stated that 
the gaming and finance industries would be the most appropriate to 
review, since these industries have invested significantly in the 
quality of their security efforts. As a result, we interviewed 
officials from four gaming entities and five major financial services 
organizations. To maintain the organizations' security and the 
confidentiality of proprietary information, we do not identify specific 
organizations in this report. 

For the gaming industry, a member of the Real Estate Roundtable 
provided a contact who was known to be active in physical security 
testing and performance measurement. This individual then arranged a 
joint interview for us with a number of gaming entities. Some of the 
representatives present at the interview were also members of the Las 
Vegas Security Chiefs Association or ASIS International Gaming and 
Wagering Council. The five financial services organizations we 
interviewed were selected because they (1) were considered to be 
leaders in their industry; (2) were recommended by others within the 
industry; (3) were members of ASIS International, the largest 
organization supporting security professionals; or (4) have had prior 
security concerns related to threats of terrorism. 

To determine if U.S. state and local governments have developed 
performance measures for facility protection, we attempted to contact 
10 state and 10 local governments. For state governments, we selected 
the 10 states receiving the most funding from the Department of 
Homeland Security's (DHS) State Homeland Security Program grant in 
fiscal year 2005. For local governments, we selected the 10 local 
governments/urban areas receiving the most funding from DHS's Urban 
Areas Security Initiative grant in fiscal year 2005.[Footnote 48] Of 
the 20 state and local governments we attempted to contact, we were 
able to obtain information from officials from 17 of them. While all 17 
of these state and local governments were engaged in facility 
protection efforts, only a few had developed performance measures to 
evaluate the effectiveness of these efforts. Table 6 shows a listing of 
these state and local governments. The agencies we approached within 
each of the state and local governments were often, but not always, the 
agencies responsible for real property or policing/security. Some of 
the state and local governments we attempted to contact were also 
identified by the Government Accounting Standards Board as having 
performance measurement initiatives on a variety of their 
organizations, departments, and projects. 

Table 6: U.S. State and Local Governments Contacted: 

Organization: U.S. state governments; 
Location: California; Florida; Georgia; Illinois; Michigan; New Jersey; 
New York; Ohio; Pennsylvania; Texas. 

Organization: U.S. local governments; 
Location: Boston, Mass; Detroit, Mich; Washington, D.C.[A]; Los 
Angeles, Calif; New York, N.Y; Philadelphia, Pa; San Francisco, Calif. 

Source: GAO. 

[A] For the purposes of this report, Washington, D.C., was treated as a 
local government. 

[End of table] 

For our work with foreign governments, we conducted international site 
visits in three foreign countries--Australia, Canada, and the United 
Kingdom--where we interviewed a number of government agencies and 
organizations about their use of performance measures for facility 
protection. (Table 7 shows a listing of each of these agencies.) We 
selected these three countries for site visits because they are known 
to have experience with threats of terrorism and because they have been 
identified by the Government Accounting Standards Board as having 
performance measurement initiatives, not necessarily for facility 
protection but for government initiatives in general. We also spoke 
with representatives from a number of other foreign governments. While 
these other governments have facility protection efforts in place, they 
said they did not use performance measures to assess the effectiveness 
of these efforts. Furthermore, officials from some of these countries 
told us that they look to the United States for guidance on a number of 
issues relating to facility protection, including how to measure 
effectiveness. For such reasons, these countries were not highlighted 
in this report. 

Table 7: Foreign Government Agencies and Organizations Visited: 

Location: Australia; 
Organization: Airservices Australia; Attorney- General's Department; 
Commonwealth Scientific and Industrial Research Organization; Customs 
Service; Department of Defence; Department of Foreign Affairs and 
Trade; Federal Police; National Audit Office; Taxation Office. 

Location: Canada; 
Organization: Bank of Canada; Corps of Commissionaires; Department of 
National Defence; National Gallery; Office of Auditor General; Public 
Works and Government Services Canada; Royal Canadian Mounted Police; 
Treasury Board. 

Location: United Kingdom; 
Organization: Cabinet Office; Department for Transport; Foreign and 
Commonwealth Office; Home Office; National Infrastructure Security 
Coordination Centre, Security Service; National Security Advice Centre, 
Security Service; Office for Civil Nuclear Security. 

Source: GAO. 

[End of table] 

In addition to interviewing officials from the nonfederal entities 
identified above, we reviewed relevant documentation obtained from 
these organizations, previous GAO reports, and performance measurement 
and facility protection literature from ASIS International and other 
sources. 

For the second objective--to determine the status of U.S. federal 
agencies' efforts to develop and use performance measures as part of 
their facility protection programs--we interviewed selected officials 
from the major civilian real property holding agencies. These agencies 
include the General Services Administration (GSA), the United States 
Postal Service (USPS), the Department of Veterans Affairs (VA), and the 
Department of Interior (Interior). GSA acknowledged the need to measure 
the performance of facility protection efforts; however, for most 
facility protection issues, they defer to the Federal Protective 
Service (FPS) within DHS. Because FPS is responsible for protecting all 
GSA buildings, we also interviewed officials from FPS. For each of the 
selected federal agencies, we reviewed agency strategic and performance 
plans, security goals, performance reports, and other relevant 
documentation provided to us. We also interviewed the Executive 
Director of the Interagency Security Committee (ISC)--a DHS-led 
committee that is tasked with coordinating federal agencies' facility 
protection efforts. Finally, we reviewed a number of national 
strategies and presidential directives; previous GAO reports; and 
relevant reports by the Office of Management and Budget, the 
Congressional Budget Office, the Congressional Research Service, and 
other government entities. We also reviewed laws and authorities 
related to facility protection. 

It is important to note that the private-sector entities, U.S. state 
and local governments, and foreign governments selected for our review 
are not representative of the universe of such organizations. 
Furthermore, GAO has not evaluated the robustness and quality of the 
performance measures cited in this report. Rather, these measures are 
simply a compilation of what we have gathered from the nonfederal and 
federal entities we have interviewed. Additionally, the performance 
measures identified in this report may not include all performance 
measures relating to the protection of federal facilities. We used our 
judgment to classify the performance measures into process/input, 
output, and outcome measures according to our definitions, but these 
performance measures could be classified differently depending on the 
performance measurement goals or objectives used by an organization. 

Also, ISC has identified GAO as an associate member, which includes the 
ability to serve on ISC subcommittees. No GAO staff member, however, 
serves on any subcommittee. Furthermore, no GAO staff member actively 
participates in ISC meetings or contributes to decisions. Rather, GAO's 
role on ISC is only to observe proceedings and obtain ISC information 
distributed to the other ISC members. Because of GAO's observational 
role, our independence in making recommendations involving ISC and in 
completing this engagement was maintained. 

Officials from nonfederal and federal entities provided much of the 
information used in this report. In most cases where officials provided 
their views as representatives of their organizations, we corroborated 
the information with other officials or with documentation provided to 
us. We requested official comments on this report from DHS, GSA, USPS, 
VA, and Interior. Furthermore, when we used examples from the private 
sector, state and local governments, foreign governments, and the 
National Institute of Standards and Technology (NIST), we provided the 
respective entity an opportunity to review relevant portions of the 
report and offer comments, thus ensuring the validity of our reporting. 
We conducted site visits and interviews from July 2005 through January 
2006. We conducted our work from May 2005 through April 2006 in 
accordance with generally accepted government auditing standards. 

[End of section] 

Appendix II: Examples of Performance Measures Used by Selected 
Organizations outside of the Federal Government: 

The performance measures below were provided by the selected 
organizations we interviewed outside of the federal government. We did 
not evaluate the quality of the performance measures, and we used our 
judgment to classify them according to the following definitions of 
performance measures: 

* Output measures focus on the quantity of direct products and services 
a program delivers and the characteristics of those outputs, including 
efficiency, cost-effectiveness, timeliness, quality, and customer 
service. 

* Outcome measures provide information on the results of the direct 
products and services a program has delivered. 

* Process/input measures address the type or level of program 
activities an organization conducts and the resources used by the 
program. 

The performance measures could be classified differently depending on 
the performance measurement goals or objectives used by an 
organization. 

Output. 

Number of risk assessments performed; 
New security projects; 
* Security checklist completed during planning stages; 
* Security officials consulted; 
Number of security requests received; 
* Security report requests; 
* New access badge requests; 
* Requests for changes to existing badges; 
Security clearance; 
* Number of background screenings completed; 
* Average time to process background screenings; 
* Average number of days to process security clearances; 
* Number of overdue security clearances by more than 4 weeks; 
* Cost per security clearance; 
* Percentage of officers/contractors who hold sufficient level of 
security clearance when compared to their duties; 
Alarm systems; 
* Responded to and cleared; 
* Alarms with unique responses (i.e., alarms requiring guards to 
respond in person); 
* Failed to arm; 
Number of police incidents/reports filed Number of threats; 
* Against employees; 
* Against facilities; Security incident reaction/response; 
* Number of avoidable incidents detected; 
* All significant investigations completed within 45 days; 
Compliance with security policies and standards; 
* Number of exceptions reviewed; 
* Number of significant policy breaches; 
* Surveillance and communication systems are compliant with standards; 
* Entry/access control systems are compliant with standards; 
* Security staff are fulfilling their contract obligations; 
Customer/client satisfaction; 
* Staffing-- training, professional appearance, professional behavior, 
turnover rate, supervision; 
* Security reporting--accuracy, timeliness, use of correct forms; 
* Management--responsiveness, understanding of issues, availability, 
number of personal contacts; 
Timely delivery of security alerts and briefings; 
Percentage of alarms responded to within 20 minutes during nonpublic 
service hours; 
Increased attendance at training courses for security officers; 
Number of new employees, contractors, and consultants who have not 
attended a security awareness session within 4 weeks of receiving their 
identification pass; 
Percentage of security guards in compliance with licensing standards 
within a 7-day period; 
All scheduled audit and compliance reports completed in 14 days. 

Outcome. 

Change in the total number of security-related incidents; 
* Accident; 
* Assault; 
* Burglary; 
* Organization assets; 
* Personal assets; 
* Drugs/ Alcohol; 
* Extortion; 
* Fire; 
* Fraud Referral; 
* Harassment; 
* Larceny/Theft; 
* Malicious damage; 
* Public disorder; 
* Robbery; 
* Suspicious activity; 
* Terrorism; 
* Vandalism; 
* Workplace violence; 
Evidence of damage to building and facilities; 
Evidence of harm to staff or tenants; 
Change in risk rating resulting from countermeasures deployed; 
Security policies and practices receive favorable comment from security 
audit program; 
Agency stakeholders view agency as a safe custodian of allocated 
resources and assets. 

Process/Input. 

Number of facilities being protected (including types and locations); 
Number of security staff; 
Number of security guards/security escorts; 
Personal security arrangements for after-hours entry/access; 
Perimeter security; 
* Assessment of entry/exit points; 
* Serviceability of perimeter security equipment (locks, door frames, 
security signs); 
* Sufficiency of perimeter lighting; 
* Presence of obstructions, waste containers/material, combustibles, 
other risk factors; 
* Evidence of vandalism, malicious damage, or other criminal activity; 
* Maintenance schedules; 
Number of security clearances undertaken; 
Number of training courses and drills conducted;
Security threats and general risks discussed at management forum and 
disseminated to all levels of agency staff; 
Security spending per square foot. 

Source: GAO. 

Note: GAO analysis of data from state, local, and foreign government 
agencies and private-sector organizations. 

[End of table] 

[End of section] 

Appendix III: Comments from the Department of Homeland Security: 

U.S. Department of Homeland Security: 
Washington, DC 20528: 

May 15, 2006: 

Mr. Mark L. Goldstein: 
Director, Physical Infrastructure Issues: 
U. S. Government Accountability Office: 
441 G Street, NW: 
Washington, DC 20548: 

Dear Mr. Goldstein: 

Re: Draft Report GAO-06-612, Homeland Security: Guidance and Standards 
Are Needed for Measuring the Effectiveness of Agencies' Facility 
Protection Efforts (GAO Job Code 543129): 

The Department of Homeland Security (DHS) appreciates the opportunity 
to review and comment on the Government Accountability Office's draft 
report. The report notes many of the challenges associated with 
developing and using meaningful performance measures. We concur with 
the overall findings and recommendations contained therein, and share 
your concern that without improved or additional means of measuring 
performance, it is difficult to assess the effectiveness and efficiency 
of efforts to protect Federal facilities. 

DHS is implementing the requirements of Homeland Security Presidential 
Directive-7, Critical Infrastructure Identification, Prioritization, 
and Protection. As an essential part of that initiative, the Government 
Facilities Sector (GFS) is establishing performance measure guidance 
for federal, state, and local governments so they can better assess the 
effectiveness of their facility protection programs. The GFS is under 
the lead of the Federal Protective Service located within Immigration 
and Customs Enforcement. The Interagency Security Committee (ISC) is 
currently partnering with the GFS on that effort with respect to 
federal facilities, and will use it as the baseline for developing the 
recommended performance measurement guidance. We believe this is the 
most appropriate course of action to leverage limited resources and 
ensure consistency and timely completion of both tasks. 

The ISC will include this task in its Action Plan for Fiscal Years 2007 
and 2008, and will ensure that the performance measurement guidance 
reflects input from all ISC members and is distributed to all federal 
agencies. Progress, however, will be largely dependent on the 
availability of sufficient resources. 

Technical comments will be sent under separate cover. 

Sincerely, 

Signed by: 

Steven J. Pecinovsky: 
Director: 
Departmental GAO/OIG Liaison Office: 

[End of section] 

Appendix IV: Comments from the Department of the Interior: 

United States Department of the Interior: 
Office Of The Assistant Secretary Policy, Management And Budget 
Washington, DC 20240: 

Take Pride' In America: 

May 15 2006: 

Mr. Mark L. Goldstein: 
Director, Physical Infrastructure Issues: 
U.S. Government Accountability Office: 
441 G Street, NW: 
Washington, D.C. 20548: 

Dear Mr. Goldstein: 

Thank you for providing the Department of the Interior the opportunity 
to review and comment on the draft U.S. Government Accountability 
Office report Homeland Security, Guidance and Standards Are Needed for 
Measuring the Effectiveness of Agencies Facility Protection Efforts 
(GAO-06-612), May 2006. In general, we agree with the findings, except 
as discussed in the enclosure, and we agree with the recommendations in 
the report. 

The enclosure provides specific comments from the Department's Office 
of Law Enforcement and Security and the Office of Planning and 
Performance Management. We hope our comments will assist you in 
preparing the final report. 

Sincerely, 

Signed by: 

R. Thomas Weimer: 
Assistant Secretary: 

Enclosure: 

U.S. General Accountability Office Draft Report Homeland Security: 
Guidance and Standards Are Needed for Measuring the Effectiveness of 
Agencies Facility Protection Efforts (GAO-06-612): 

Specific Comments: 

Office of Law Enforcement and Security: 

(1) Pages 27-30: 

This section of the document refers to the Bureau of Land Management 
(BLM) conducting vulnerability assessments. This information regarding 
the BLM is incorrect and all reference to BLM should be removed from 
this section of the document. BLM has not been conducting the types of 
vulnerability assessments discussed in the report, and does not have a 
specialized physical security assessment methodology. 

(2) Page 30: 

"However, the OLES officials told us that they do not have formal 
performance measures and targets for reducing the risk ratings." In 
April 2006, OLES developed and submitted for inclusion in the 
Departmental Strategic Plan performance measures related to the 
reduction in the percent of physical security vulnerabilities 
identified at departmental facilities. 

General Comments: 

Office of Planning and Performance Management: 

The report is unclear as to the value of a common set of government- 
wide facility protection performance measures and for whom this 
additional information would be directed. The report seems to express 
concern that there is not a single set of government-wide facility 
protection performance measures. A more critical need should be 
determining if each agency has effectively assessed and corrected its 
own facility vulnerabilities. Such an agency-by-agency assessment more 
effectively considers the different levels of criticality and 
protection needs of each facility in terms of its mission and 
individual condition, than could be covered by a cross-agency facility 
protection performance measure. 

It is also not clear as to who is meant to be the recipient and 
benefactor of such a cross-agency assessment of facility protection 
performance. It is confusing in the report if GAO's concern is that 
there is not adequate information within agencies to make effective 
decisions about protecting their own facilities or if the results of 
this cross-agency assessment on facility protection is meant for public 
documents that would be related to implementation of the Government 
Performance and Results Act or the Program Assessment Rating Tool for 
publication on the OMB ExpectMore.Gov website. If the report is 
promoting a more public viewing of facility protection performance, it 
should also discuss any guidelines for ensuring that such information 
provides adequate accountability without revealing weaknesses or 
vulnerabilities that could be exploited. The GAO report needs to be 
clearer as for whom this information is targeted, guidelines for how 
such information could be made available and yet protected, and the 
value for expending the resources to collect this cross-agency 
information vs conducting a more direct internal agency-by-agency 
assessment of facility vulnerability and correction. 

The following are GAO's comments on Interior's letter dated May 15, 
2006. 

GAO Comments: 

1. Interior suggested that an agency-by-agency assessment of each 
federal agency's facility vulnerabilities would be more effective than 
a cross-agency facility protection performance measure. We agree that 
identifying vulnerabilities and monitoring efforts to address those 
vulnerabilities is a useful part of an agency's comprehensive facility 
protection program. For example, the Department of Veterans Affairs 
conducts vulnerability assessments, and one Australian government 
agency we interviewed monitors the effect of different security 
investments on its facilities' risk ratings (which typically involve 
threat and vulnerability factors). However, we believe it is also 
important for decision makers to have comparable information about the 
relative security performance of facilities within an agency, rather 
than just in one bureau or service, as well as across the federal 
government. Such information could help reduce the risk of spending 
more money for less effective physical security or investing in the 
wrong areas. 

2. Interior expressed concern that a more public viewing of agency 
facility protection performance could reveal weaknesses or 
vulnerabilities that could be exploited. We agree that this could be a 
concern, but choose to leave the development of guidelines for using 
and protecting such information to the Interagency Security Committee 
and its member agencies. 

[End of section] 

Appendix V: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Mark Goldstein (202) 512-2834 or goldsteinm@gao.gov: 

Staff Acknowledgments: 

Other key contributors to this report were Roshni Davé, Tamera Dorland, 
Brandon Haller, Anne Izod, Jessica Lucas-Judy, Susan Michal-Smith, 
David Sausville, Scott Tessier, Candice Wright, and Dorothy Yee: 

FOOTNOTES 

[1] GSA, Overview of the United States Government's Owned and Leased 
Real Property: Federal Real Property Profile As of September 30, 2004 
(Washington, D.C.) This property includes government-owned and leased 
space. 

[2] GAO, Homeland Security: Actions Needed to Better Protect National 
Icons and Federal Office Buildings from Terrorism, GAO-05-790 
(Washington, D.C.: June 24, 2005), p. 1. 

[3] GAO, High-Risk Series: Federal Real Property, GAO-03-122 
(Washington, D.C.: January 2003). 

[4] In this report, facility protection denotes the protection of not 
only the facilities but also the people, equipment, and other assets 
within them. Additionally, this report focuses largely on protecting 
facilities from threats and acts of terrorism. However, it is important 
to note that facilities are also vulnerable to other types of hazards, 
such as natural disasters and workplace violence, and information in 
this report may be applicable to those hazards as well. 

[5] GAO, Homeland Security: Further Actions Needed to Coordinate 
Federal Agencies' Facility Protection Efforts and Promote Key 
Practices, GAO-05-49 (Washington, D.C.: Nov. 30, 2004). Since the time 
of that report, the ISC Chair noted that he is in the process of 
creating and establishing an action plan with the ISC membership, 
although little progress has been made because of limited resources. 
The Chair anticipates that this action plan, which will articulate a 
roadmap for the ISC to follow in meeting its responsibilities, will 
incorporate portions of the material and related concepts contained in 
GAO reports. 

[6] Risk management is a tool for assessing risks, evaluating 
alternatives, making decisions, and implementing and monitoring 
protective measures. More specifically, risk can be calculated as 
follows: risk = (threat x vulnerability) x consequence. Threat is the 
probability that a specific type of attack will be initiated against a 
particular target or class of targets. The vulnerability of an asset is 
the probability that a particular attempted attack will succeed against 
a particular target or class of targets. It is usually measured against 
some set of standards, such as availability/predictability, 
accessibility, countermeasures in place, and target hardness (the 
material construction characteristics of the asset). The consequence of 
a terrorist attack is characterized as the expected worst case or worst 
reasonable adverse impact of a successful attack. 

[7] Performance measurement is the ongoing monitoring and reporting of 
program accomplishments, particularly progress toward preestablished 
goals. It is typically conducted by program or agency management. 

[8] Of the 20 state and local governments we attempted to contact, we 
were able to obtain information from officials from 17 of them. 

[9] ISC membership includes the Departments of State, Treasury, 
Defense, Justice, Interior, Agriculture, Commerce, Labor, Health and 
Human Services, Housing and Urban Development, Transportation, Energy, 
Education, and Veterans Affairs; GSA; Environmental Protection Agency; 
Central Intelligence Agency; and the Office of Management and Budget. 
Other members of ISC include the Director, U.S. Marshals Service; the 
Director, Security Policy Board; and the Assistant to the President for 
National Security Affairs. As a member of ISC, the Department of 
Defense participates in meetings to ensure that its physical security 
policies are consistent with ISC security standards and policy 
guidance, according to the Executive Director of ISC. 

[10] U.S. Department of Justice, Vulnerability Assessment of Federal 
Facilities, June 28, 1995. 

[11] Office of Homeland Security, The National Strategy for Homeland 
Security, July 2002. 

[12] The other critical infrastructure sectors and key assets 
identified in the National Strategy include agriculture and food, 
water, public health, emergency services, defense industrial base, 
telecommunications, energy, transportation, banking and finance, 
chemical industry and hazardous materials, postal and shipping, 
national monuments and icons, nuclear power plants, dams, and key 
commercial assets. 

[13] Executive Order 13286, dated February 28, 2003, amended numerous 
executive orders to reflect the transfer of certain functions and 
responsibilities to the Secretary of Homeland Security. Section 23 of 
the Executive Order transferred the ISC chairmanship responsibility 
from GSA to DHS. 

[14] Homeland Security Presidential Directive Number 7, Critical 
Infrastructure Identification Prioritization and Protection, Dec. 17, 
2003. 

[15] For example, see GAO, High-Risk Series: An Update, GAO-05-207 
(Washington, D.C.: January 2005); GAO-05-790; and GAO-05-49. 

[16] GAO-05-49. 

[17] Pub.L. No. 103-62, 107 Stat. 285 (1993). 

[18] See GAO, Tax Administration: IRS Needs to Further Refine Its Tax 
Filing Season Performance Measures, GAO-03-143 (Washington, D.C.: Nov. 
22, 2002), pp. 2-3, 46-53. 

[19] GAO, Risk Management: Further Refinements Needed to Assess Risks 
and Prioritize Protective Measures at Ports and Other Critical 
Infrastructure, GAO-06-91 (Washington, D.C.: Dec. 15, 2005), pp. 24, 
105. 

[20] See GAO, Managing For Results: Enhancing Agency Use of Performance 
Information for Management Decision Making, GAO-05-927 (Washington, 
D.C.: Sept. 9, 2005), pp. 7-17 and 21. 

[21] For this report, we categorized the District of Columbia as a 
local government. 

[22] A countermeasure is any action taken or physical equipment used 
principally to reduce or eliminate one or more vulnerabilities. 

[23] The Australian government's Protective Security Manual contains 
governmentwide policies and guidelines that establish the minimum 
standards for the protection of Australian government resources 
(including information, personnel, and assets) that all agencies 
governed by the country's Financial Management and Accountability Act 
of 1997 must meet. 

[24] See GAO-05-927. 

[25] GAO-05-927. 

[26] GAO-03-143. 

[27] GAO-05-927. 

[28] GAO-05-790. 

[29] Although FPS considers this an outcome measure, it is intended to 
reflect the composite level of performance of its three output 
measures. 

[30] The Homeland Security Act requires that, beginning in fiscal year 
2005, DHS prepare the Future Years Homeland Security Program document-
-a 5-year resource plan that outlines departmental priorities and the 
ramifications of program and budget decisions. See GAO, Results 
Oriented Government: Improvements to DHS's Planning Process Would 
Enhance Usefulness and Accountability, GAO-05-300 (Washington, D.C.: 
Mar. 31, 2005). 

[31] GAO, General Services Administration: Many Building Security 
Upgrades Made But Problems Have Hindered Program Information, GAO/ T-
GGD-98-141 (Washington, D.C.: June 4, 1998). 

[32] However, to centrally manage Interior's security initiatives, the 
department established in 2002 a central coordination and oversight 
office for activities related to homeland security. This office--the 
Office of Law Enforcement and Security--has worked within Interior to 
identify assets that are likely targets, conduct risk assessments, and 
coordinate efforts by Interior's bureaus to enhance security at 
individual locations. See GAO-05-790. 

[33] OMB developed PART to support the integration of performance 
information and budgeting. OMB describes it as a diagnostic tool meant 
to provide a consistent approach to evaluating federal programs as part 
of the executive budget formulation process. 

[34] According to OMB, a moderately effective rating means that a 
program is well managed and has established ambitious goals. Programs 
with this rating likely need to improve their efficiency or address 
other problems in design or management to achieve better results. See 
www.expectmore.gov, which is a Web site that was developed by OMB and 
federal agencies to provide information on PART ratings. 

[35] See GAO-05-790. Before the development of this approach, Interior 
did not have a uniform comprehensive risk management approach for 
national icons and monuments--most of which are highly visible and tend 
to have public access. It relied instead on the judgment of senior 
officials in determining where resources should be directed, and the 
risk assessments completed at individual sites were done by a number of 
external experts using different methodologies. In our June 2005 
report, we recognized that Interior had made progress in addressing 
this concern but recommended that the agency link the results of its 
risk assessments and related risk rankings to its funding priorities 
and develop guiding principles for balancing security initiatives with 
its core mission. Regarding the recommendation to develop guiding 
principles, Interior officials told us that they have not made any 
progress on this effort, in large part because resources have been 
dedicated to meeting the requirements of a presidential directive that 
calls for governmentwide identification standards and processes for 
federal employees and contractors. 

[36] Interior officials said that they consider the following 
characteristics in determining which monuments and icons are nationally 
significant: (1) asset is widely recognized to represent the nation's 
heritage, tradition, or values or is widely recognized to represent 
important national cultural, religious, historical or political 
significance; (2) asset's primary purpose is to memorialize or 
represent some significant aspect of the nation's heritage, tradition, 
or values, and to serve as a point of interest for visitors and 
educational activities; (3) if asset were successfully attacked, it 
would damage the American psyche and/or international confidence in the 
United States; and (4) asset is a monument, physical structure, or 
geographic site. 

[37] Consequence categories include casualties, economic impact, and 
length of disruption. 

[38] In addition to the Inspection Service, USPS also has an Emergency 
Preparedness group that works in close conjunction with the Inspection 
Service to integrate emergency preparedness training and awareness from 
an operational perspective. 

[39] Inspection Service officials told us that they chose 800 as the 
threshold score because they wanted to further review the security of 
the top 10 percent of the most vulnerable facilities. When this 
performance measure was implemented, the top 10 percent of most 
vulnerable facilities scored above 800. While this threshold remains 
the same today, the threshold score may decrease or increase over time 
due to implementation of countermeasures and changes in risk elements. 
To date, the Inspection Service has decided not to change the threshold 
score in order to keep the scoring methodology consistent. 

[40] In its Strategic Transformation Plan 2006-2010, USPS has 
identified four strategic goals: (1) generate revenue; (2) reduce 
costs; (3) achieve results with a customer-focused, performance-based 
culture; and (4) improve service. 

[41] VA officials noted that the majority of the space occupied by VA's 
Veterans Benefit Administration is in GSA-held buildings. As such, FPS 
is responsible for security at these facilities. 

[42] VA officials noted that most Veterans Health Administration 
buildings are designed for maximum public access and therefore do not 
have magnetometers or metal detectors, so such tests are not conducted 
in those facilities. In addition, many Veterans Benefit Administration 
facilities are in GSA buildings, so FPS is responsible for providing 
security and conducting related tests. 

[43] National Institute of Standards and Technology, Security Metrics 
Guide for Information Technology Systems, NIST Special Publication 800- 
55 (July 2003). 

[44] GAO-06-91. 

[45] DHS released the first Draft NIPP for public comment in November 
2005. In January 2006, DHS released a revised Draft NIPP that 
incorporated some of the comments it had already received. 

[46] As mentioned earlier, HSPD-7 establishes a national policy for 
federal departments and agencies to identify and prioritize U.S. 
critical infrastructures and key assets so that they can be protected 
from terrorist attacks. 

[47] See GAO-05-49. 

[48] The State Homeland Security Program and Urban Areas Security 
Initiative grants can be applied to a number of homeland security 
efforts, including facility protection. See U.S. Department of Homeland 
Security, Fiscal Year 2005 Homeland Security Grant Program, Program 
Guidelines and Application Kit. 

GAO's Mission: 

The Government Accountability Office, the investigative arm of 
Congress, exists to support Congress in meeting its constitutional 
responsibilities and to help improve the performance and accountability 
of the federal government for the American people. GAO examines the use 
of public funds; evaluates federal programs and policies; and provides 
analyses, recommendations, and other assistance to help Congress make 
informed oversight, policy, and funding decisions. GAO's commitment to 
good government is reflected in its core values of accountability, 
integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains 
abstracts and full-text files of current reports and testimony and an 
expanding archive of older products. The Web site features a search 
engine to help you locate documents using key words and phrases. You 
can print these documents in their entirety, including charts and other 
graphics. 

Each day, GAO issues a list of newly released reports, testimony, and 
correspondence. GAO posts this list, known as "Today's Reports," on its 
Web site daily. The list contains links to the full-text document 
files. To have GAO e-mail this list to you every afternoon, go to 
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order 
GAO Products" heading. 

Order by Mail or Phone: 

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to: 

U.S. Government Accountability Office 

441 G Street NW, Room LM 

Washington, D.C. 20548: 

To order by Phone: 

Voice: (202) 512-6000: 

TDD: (202) 512-2537: 

Fax: (202) 512-6061: 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm 

E-mail: fraudnet@gao.gov 

Automated answering system: (800) 424-5454 or (202) 512-7470: 

Public Affairs: 

Jeff Nelligan, managing director, 

NelliganJ@gao.gov 

(202) 512-4800 

U.S. Government Accountability Office, 

441 G Street NW, Room 7149 

Washington, D.C. 20548: