Detailed Information on the
Federal Emergency Management Agency: U.S. Fire Administration Assessment

Perform an assessment of program using performance data and independent analysis to identify impediments and make aggressive recommendations to improve performance.

Ensure program is functioning in a manner consistent with current legislative and appropriations language.

The National Fire Programs Division should develop and publish for decision-makers guidance on mission scope and efforts to prevent redundancy with partner organizations.

Refine data collection processes by Q2 to procure updated fire-related events data, which then will be used in FY07 to update the trend analysis that is awaiting this data.

Determine by end-Q2 a clear timeframe to achieve programmatic goals. In FY07, USFA will develop annual targets and set up procedures to collect and report data on a timely annual basis.

Measure: Percentage reduction in the rate of loss of life from fire-related events from the 2000 baseline of 13.5 deaths per million population.

Explanation:

Measure: Majority of supervisors reporting that their subordinate is better prepared to respond to disasters and emergencies as a result of the training they received.

Explanation:

Measure: Maintain the cost per student/user day (factoring in 4% for inflation) to measure operating efficiency.

Explanation:

Measure: Increased level of student enrollments.

Explanation:

Measure: Increased use of USFA programmatic information and training materials.

Explanation:

Is the program purpose clear?

Explanation: The U.S. Fire Administration' (USFA) primary purpose is to reduce life and economic losses due to fire and related hazards. Through leadership, advocacy, coordination and support, the USFA serves the nation through collective partnerships with other federal, state and local agencies. The National Fire Prevention and Control Administration (later renamed the U. S. Fire Administration) was established by Public Law 93-498, (the Federal Fire Prevention and Control Act of 1974). From its inception, the USFA was charged with: 1) reducing the Nation's losses caused by fire through better fire prevention and control; 2) supplementing existing programs of research, training, and education, and 3) encouraging new and improved programs and activities by state and local governments. Over time, USFA has expanded its programs to address a range of hazards other than fire, and though the fire service is an integral part of 'all hazards' response, this broader mission creates the potential for diverted resources and program overlap.

Evidence: America Burning www.usfa.fema.gov/downloads/pdf/ab_ch1.pdf, America Burning/Recommissioned www.fema.gov/news/newsrelease.fema?id=9991, Public Law 93-498 16 (http://thomas.loc.gov/cgi-bin/bdquery/z?d093:SN01769:%7CTOM:/bss/d093query.html%7C, FYHSP FY2008-2012 Budget, USFA Mission Statement http://www.usfa.fema.gov/about/orgchart/fireadmin.shtm

Does the program address a specific and existing problem, interest, or need?

Explanation: Though the number of fire-related deaths has declined by more than 60 percent in the last 30 years, the fire problem today in the United States, on a per capita basis, continues to be one of the worst in the industrialized world. Thousands of Americans continue to die each year in fires, tens of thousands of people are injured, and property losses continue to reach in the billions of dollars. The Fire Administration orients current programs based on two major findings from a 1991 report entitled "America Burning: Recommissioned" from the National Commission on Fire Prevention and Control. The report called for a substantial role by the federal government in funding and technical support for known loss reduction strategies, and concluded that firefighters and their communities would benefit if their approach to avoiding loss from fires was equal to the dedication shown in firefighting and rescue operations.

Evidence: National Center for Health Statistics http://www.cdc.gov/nchs/products/pubs/pubd/hestats/finaldeaths03/finaldeaths03.htm; http://www.everyonegoeshome.org/PDF/NIST_FINAL_SymposiumReport.pdf, National Fire Protection Association (NFPA) www.nfpa.org , USFA/NFPA Fire Service Needs Assessment Report http://www.usfa.fema.gov/downloads/pdf/publications/fa-240.pdf , http://www.usfa.fema.gov/downloads/pdf/publications/fa-303-508.pdf , USFA Firefighter Fatalities Database http://www.usfa.fema.gov/applications/ffmem/tally_report.jsp, U.S. Census Bureau Population Statistics http://www.usfa.fema.gov/applications/census/, World Fire Statistics Organization Data http://www.genevaassociation.org/WFSC.htm, America Burning www.usfa.fema.gov/downloads/pdf/ab_ch1.pdf, America Burning/Recommissioned www.fema.gov/news/newsrelease.fema?id=9991, Public Law 93-498 (http://thomas.loc.gov/cgi-bin/bdquery/z?d093:SN01769:%7CTOM:/bss/d093query.html%7C, Fire in the United States http://www.usfa.fema.gov/statistics/reports/pubs/fius13th.shtm, USFA Operational Objectives

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: The USFA works to reduce the Nation's losses caused by fire through research, training, and education on better fire prevention and control; encourages new and improved programs and activities by state and local governments; and delivers other programs on emergency response and incident management. The USFA develops and delivers national level training, education, and prevention programs that are intended for replication and use at the state/local and tribal level. While the USFA's National Fire Academy support to the Nation's 50 state fire training systems represents some supplanting of state and local responsiblity, duplicaton of existng fire-related training is reduced by focusing on mid and upper level officers. Howeve, in expanding it's mission to other 'all-hazards' missions and emergency services, USFA is duplicative of similar programs in other DHS and Federal entitites. The USFA's National Fire Programs Division provides expertise in the areas of fire prevention and suppression technologies, incident management, leadership and advanced managerial skills, hazardous materials response, emergency medical services and terrorism; but such expertise is found in many other Federal, state, and local programs. The USFA's National Fire Incident Reporting System (NFIRS) is based on the separate but complementary roles played by Federal, state and local agencies, and combines information from these sources to form a national system that does not exist elsewhere. Similarly, the creation of the Fire Department Census filled the absence of an available public reference. The Hotel-Motel National Master List identifies places of public accommodation that meet specific fire safety requirements. This list is created in cooperation with State and the Authority Having Jurisdiction and is the single consolidated national list. Additionally, to avoid duplication and to leverage existing research, USFA studies are coordinated with the Department of Homeland Security's Science and Technology Directorate, national fire service organizations, established research laboratories, and the National Fire Service Research Agenda Symposium last held in 2005.

Evidence: PL 93-498 (http://thomas.loc.gov/cgi-bin/bdquery/z?d093:SN01769:%7CTOM:/bss/d093query.html%7C, Fire Department Census http://www.usfa.fema.gov/applications/census/, America Burning www.usfa.fema.gov/downloads/pdf/ab_ch1.pdf,, USFA Mission/Functions http://www.usfa.fema.gov/about/orgchart/fireadmin.shtm, HR1550 Report, Hotel/Motel Safety List http://www.usfa.fema.gov/applications/hotel/

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The USFA was established by Public Law 93-498, (the Federal Fire Prevention and Control Act of 1974). From its inception, the USFA was charged with: 1) reducing the Nation's losses caused by fire through better fire prevention and control; 2) supplementing existing programs of research, training, and education, and 3) encouraging new and improved programs and activities by state and local governments. Using data collection, public education, research and training, the USFA program delivers technical assistance and training programs with the goal of reducing the effect of fire and related emergencies by improving the delivery of state and local fire and emergency services through training, guidance and technical support.

Evidence: Public Law 93-498 (http://thomas.loc.gov/cgi-bin/bdquery/z?d093:SN01769:%7CTOM:/bss/d093query.html%7C, FYHSP FY2008-2012 Budget, USFA Mission and Functions Statement http://www.usfa.fema.gov/about/orgchart/fireadmin.shtm

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: The USFA program uses its resources to targeting programs for three groups that it considers to be at high risk: 1) the young, 2) the aged, and 3) firefighters. Examples of targeted programming include: the development/delivery of 46 different prevention and risk reduction courses/programs that target high risk populations. Some of these courses/programs are aimed at groups most susceptible to fires - the young and the elderly, and others are aimed at fire prevention in general. The command/control courses focus on emergency scene control and management including personnel resource accountability. Many USFA publications/technical reports also address high risk problem areas, as well as the national fire prevention campaigns, the issuance of State Fire Training Assistance Grants to the 50 State Fire Training Agencies to help increase the availability of National Fire Academy training at the State and local level, and the National Fire Academy's Volunteer Incentive Program that includes intensive 6-day educational opportunities designed specifically for volunteer and combination department emergency services personnel. In addition, the USFA helps individual communities develop comprehensive all-hazard risk reduction plans, but it is unclear whether these are targeted to communities with greater risk or incidence of fire.

Evidence: USFA Operational Objectives, Juvenile Fire Setter Intervention Program http://www.usfa.fema.gov/subjects/youth/, Prepnet http://www.usfa.fema.gov/training/prepnet/, STFA Package, NFA Catalog http://www.usfa.fema.gov/training/nfa/catalog/

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: USFA has two specific long-term performance measures. The first measure focuses on the potential outcome of the USFA program, and the second, targets the specific impact of the USFA's training activities. The broader measure reflects the purpose and intent of the USFA program as a whol: to achieve reductions over time in the rate of loss of life from fire-related event. The secondary measure reflects the impact of the USFA's training activities in individual personnel. More specifically, this measure is based on Level III of Kirkpatrick's Four Levels of Evaluation and assesses the transfer of training skills/learning and the impact the training/learning has on the individual's ability to prepare for and respond to incidents. In addition to these measure, a long-term training target has been established to help increase the preparedness and response capabilities of the Nation's fire and emergency response personnel.

Evidence: FYHSP FY2008-2012 Budget, National Center for Health Statistics http://www.cdc.gov/nchs/products/pubs/pubd/hestats/finaldeaths03/finaldeaths03.htm; National Fire Protection Association www.nfpa.org, Kirkpatrick's Four Levels of Evaluation http://coe.sdsu.edu/eet/articles/k4levels/index.htm, National Fire Academy Level III Training Evaluation Data http://www.usfa.fema.gov/training/nfa/evaluation/, NFA Evaluation Questionnaires, USFA Firefighter Fatalities Database http://www.usfa.fema.gov/applications/ffmem/tally_report.jsp,.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The reduction in the rate of loss of life from fire-related events is measured and reported on an annual basis versus a baseline, and can be calcuated either by comparing the total number or the per capita rate. Trends in the data over a ten year period (1994-2003) show that fires have declined by 21%, civilian deaths have declined by 22% and injuries by -33%. The 2006 target for the percentage reduction in the rate of loss of life from fire-related events from the 2000 baseline of 3,809 per year is 18%. Essentially, USFA's target is a --25% reduction over ten years, rather than the -22% trend, accelerating the reduction by 14%. To date, the actual percentage reduction in the rate of loss of life from fire-related events from the 2000 baseline is 9% based on 2000-2002 National Center for Health Statistics Mortality Data. The reported decrease of 9% represents the change from 2000 (baseline year) through 2002 (the latest year for which data are available). However, the unvailability of more recent data makes updating this comparison difficult. The secondary measure reflects the impact of the USFA's training activities and more specifically, the transfer of training skills/learning and the impact the training/learning has on the individual's ability to prepare for and respond to disasters or other emergencies. The National Fire Academy (NFA) has established a standard of 80% as the target for all supervisors reporting that their subordinate's job performance has improved as a direct result of the NFA training. In FY2005, the annual reporting included 82% of supervisors reporting that their subordinate's job performance had improved as a direct result of NFA training. The USFA's training programs target chief officers which represent 300,000 of the Nation's 1.2 million fire service personnel. USFA programs contribute to training more than 70,000 fire and emergency response personnel annually. The program has suggested a long-term goal of re-training the entire chief officer population of 300,000 on an annual basis, which is ambitious but unrealistic and potentially duplicative.

Evidence: : National Center for Health Statistics http://www.cdc.gov/nchs/products/pubs/pubd/hestats/finaldeaths03/finaldeaths03.htm, Kirkpatrick's Four Levels of Evaluation http://coe.sdsu.edu/eet/articles/k4levels/index.htm, National Fire Academy Level III Training Evaluation Data http://www.usfa.fema.gov/training/nfa/evaluation/, NFA Evaluation Questionnaires, U.S. Census Data http://www.usfa.fema.gov/applications/census/, U.S. Census Bureau http://www.census.gov/popest/national/, USFA Firefighter Fatalities Database http://www.usfa.fema.gov/applications/ffmem/tally_report.jsp, USFA Annual Report, FYHSP FY2008-2012 Budget, NFA FY 2005 Program Participation Summary,

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The USFA has established a variety of annual measures that aim to reflect long-term outcome and purpose of its mission. Annual output performance measures that directly track to USFA activities include: student enrollments, the majority of chief officers using and applying specific technical program information on the job, and a continued high level of student satisfaction with training materials. USFA has proposed measures that track fire deaths among high-risk groups: 1) the young, 2) the aged, and 3) firefighters. Annual USFA programs can be presumed to have a clear, if indirect, relation to firefighter mortality, and such data is collected annually. However, it is unclear how or if USFA's annual activities can have a direct impact on fire mortality the young and elderly, and there are significant lags in data collection that make it difficult to establish such linkages.

Evidence: USFA Operational Objectives, Kirkpatrick's Four Levels of Evaluation http://coe.sdsu.edu/eet/articles/k4levels/index.htm, National Fire Academy Level I and Level III Training Evaluation Data http://www.usfa.fema.gov/training/nfa/evaluation/, NFA Evaluation Questionnaires, NFA FY 2005 Program Participation Summary, NFPA Survey Results www.nfpa.org, National Center for Health Statistics Mortality Data http://www.cdc.gov/nchs/products/pubs/pubd/hestats/finaldeaths03/finaldeaths03.htm,, U.S. Census Bureau Population Estimates http://www.census.gov/popest/national, USFA Firefighter Fatalities Database http://www.usfa.fema.gov/applications/ffmem/tally_report.jsp

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The USFA uses National Center for Health Statistics mortality and US Census data as its baseline and has established the following targets: Reduce the loss of life from fire by 15 percent: 1) by reducing by 25 percent the loss of life of the young, 2) by reducing by 25 percent the loss of life in the aged, and 3) by reducing by 25 percent the loss of life of firefighters (measured in absolute numbers). However, USFA has not determined a clear timeframe to achieve these goals, developed consistent annual targets, or set up procedures to collect and report data on a timely annual basis. An increase of 3 percent, or 2,100, has been targeted as the increase in student enrollments; a target of 95 percent has been set as the student satisfaction level (Agree/Strongly Agree) with USFA training courses, and a75/75 percent target has been established for the number of students (75 percent) using/applying (75 percent) of the specific technical program information they received once they return to the job.

Evidence: National Center for Health Statistics http://www.cdc.gov/nchs/products/pubs/pubd/hestats/finaldeaths03/finaldeaths03.htm , U.S. Census http://www.census.gov/popest/national, USFA Firefighter Fatalities Database http://www.usfa.fema.gov/applications/ffmem/tally_report.jsp, Kirkpatrick's Four Levels of Evaluation http://coe.sdsu.edu/eet/articles/k4levels/index.htm, National Fire Academy Level I and Level III Training Evaluation Data http://www.usfa.fema.gov/training/nfa/evaluation/, NFA Evaluation Questionnaires, NFA FY 2005 Program Participation Summary, USFA Operational Objectives

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: While the USFA has solid relationships with an array of partners on promoting injury prevention and life-safety goals, it does not have a formal process or mechanism to ensure that program goals are clearly articulated. Rather, each program has a set of established partners with whom it colloaborates closelyu. Such relationships are strongest the National Fire Academy, through its long standing partnership with the Nation's State Fire Training System, and the delivery of training courses through state and local fire training systems and colleges and universities.

Evidence: PARADE http://www.usfa.fema.gov/subjects/fireprev/parade.shtm and TRADE Grant http://www.usfa.fema.gov/training/nfa/trade/, FESHE http://www.usfa.fema.gov/training/nfa/higher_ed/feshe_conf/, HMEP Curriculum Guidelines http://hazmat.dot.gov/training/state/hmep/hmepfact.htm, IAG U.S. Forest Service, SFTA Package, USFA Partnership List(s)

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: The USFA is a relatively small program, and uses annual external reports and periodic studies to strengthen and improve the effectiveness of its programs. The annual Board of Visitor's program examines the current status and overall effectiveness of all USFA programs. More infrequently, the American Council on Education review/accreditation recommendations for NFA courses that are used by students seeking college level credit; and a 2001 review of National Fire Academy courses and curricula.

Evidence: Kirkpatrick's Four Levels of Evaluation http://coe.sdsu.edu/eet/articles/k4levels/index.htm, National Fire Academy Level I and Level III Training Evaluation Data http://www.usfa.fema.gov/training/nfa/evaluation/, NFA Evaluation Questionnaires, Stakeholder's Report, NFA Board of Visitors Reports http://www.usfa.fema.gov/training/nfa/about/bov.shtm and http://www.fido.gov/facadatabase/, Board of Visitors Recommendations, TRADE Report, American Council on Education Findings https://www.acenet.edu/nationalguide//pdf/NG05_Introduction.pdf and https://www.acenet.edu/nationalguide/results/index.cfm?sid=26578 , HR1550 Report/Comprehensive Curriculum Review

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The USFA has integrated budget requests with performance objectives/measures. The FY 2007 Congressional Justification includes USFA's long-term performance goal and associated measures and targets. Allocating its resources at the division level, the USFA uses individual milestones and annual spending plans to track the progress of its activities. Roll-up and quarterly reporting at the Directorate level includes: performance measure reporting, tracking of red/yellow/green light priorities, budget status, issues and resource constraints.

Evidence: Fiscal Year 2006 Congressional Budget Justification

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: In 1999 the USFA began working on key elements of its organizational planning and culture by implementing a continuous improvement process through the work of its Action Plan committees. Individual committees were formed to review and validate issues surrounding: core mission, leadership, communication, staff development, advocacy, partnership and marketing. Senior managers and other USFA staff dedicated months of work drafting recommendations for each issue. To date 176 recommendations have been reviewed and addressed. More recently, the USFA's National Fire Programs Division began a strategic review of all projects and programs in order to retarget resources and to align its programs to better support emergent DHS and Preparedness priorities

Evidence: Summary Report/List of Approved Action Plan Committee Recommendations

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The USFA program annually consults with key state and fire service partners via formal stakeholders meetings. In addition, USFA regularly meets with its program partners, i.e., State Fire Training Agencies, Metropolitan Fire Departments (160 of the largest fire departments located in metropolitan areas having a minimum staffed strength of 400 fully paid career fire fighters), State Fire Marshals, national fire service groups, and colleges and universities. The USFA training development function also uses the National Fire Academy's comprehensive on-going training evaluation program to improve and update existing courses. This information is used to identify and address needed changes to program direction and budget allocations. In addition, the USFA has achieved a full integration of linking budget requests to performance objectives/measures and results through the use of the Administration's Planning, Programming, Budgeting and Execution System.

Evidence: Stakeholder's Report, Firefighter Fatality Reporting http://www.usfa.fema.gov/applications/ffmem/tally_report.jsp, Kirkpatrick's Four Levels of Evaluation http://coe.sdsu.edu/eet/articles/k4levels/index.htm, National Fire Academy Level I/II/III Training Evaluation Data http://www.usfa.fema.gov/training/nfa/evaluation/, NFA Evaluation Questionnaires, National Center for Health Statistics (NCHS) Data http://www.cdc.gov/nchs/products/pubs/pubd/hestats/finaldeaths03/finaldeaths03.htm National Fire Protection Association (NFPA) Survey Data http://www.nfpa.org/categoryList.asp?categoryID=953&URL=Research%20&%20Reports/Fire%20statistics/Trends, USFA Web Usability Studies, Fire Department Census Data http://www.usfa.fema.gov/applications/census/, Fire Service Needs Assessmenthttp://www.usfa.fema.gov/downloads/pdf/publications/fa-240.pdf , TRADE Reports, PARADE Reports,

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: While all USFA program managers are held accountable through the established employee performance rating process, this process does not incorporate explicit cost and schedule result, or the accomplishment of broader program perforance objectives.

Evidence: FEMA Form 30-60, Contract/Quality Assurance Plan Example (NFA Evaluation Center), PARADE http://www.usfa.fema.gov/subjects/fireprev/parade.shtm and TRADE Grant http://www.usfa.fema.gov/training/nfa/trade/, FESHE http://www.usfa.fema.gov/training/nfa/higher_ed/feshe_conf/, NFA Catalog http://www.usfa.fema.gov/training/nfa/catalog/, FEMA Form 75-5a

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: The USFA has established a solid reputation for executing its budget and effectively obligating its resources. USFA projects costs using a Staffing Plan Program and monitors the actual costs provided by the National Finance Center and posted in the Integrated Financial Management Information System (IFMIS) against projections. A travel plan is developed each year and funds are allocated based on the planned trips. Senior anagers rely on program managers to adhere to established Spending Plan guidelines and priorities and Acquisition/Procurement Planning Documents. Status of Funds Reports are monitored regularly to ensure that funds are being spent in a timely manner. A review of the status of funding is also included in the quarterly performance reviews. Over the years USFA has worked with Acquisition Management to obligate a significant portion of funding in the first and second quarters of the fiscal year, so that any funds returned to the accounts from contracts can be reobligated quickly. This process has resulted in USFA spending more than 95% of their budget for more than five years.

Evidence: Financial Management Resource Reports, USFA Spending Plan, Funds Disposition Report

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The USFA has established a training cost-efficiency ratio (cost per student/user hour of training program delivered) as its efficiency measure. Costs include course development, instructional delivery, travel stipend reimbursement, evaluation, and other direct contract and miscellaneous instructional delivery costs, as well as facility overhead such as student support services, security, and training facility utilities and maintenance. While there is not a systematic process to re-evaluate and lower costs for on-campus training, the USFA does explore and review opportunities to disseminate training through other means such as distance learning, distributed curricula, and train-the-trainer programs.

Evidence: USFA Traditional Training Cost Data Sheet, MOSS Cost Per User Sheet, FEMA Instruction/NETC 7900.1 - Administration of the Student Stipend Program, Prompt Payment Act http://fms.treas.gov/prompt/index.html

Does the program collaborate and coordinate effectively with related programs?

Explanation: The USFA generally has a stronger relationship with non-Federal programs than with related Federal programs. Though no longer in FEMA, USFA has not revised it's mission statement to reduce overlap with ongoing FEMA roles and responsiblities for emergency managment. Though serving with other DHS training programs on the TRADE working group, DHS still lacks a consistent approach to assessing first responder training. There is little explicit collaboration with the National Interagency Fire Centers wildfire program, though DHS funds significant wildfire suppresion costs. At the direction of Congress, USFA plays a significant role developing criteria for the Assistance to Firefighters Grant program, and the resulting criteria vary significantly from broader DHS grant policy priorities.

Evidence: SFTA Package, , Interagency Agreements - U.S. Forest Service, EMR-ISAC (public private sector); http://www.usfa.fema.gov/subjects/emr-isac/ These partners include, but are not limited to: National Fire Protection Association (NFPA); International Association of Fire Chiefs (IAFC) and its Volunteer and Chief Officers Section (VCOS); Fire Safety Council (FSC); International Association of Fire Fighters (IAFF); National Volunteer Fire Council (NVFC); National Association of Hispanic Firefighters (NAHF); International Association of Black Professional Fire Fighters (IABPFF); National Association of State Fire Marshals (NASFM); International Fire Marshals Association (IFMA); Society of Fire Protection Engineers (SFPE); International Association of Arson Investigators (IAAI); Congressional Fire Services Institute (CFSI); Fire Corps; State, County and Local Fire Organizations; Department of Homeland Security Grants and Training; Federal Emergency Management Agency (FEMA); Corporation for National and Community Services (CNCS); Centers for Disease Control (CDC); Indian Health Service (IHS); National Institute of Standards and Technology (NIST); Bureau of Alcohol, Tobacco, and Firearms (BATF); Federal Bureau of Investigation (FBI); International Fire Service Training Association (IFSTA); Consumer Product Safety Commission (CPSC); Women in the Fire Service (WIFS); National Wildland Coordinating Group (NWCG), Offices of State Governors and State, County, and Local Emergency Management and Law Enforcement Offices; Safe Kids Worldwide; Home Safety Council (HSC); American Red Cross (ARC); Various Technical Colleges and Fire Academies; Underwriter's Laboratory (UL); TYCO; Insurance Services Office (ISO); Home Fire Sprinkler Association (HFSA); National Fire Sprinkler Association (NFSA); Factory Mutual (FM); International Code Council (ICC); American Academy of Pediatrics (AAP); ZERO to THREE; National Media Corps; and others;

Does the program use strong financial management practices?

Explanation: USFA uses the Federal Emergency Management Agency's (FEMA) financial management system, the Integrated Financial Management Information Systems (IFMIS). IFMIS allows USFA to allocate funds to the individual program areas. The quarterly allocations are based on when obligations will be awarded. The quarter of allocation is included for each action identified on the USFA Spending Plan. Each program office has an internal tracking system which is reconciled with IFMIS. The program head's approval is required before funds are certified and project officer's approval is required before payments are disbursed. USFA uses FEMA's IFMIS system but has its own financial office for obligating and disbursing funds. USFA's financial office monitors the certification, obligation, and payment of all USFA's funds. The review of open obligations is an annual process. There is a definite separation of duties and a system of checks and balances in our financial process. The financial process from certification to payment approval requires a minimum of three individuals.

Evidence: Financial Management Resource Reports, USFA Spending Plan, Funds Disposition Report, IFMIS Access Forms

Has the program taken meaningful steps to address its management deficiencies?

Explanation: [How is A-123 a major issue?] The Office of Management and Budget (OMB) issued Circular A-123, making Federal managers responsible for "establishing and maintaining internal control to achieve the objectives of effective and efficient operations, reliable financial reporting, and compliance with applicable laws and regulations." Under OMB A-123, Federal managers are required to "take systematic and proactive measures to develop and implement appropriate, cost-effective internal control for results-oriented management." The USFA established its own Internal Control Review Committee in the early 1990's. The committee is comprised of representatives from each USFA program area. Each USFA "assessable" area is reviewed in accordance with the committee's schedule and independent recommendations relative to improvements in efficiency and effectiveness are forwarded to USFA senior managers for action.

Evidence: OMB Circular A-123, FEMA/DHS Instructions, Internal Controls Review Committee Reports/Recommendations, Property Management (Instruction/Review)

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The reduction in the rate of loss of life from fire-related events is measured and reported on an annual basis, though final data may lag by 2-3 years. Data available from 2000, when the most recent baseline was set, to 2003 indicate that fire-related deaths are declining at at rate of X percent annually. If this trend continues, fire deaths will have been reduced by at least 25% by 2010. The second long-term performance measure reflects the impact of the USFA's training activities; more specifically, the transfer of training skills/learning and the impact the training/learning has on the individual's ability to prepare for and respond to disasters or other emergencies. Eights years of data shows that on average 87 percent of supervisors of students receiving National Fire Academy training report that their subordinate's job performance improved as a direct result of NFA training.

Evidence: Fire in the U.S. Report http://www.usfa.fema.gov/statistics/reports/pubs/fius13th.shtm, NFPA Reports www.nfpa.org, National Center for Health Statistics Mortality Data http://www.cdc.gov/nchs/products/pubs/pubd/hestats/finaldeaths03/finaldeaths03.htm, http://www.usfa.fema.gov/statistics/quickstats/, USFA Firefighter Fatality Database, http://www.usfa.fema.gov/applications/ffmem/tally_report.jsp, Kirkpatrick's Four Levels of Evaluation http://coe.sdsu.edu/eet/articles/k4levels/index.htm, National Fire Academy Level III Training Evaluation Data http://www.usfa.fema.gov/training/nfa/evaluation/, NFA Evaluation Questionnaires

Does the program (including program partners) achieve its annual performance goals?

Explanation: The 2006 target for the percentage reduction in the loss of life from fire-related events from the 2000 baseline of 3,809 is 18%, or 3124 fatalities. The current reported actual percentage reduction in the rate of loss of life from fire-related events from the 2000 baseline is 9% based on 2000-2002 National Center for Health Statistics Mortality Data. The reported decrease of 9% represents the change from 2000 (baseline year) through 2002 (the latest year for which data are available). The steep reduction over three years is promising, but the absence of more recent data makes it difficuld to determine whether these improvements are being sustained.

Evidence: Fire in the U.S. Report http://www.usfa.fema.gov/statistics/reports/pubs/fius13th.shtm, Annual Report to Congress, Kirkpatrick's Four Levels of Evaluation http://coe.sdsu.edu/eet/articles/k4levels/index.htm, National Fire Academy Level III Training Evaluation Data http://www.usfa.fema.gov/training/nfa/evaluation/, NFA Evaluation Questionnaires

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The USFA has established a training cost-efficiency ratio (cost per student/user hour of training program delivered). Costs include course development, instructional delivery, travel stipend reimbursement, evaluation, and other direct contract and miscellaneous instructional delivery costs, as well as facility overhead such as student support services, security, and training facility utilities and maintenance. While the the cost-efficiency of on-campus training is expected to change, overall efficeincy may increase as technology is used to broaden delivery of training programs.

Evidence: FYHSP Budget - Efficiency Measure, USFA Traditional Training Cost Data Sheet, MOSS Cost Per User Sheet, NFA FY 2005 Program Participation Summary

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: While USFA fills a unique role in the Federal government, it's individual training and research programs are directly comparable to efforts in related programs such as the National Interagency Fire Center, National Fire Protection Association, state and local fire training academies, or other Federal training programs. For example, DHS has not developed a consistent methodology for evaluating the peformance and efficiency of various training programs. USFA data and research programs encounter significant lags in the collection and publication of data, but data collection programs in DOJ and HHS face similar challenges.

Evidence: USFA Mission Statement http://www.usfa.fema.gov/about/orgchart/fireadmin.shtm

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: Although the USFA is a relatively small program it has several ongoing methods of independent program assesment and oversighnt. The annual Board of Visitor's program review examines the current status and overall effectiveness of all USFA programs. The American Council on Education reviews and accredits recommendations for NFA courses that are used by students seeking college level credit. The 2001 Congressionally mandated review of National Fire Academy courses found that each curriculum area is up-to-date and not duplicative or redundant of courses offered elsewhere.

Evidence: Kirkpatrick's Four Levels of Evaluation http://coe.sdsu.edu/eet/articles/k4levels/index.htm, National Fire Academy Level I and Level III Training Evaluation Data http://www.usfa.fema.gov/training/nfa/evaluation/, Stakeholder's Report, NFA Board of Visitors Reports http://www.usfa.fema.gov/training/nfa/about/bov.shtm and http://www.fido.gov/facadatabase/, Board of Visitors Recommendations, TRADE Report, American Council on Education Findings https://www.acenet.edu/nationalguide//pdf/NG05_Introduction.pdf and https://www.acenet.edu/nationalguide/results/index.cfm?sid=26578 , HR1550 Report/Comprehensive Curriculum Review