Detailed Information on the
Federal Emergency Management Agency: Grants and Training Office Assistance to Firefighters Grants Assessment

Increase outreach (e.g., conference workshops, regionally based workshops) to applicants in an effort to encourage the submission of grant requests that are risk oriented and reflect regional strategies (e.g., interoperability projects), and thus more closely align the grant activities to the DHS National Preparedness Goal and Target Capabilities List.

Establish a continuing strategic planning process to guide the evolution of the program. The process will involve collaboration with America's Fire Service and other program partners and provide an appropriate means by which the partners can have meaningful input into the strategic direction of the program.

Improve program transparency by examining grant award, management and results data; and then defining and implementing cost effective/efficient methods of compiling and presenting the information to the public.

Measure: Percent reduction in firefighter injuries in jurisdictions receiving AFG grants, compared to national average.

Explanation:This measure determines whether AFG funding is having an effect on fire departments who receive it. It shows that AFG is achieving one of its primary program outcomes: to reduce firefighter on-scene injuries. Each year, AFG will identify applicants who received an AFG award two years earlier. Those applicants will have included in their current and previous applications' data the preceding three years of firefighter injuries and their corresponding number of active firefighters. The ratio of firefighter injuries to the number of active firefighters over the two years will then be computed as follows: total number of firefighter injuries will be divided by the total number of available firefighters for each of the 3-year reporting periods from each application cycle, then the percent change in the ratio from the original application to the current application will be calculated. This figure is then compared to the percent change in the national average ratio of firefighter injuries to active firefighters as reported by the National Fire Prevention Association across the same periods. The two figures are reported side by side to show the effect that AFG funds are having on firefighter injury rates compared to the national injury trend; an AFG number indicating a larger improvement (or smaller deterioration) in the injury ratio among AFG award recipients relative to that of the NFPA national average is desired. A comparison indicating that the injury rate for AFG jurisdictions improved less (or worsened more) than the national average would imply lack of effectiveness. The years 2003 and 2006 are used for comparison because of the way the question is asked in the applications. From 2003 - 2005 the applicants were asked to report the number of injuries for the previous three years; in 2006 this question was changed to ask for the number of injuries occuring each year, for the previous three years. In order to ensure the integrity of the analysis 2003 is compared to 2006 to avoid any duplication in reported injuries. Based upon the demonstrated ratio the target is to reduce the ratio by 1.9% each year to 25.2% in 2009.

Measure: Percent reduction of fire-related civilian deaths in jurisdictions that received Assistance to Firefighter Grant funding compared to the national average

Explanation:This measure evaluates the number of civilian deaths from fire in jurisdictions receiving funding through the Assistance to Firefighters Grant program. It evaluates the outcomes from improvements in fire capabilities in jurisdictions receiving fire / fire and prevention safety grants by assessing annual reductions in civilian deaths from fire. The measure assesses the ultimate impact of fire grant funding on improving the safety of civilians from fire by comparing the Program's grant recipients' outcomes to the national average rate, calculated annually by the center for Disease Control and Prevention's Web-based Injury Statistics Query and Reporting System, a reporting tool that calculates the mortality rates for various causes of death on an annual basis. AFG will each year identify applicants who received an AFG award two years earlier. Those applicants will have included in their application's data the preceding year's number of fire-related civilian fatalities and the corresponding number of total population in the jurisdictions they serve. The percent change in the ratio of fatalities as a percentage of population served over the two years will then be computed as follows: total number of fire-related civilian fatalities will be divided by the total population living in those jurisdictions for each application year, then the percent change from year one to year three will be calculated. This figure is then compared to the national average rate of reduction in fire-related civilian fatalities reported annually by the CDC WISQARS, which will be derived using the same calculation steps and years. The two figures are reported side by side to show the effect that AFG funds are having on civilian fatality rates compared to the national trend; an AFG number indicating a larger improvement (or smaller deterioration) in the civilian fatality rate among AFG award recipients relative to that of the CDC's calculated national average is desired. A AFG number indicating AFG jurisdictions improved less (or worsened more) than the national average would imply a lack of effectiveness.

Measure: Percent of AFG awards with a training element.

Explanation:This measure evaluates the AFG Program's impact on the training of those core competency skills required of firefighters for the performance of their duties. It incorporates certified training courses funded by the Program as well as training related to the competent and safe operation of vehicles and equipment purchased by the Program and local jurisdiction wellness and fitness programs funded through the Program. The AFG Program has received significant anecdotal evidence that training directly impacts the safety of firefighters responding to incidents. A higher percentage of awards which include training is an improvement. The data set consists of the number of awards including certified training courses and / or the number of awards that require firefighters to train on equipment and vehicles purchased, and / or the number of awards that require firefighters to participate in wellness and fitness training as reported in the jurisdiction's application, close-out reports, and is verified through program monitoring activities. Supporting data is derived from applications and close-out reports submitted by award recipients through the AFG on-line grant management system. The AFG Program maintains the grant management system in which all applications, awards, progress reports, close-out reports, and monitoring results are maintained. The database can be searched annually to report the number of awards including AFG-funded training components. The long-term goal is to achieve and sustain not less than 90% of all awards to have a training component.

Measure: AFG grant dollars spent per firefighter trained.

Explanation:This measure determines the efficiency in which training is delivered to firefighters by the AFG program. This number should decrease over time as AFG achieves its goal of increasing the number of firefighters it trains on an annual basis. This improved efficiency helps achieve the long term outcome of reducing firefighter injuries and fire-related civilian deaths. This measure is calculated by determining the total number of firefighters trained via the amount of award dollars for certified training courses and / or firefighter training on equipment and vehicles purchased, and / or firefighter participation in wellness and fitness training as reported in jurisdictions' application and close-out (and verified through program monitoring activities). The AFG Program maintains the grant management system in which all applications, awards, progress reports, close-out reports, and monitoring results are maintained. The database can be searched annually to report the number of firefighters trained through AFG-funded training components.

Measure: Percent of grants for which a closeout report was received within 15 months of the award date.

Explanation:The program office will monitor the percent of awards whereby the recipients have completed all necessary grant actions and reporting requirements within the prescribed 12-month period of performance plus the three month expenditure period. This provides an indication of the efficiency of the grant management functions, the clarity of program guidance, and how well staff assist award recipients with grant administration. Rapid procurement and grant close-out directly indicates the rate at which equipment and training is procured and placed into service to meet the needs of the program and the communities it assists. A higher percentage indicates improvement. Sustainment at 90% is the goal. The online grant management system allows the Program Office to see on a daily basis whether or not required reports are submitted on time. Lists of delinquent grantees are regularly distributed to the ten regional offices for staff to follow-up on and ensure reports are submitted or provide technical assistance as needed.

Measure: Ratio of AFG funds awarded to population served.

Explanation:This measure calculates the average amount of AFG money awarded per civilian in jurisdictions receiving AFG awards. Because fire service is a benefit shared by all citizens who receive protection within a jurisdiction, this measure shows the efficiency of the AFG program at delivering enhanced protection to as many citizens as possible. This measure is computed annually and incorporates those grants that have been closed-out. The annual targeted reduction of 15 ?? is the average rate of change between FY 2003 and FY 2005, a period which showed steady decline in the amount awarded per civilian. The goal is to reduce the average cost-per-civilian at a sustained rate of 15?? per year. The data for this measure consists of the financial reporting received and maintained by the AFG online grant management system and will be computed annually as grants awards are closed out. All required reports are submitted through the AFG online grant management system; award recipient requests for drawdown of federal funds are managed through the AFG grant management system; financial expenditures are tracked through the IFMIS system and available to the AFG Program office in real-time. This measure uses the amount of a grant, which is double checked by the NPD's finance office as well as the AFG program, as well as the reported number of people living in jurisdictions, as reported by those jurisdictions. This information is verifiable using both census data and jurisdictional reporting to NFRS.

Is the program purpose clear?

Explanation: Explanation: The purpose of the Assistance to Firefighters Grant Program (AFG) is to help build the firefighting and emergency response capabilities of fire departments and non-affiliated emergency medical service (EMS) organizations to enhance their ability to protect the public and first responder personnel from fire and fire-related hazards. This is done through grants made directly to fire departments and EMS organizations under a governing statute that not only articulates concisely the overall purpose of the grants -- "??protecting the health and safety of the public and firefighting personnel against fire and fire-related hazards." -- but also states clear parameters for much of the program's operations, including: 1) providing technical assistance to the applicants; 2) achieving balances among department and community types in the awards; and 3) supplementing, not replacing, existing resources of the grantees.

Evidence: Section 33 of the Federal Fire Protection and Control Act of 1974, as amended. CFDA Program Description 97.044

Does the program address a specific and existing problem, interest, or need?

Explanation: Explanation: The AFG Program was created to address the fire problem in America. Compounding this problem was the fact that State and local governments were not always delivering the tools (including vehicles, training, and equipment) that fire departments needed to protect their firefighters and their communities. Further, fire department missions were expanding with the inclusion of new response areas such as hazardous materials accidents and emergency management services, yet their respective local governments were not increasing funding commensurate with increasing responsibilities. With regards to firefighter safety, the number of firefighter deaths as a ratio to the number of fire incidents has not improved much in recent decades. In 1996 the rate of reduction in the number of structure fire incidents in the United States appeared to flatten out. Although both structure fires and firefighter deaths have declined significantly in the past 25 years, the rate of firefighter deaths per fire has declined less dramatically. The leading causes of non-heart attack firefighter deaths in a structural fire are 1) getting lost inside a structure; 2) structural collapse, and 3) fire progress. The ratio of fatalities (due to these three causes) per 100,000 structural fires has actually increased since the early 1980s. With regards to the risk to the health and safety of the public from fire-related hazards the Centers for Disease Control and Prevention reports that deaths from fires and burns are the fifth most common cause of unintentional injury deaths in the United States and the third leading cause of fatal home injury. Statistics indicate the number of fatalities and injuries caused by residential fires has declined gradually over the past several decades, however residential fires continue to pose a significant public health problem. Additionally, the Department of Homeland Security is committed to improving preparedness for natural and man-made disasters by employing an all-hazards perspective. In its current place within DHS, the AFG program clearly complements the first three of the four part taxonomy used to analyze and communicate preparedness: prevention, protection, response, and recovery. A DHS review of state preparedness programs released February 10, 2006 identified that many States and urban areas self-reported that they are not well prepared for large-scale events. Recent events have drawn attention to the tasks required at all levels of government to prepare for large-scale events. Fire departments and EMS organizations are typically at the center of community and regional planning and response for such events. To help focus on the tasks required to accomplish this, DHS published its Target Capabilities List (TCL), of which many target capabilities are under the purview of the fire service and are being explicitly referenced in the FY 2007 AFG online application.

Evidence: National Academy of Public Administration, Assistance to Firefighters Grant Program: Assessing Performance, April, 2007; Department of Homeland Security, Target Capabilities List, found at: https://www.rkb.mipt.org/hspd8.cfm; Fahy, Rita F. (July, 2002). U.S. Fire Service Fatalities in Structure Fires, 1977-2000. National Fire Protection Association: Quincy, MA (Available at http://www.nfpa.org/assets/files/PDF/fffstructure.pdf) Centers for Disease Control and Prevention. Web-based Injury Statistics Query and Reporting System (WISQARS) [Online]. (2005). National Center for Injury Prevention and Control, Centers for Disease Control and Prevention (producer). Available from: URL: www.cdc.gov/ncipc/wisqars. [Cited 2006 Aug 21]. Centers for Disease Control and Prevention, Fire Deaths and Injuries: Fact Sheet, found at: http://www.cdc.gov/ncipc/factsheets/fire.htm

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: Explanation: Over the last four years, we estimate that between 40% and 50% of AFG funding has been awarded to support basic training, equipment, and vehicle requests which are instrumental to State and local firefighting budgets and are not allowable under potentially similar Federal grant programs. The AFG Program eliminates duplication at the local level by enforcing the statutory requirement that grant recipients minimally maintain their local budget allocations at the spending levels at time of application. This maintenance of expenditures requirement ensures the federal funds are supplementing local funds and not supplanting them. An average of 39% of AFG funding has gone toward mission areas which are allowable under other Federal grant programs including 2% for interoperable communications projects. However, at the state level, the AFG program office makes all applications that include requests either related to or directly for major communications systems or other state homeland security needs available for review by the State Administrative Agencies for the Homeland Security Grant (SHSGP) and Urban Area Security Initiative Grant Programs (UASI). By giving states the opportunity to review and comment, the AFG Program ensures there is no duplication of statewide or regional efforts. The requirement for State reviews is made clear to applicants and others in the annual program guidance. The FY2007 guidance is included as evidence. The organizational collocation of the AFG Program with other preparedness grant programs in the Office of Grants and Training of DHS has enhanced federal collaboration and coordination. HSGP program staff-participate in the AFG panel reviews; program guidance development is coordinated at every staff level; and potential duplication is largely avoided since the AFG Program funds basic items, such as turnout gear and hoses that are not allowable expenditures under the HSGP and UASI grant program guidelines. Those programs focus on the specialized equipment needed for counter-terrorism and weapons of mass destruction while the AFG funds more basic firefighter equipment. Similarly, AFG funds Firefighter I and Firefighter II training certification, but the HSGP and USAI programs do not. CEDAP and AFG share award information with one another in order to assure that duplicative funding does not occur. An MOU with the Departments of Agriculture and Interior addresses the minimal duplication of efforts regarding funding for wildland fire equipment and training. This duplication is eliminated by collaborating through their participation in AFG panel reviews and their technical advice to our program office.

Evidence: Award articles of agreement regarding supplanting, Article II MOU for Coordination and Cooperation of Fire Service Wildland Fire Assistance Programs Among the United States Department of Interior, United Stated Department of Agriculture, and the United States Department of Homeland Security; dated December 5, 2005. U. S. Department Profile through 2005, NAPA, 1Battery Park, Quincey, MA HQ correspondence (email) reflecting coordination process between AFG & Systems Support Division regarding the CEDAP program.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: DHS IG determined in 03 that program's application review process provides an equitable review. Further steps were taken to improve review process for program effectiveness: increasing consideration given to population-served and call volume in award decisions. This is cost effective - departments with the greatest usage rates are benefiting the most from the equipment acquired. Other actions on effectiveness: increased award cap to $2.7 million and reduced urban cost share from 30% to 20%, thus increasing AFG presence in urban and suburban areas. Beginning in FY07, urban and suburban departments may apply for multiple vehicle awards and the prohibition against second vehicle awards to all applicants has been lifted. Based on FY07 application statistics there is a 20% increase in number of applications submitted by departments serving populations of > 50,000. AFG includes in its governing statute requirements that it achieve a "balance" of awards to USR fire departments. Since program is competitive and open, it receives applications from USR areas that are reflective of profile of the U.S. Fire Service. AFG has approached this "balance" effort by ensuring that its balance of awards is comparable to balance of applications that it receives. AFG also operates with statutory cost share formulas and funding caps that are both based on size of populations served. Cost share was significantly reduced in FY05 reauthorization. AFG notes that, contrary to its experience prior to reauthorization, there have been no declinations of grant awards due to burden of cost share since the start of FY05. Also important to note is that funding caps have rarely been reached at any time in AFG's history. Therefore, there is no evidence that either funding caps or cost share formulas are providing any disincentives to scope or number of applications received from jurisdictions of various sizes. AFG maximizes efficiency in providing federal funds to assist local fire departments to acquire tools and resources that protect firefighters and communities they serve. A key factor in efficient funds usage is AFG's direct relationship with local fire & EMS department grantees. This eliminates many overhead and indirect costs associated with funds provided though multiple layers of government. AFG applicants typically do not request indirect or other M&A associated funds. The online system aids AFG in minimizing number of staff needed to properly manage grants and awards, and minimizing contractor support. Program staff is comprised of 12 FTE in HQ and 12 FTE located regionally. Regional staff is supplemented with 8 employees under intergovernmental personnel assignments. Financial staff is comprised 18 FTEs. This staff of about 50 people annually receives over 25,000 applications; conducts 400 applicant workshops; awards about 5,500 grants; processes 3 payment requests and 0.4 amendment requests per award; reviews performance reporting every six months; carries out a monitoring effort including universal desk reviews and numerous site visits; and pursues the closure of open grants. AFG uses two major support contracts - one for program/financial work including support to peer reviews and one for online system. Historically, "direct" charges for M&A budget for AFG have been between $15-18M. This cost range is due to fluctuating M&A requirements (e.g., in FY05 we introduced new programs (SAFER) and an AFG redesign. FY06 realized savings as start up costs were not necessary). Activities that enhance the AFG's efficiency include the electronic grant system, educational tutorials, and improved close-out rates. An example of resulting efficiency of this combination of efforts - from 01 to 04 AFG made 24,926 awards, of which 78% have been closed out. Improved monitoring methods not only ensure more grants are reviewed, but serve dual purpose of identifying program effectiveness in departments and communities they serve.

Evidence: FY 2007 AFG Program Guidance Close-out Rates Tutorial Activity Reports AFG Algorithm Worksheets available upon request (FOUO) DHS OIG, A Review of the Assistance to Firefighters Grant Program; dated September, 2003 (Page 13) Urban/suburban/rural AFG comparison chart FY2003-2007

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: Explanation: The program targets its efforts through an application scoring regime and panel review process which incorporates a comparison of department call volume and population served. In doing this the program makes an effort to factor in the increased risk associated with greater population density without compromising the legislative requirement of making its awards available to all fire departments and non-affiliated EMS responders through a competitive process. The program's application scoring regime and review process ensures its awards are focused on the program priorities established each year by the Fire Service during the criteria development workshop. The criteria development is driven by any changes prescribed by the appropriations legislation which is then compared to the needs identified by the Fire Service in general and the previous year's applicants. Where the legislation can accommodate those identified needs, the program priorities are modified. This criteria development process in concert with the risk-based analysis of the application reviews combine to ensure the awards are made in accordance with the program purpose and to the intended beneficiaries subject to statutory limitations. However, a combination of factors related to the program design (including application scoring priorities) and statutory requirements, have lead to certain types of departments (and as a result types of communities) being dramatically over-represented in both share of applications and awards relative to the percentage of the US population which they protect. Although program resources do not necessarily need to be distributed by population to qualify as effective, the mismatch is too dramatic to justify as consistent with efficient and direct targeting of resources to intended beneficiaries (firefighters and the public). Further, there is not sufficient evidence that program funding does not serve as a subsidy to activities that would have otherwise occurred.

Evidence: Criteria development workshop agenda for FY 2007 Criteria development workshop panel remarks for FY 2007 FY 2007 AFG Program Guidance FY 2007 Federal Register Rules and Notice of Funds Available Section 33 of the Federal Fire Protection and Control Act of 1974,as amended.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: 1) Percent reduction in fire-fighter injuries in jurisdictions receiving AFG grants, compared to national average. This measure determines whether AFG funding is having an effect on fire departments who receive it. It shows that AFG is achieving one of its primary program outcomes: to reduce firefighter on-scene injuries. Each year, AFG will identify applicants who received an AFG award two years earlier. Those applicants will have included in their current and previous applications' data the preceding three years of firefighter injuries and their corresponding number of active firefighters. The ratio of firefighter injuries to the number of active firefighters over the two years will then be computed as follows: total number of firefighter injuries will be divided by the total number of available firefighters for each of the 3-year reporting periods from each application cycle, then the percent change in the ratio from the original application to the current application will be calculated. This figure is then compared to the percent change in the national average ratio of firefighter injuries to active firefighters as reported by the National Fire Prevention Association across the same periods. The two figures are reported side by side to show the effect that AFG funds are having on firefighter injury rates compared to the national injury trend; an AFG number indicating a larger improvement (or smaller deterioration) in the injury ratio among AFG award recipients relative to that of the NFPA national average is desired. A comparison indicating that the injury rate for AFG jurisdictions improved less (or worsened more) than the national average would imply lack of effectiveness. 2) Percent reduction of fire-related civilian deaths in jurisdictions that received Assistance to Firefighter Grant funding compared to the national average. This measure evaluates the number of civilian deaths from fire in jurisdictions receiving funding through the Assistance to Firefighters Grant program. It evaluates the outcomes from improvements in fire capabilities in jurisdictions receiving fire / fire and prevention safety grants by assessing annual reductions in civilian deaths from fire. The measure assesses the ultimate impact of fire grant funding on improving the safety of civilians from fire by comparing the Program's grant recipients' outcomes to the national average rate, calculated annually by the center for Disease Control and Prevention's Web-based Injury Statistics Query and Reporting System, a reporting tool that calculates the mortality rates for various causes of death on an annual basis. AFG will each year identify applicants who received an AFG award two years earlier. Those applicants will have included in their application's data the preceding year's number of fire-related civilian fatalities and the corresponding number of total population in the jurisdictions they serve. The percent change in the ratio of fatalities as a percentage of population served over the two years will then be computed as follows: total number of fire-related civilian fatalities will be divided by the total population living in those jurisdictions for each application year, then the percent change from year one to year three will be calculated. This figure is then compared to the national average rate of reduction in fire-related civilian fatalities reported annually by the CDC WISQARS, which will be derived using the same calculation steps and years. The two figures are reported side by side to show the effect that AFG funds are having on civilian fatality rates compared to the national trend; an AFG number indicating a larger improvement (or smaller deterioration) in the civilian fatality rate among AFG award recipients relative to that of the CDC's calculated national average is desired. A AFG number indicating AFG jurisdictions improved less (or worsened more) than the national average would imply a lack of effectiveness.

Evidence: AFG Applicant-Reported Firefighter Injury Report AFG Applicant-Reported Civilian Fatality Report National Fire Prevention Association Longitudinal Study on FireFighter Injuries CDC WISQARS Reports on fire-related Civilian Fatalities DHS OIG Report recommendation to more narrowly focus Program goals; September, 2003 AFG Longitudinal Comparison to NFPA and CDC

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: Percent reduction in fire-fighter injuries in jurisdictions receiving AFG grants, compared to national average. AFG has calculated the percent reduction in firefighter injuries between the years 2002 and 2004; 2003 and 2005; and 2004 and 2006. For each pair of years, the previous three years of data is aggregated to determine the rate of firefighter injuries. AFG will strive to consistently beat the national average and has established a baseline of 15.5% reduction in the injury-to-firefighter ratio, which is the average reduction for the three pairs of years referenced above. For the years 2002 and 2004; and 2003 and 2005, we have also calculated the national average reduction in firefighter injuries. This number is reported alongside the AFG results to show improvements (or deteriorations) in context. Data on the national average reduction for years 2004 to 2006 is not reported because the 2006 data is not available at this time. AFG's long-term targets have been developed to show an improvement over time in AFG's ability to produce progressively greater outcomes through grant targeting and distribution, and to consistently beat the national average. However, the results of this measure are related to national trends in firefighter injuries. When large scale disasters strike, both the national average and the AFG jurisdictions results are going to be effected. Our annual goal is to reduce injuries at a greater rate than the national average while our long-term goal is to reduce the absolute injury-to-firefighter ratio excluding one-time events. Our quantifiable targets for the AFG program are: 2006 Actual: 44.3% / N/A National Average 2007: 15.5% 2008: 18% 2009: 21% 2010: 24% 2011: 25% 2012: 25% 2013: 25% A 25% reduction is the anticipated steady state. Target shall be adjusted . All data is corroborated with NFIRS and rates of change are compared to annual report published by the NFPA: "Firefighter Injuries in the United States", and "Fire Loss in the United States". Percent reduction of fire-related civilian deaths in jurisdictions that received Assistance to Firefighter Grant funding compared to the national average. AFG has calculated the percent reduction in fire-related civilian death ratio for AFG jurisdictions between the years 2002 & 2004; 2003 & 2005; and 2004 & 2006. For each of these pairs of years, the rate of fire-related civilian deaths to total population served is calculated from AFG grant applications. AFG has calculated a baseline of 7.8% reduction for this measure by finding the average of the three pairs of years. For the period between 2002 and 2004, the national average reduction in fire-related civilian death ratio was -0.3%, which is an increase. Data is not available for the national averages for fire-related civilian deaths for 2005 or 2006, but this information is forthcoming from the CDC. It will be updated and reported in PARTWeb as it becomes available, two years after the end of the calendar year. As the jurisdictions that AFG works with change over time, our long-term targets have been developed to show an improvement over time in AFG's ability to produce progressively greater outcomes through grant distribution and targeting. However, the results of this measure are related to national trends in fire-related civilian deaths. When large scale disasters strike, both the national average and the AFG jurisdictions results are going to be effected. Our annual goal is to reduce the fire-related civilian death ratio at twice the rate of the national average while our long-term goal is to reduce the absolute ratio excluding one-time events. Our quantifiable targets for the program are: 2006 Actual: 7.2% / National Average not available 2007: 7.8% 2008: 9% 2009: 11% 2010: 13% 2011: 15% 2012: 17% 2013: 19% A 20% reduction is the anticipated steady state. We will adjust this target as more data becomes available.

Evidence: AFG-Reported Firefighter Injury Rates Overview AFG-Reported Civilian Deaths Overview www.NFPA.org AFG Longitudinal Comparison to NFPA and CDC

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: Percent of AFG awards with a training element. This measure evaluates the AFG's impact on the training of firefighters for the performance of their duties. It incorporates training courses, wellness and fitness programs, and training related to vehicle operation and equipment purchased by the AFG. There is significant anecdotal evidence that training impacts the safety of firefighters. A higher percentage of awards that include training is an improvement. The data set consists of the number of awards including training courses, the number of awards that require firefighters to train on equipment and vehicles, andthe number of awards that require firefighters to participate in wellness and fitness training as reported in the application, close-out reports (as verified through AFG monitoring activities). Data is derived from applications and close-outs submitted by grantees through the on-line system. The long-term goal is to achieve and sustain not less than 90% of all awards to have a training component. AFG grant dollars spent per firefighter trained. This measure determines the efficiency in which training is delivered to firefighters by the AFG. This number should decrease over time as AFG achieves its goal of increasing the number of firefighters it trains on an annual basis. This improved efficiency helps achieve the long term outcome of reducing firefighter injuries and fire-related civilian deaths. This measure is calculates the total number of firefighters trained via the amount of award dollars for training courses, firefighter training on equipment and vehicles and firefighter participation in wellness and fitness training as reported in jurisdictions' application and close-out (and verified through program monitoring activities) Percent of grants for which a closeout report was received within 15 months of the award date. AFG monitors the percent of awards which have completed all necessary requirements within the 12-month period of performance plus the 90 day expenditure period. This provides an indication of the efficiency of the grant management functions. Rapid procurement and grant close-out directly indicates the rate at which equipment and training is procured and placed into service to meet the needs of the program and the communities it assists. A higher percentage indicates improvement. Sustainment at 90% is the goal. The online system is queried regularly for lists of delinquent grantees that are then distributed to the regional staff for follow-up. Ratio of AFG funds awarded to population served. This measure calculates the average amount of AFG money awarded per person in jurisdictions receiving AFG awards. This measure shows the efficiency of the AFG at delivering enhanced protection to as many people as possible. This measure is computed annually and includes grants that have been closed-out. The annual targeted reduction of 15 ?? is the average rate of change between FY03 and FY05. The data for this measure comes from the financial reporting received and maintained by the AFG online system. All required reports and drawdown requests are submitted through the online system; financial expenditures are tracked through the IFMIS system and available to the AFG in real-time. This measure uses the grant amount, which is double checked by NPD's finance office as well as the AFG, as well as the reported population, reported by those jurisdictions.

Evidence: Mapping Critical Task Exercise Analysis to the Firefighting Operations/Support Capability Report of Close out Reports Received w/in 15 Months of Award Cost Benefit Analysis of AFG Awards: Population to Award Amount Ratio AFG Longitudinal Comparison to NFPA and CDC

Does the program have baselines and ambitious targets for its annual measures?

Explanation: Percent of AFG awards with a training element. This measure uses the results from FY06 as a baseline. The previous years do not capture training associated with equipment grants as the program did not ask for this data prior to '06. AFG calculated the percent of training awards in '06 that were for equipment grants and used this percentage to estimate the number of equipment-related awards that would have been present in '05 and '06. AFG found there was an estimated increase of 5% between the projected '05 number and '06 findings. The long term targets use this 5% increase to calculate out-year targets from the '06 baseline. 2006 Actual: 40.2% 07: 42.2% 08: 44.3% 09: 46.5% 10: 48.8% 11: 51.3% 12: 53.8% 13: 56.5% AFG grant dollars spent per firefighter trained. This measure uses the FY'06 results as the baseline, given that this is the first year that equipment based training is able to be reported. The AFG used the percent of equipment awards that included training and total firefighters trained in this way in FY'06 to estimate what the grant dollars spent per firefighter trained for FY'04 and FY'06. AFG used this estimate to calculate the targets for this measure, not as actual results. AFG determined that over the last three years, the cost decreased on average by 2%. Using '06 as a baseline, the targets show an annual 2% reduction. 2006 Actual: $4,081 07: $3,999 08: $3,919 09: $3,841 10: $3,764 11: $3,689 12: $3,615 13: $3,543 Percent of grants for which a closeout report was received within 15 months of the award date. This measure uses the figure of 78.2% as the baseline. This figure is the average of FY'02, '03, & '04 grant cycles. The actual rate for the FY'05 cycle will be computed; it is currently estimated to be 90% after all grants have closed. Because a 15-month period is being measured, there is a lag in annual reporting. The results for '06 will be from the FY'05 awards. The goal is to sustain at a rate of 90%. To reach this target the AFG instituted two simultaneous actions: 1) the deployment of the on-line grant management tutorial assist grantees with the administration of their awards; and, 2) a monitoring program with the implementation of desk reviews and vehicle-specific reviews. As more FY'05 awards reach their 15-month mark and regional personnel gain greater experience with these responsibilities this rate is expected to improve in FY'07. Our targets for the AFG program are: 2006: 90% 07: 90% 08: 90% 09: 90% 10: 90% 11: 90% 12: 90% 13: 90% Ratio of AFG funds awarded to population served. This measure calculates the average ratio of funds awarded to jurisdictions divided by the population served in those jurisdictions, showing the cost efficiency of the AFG. It is computed annually and incorporates those grants that have been closed-out. The baseline for this measure is $4.97 which is the average of the last three years of data ('04-'06). AFG is using the average decrease of the period '03-'05 to set annual targets. FY'06 results will be set as the baseline. 2006 Actual: $5.72 07: $5.57 08: $5.42 09: $5.27 10: $5.12 11: $4.97 12: $4.82 13: $4.67

Evidence: Mapping Critical Task Exercise Analysis to the Firefighting Operations/Support Capability Report of Close out Reports Received w/in 15 Months of Award Cost Benefit Analysis of AFG Awards: Population to Award Amount Ratio AFG Longitudinal Comparison to NFPA and CDC

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: Applications that do not support the Program's priorities and long-term goals are not awarded. Award recipients agree to provide data that reflect the program's long-term goals, such as the number of firefighter injuries, civilian fatalities, and call volume. The program gives higher priority to applications focused on firefighters and members of the public at greatest risk, such as children and seniors. Further, applications for training or which include a training component receive greater consideration as there is evidence that this activity is the most effective at achieving the programs strategic goals.

Evidence: Grant Evaluation Plan FY 2006 Sample of Articles of Agreement Sample of Department Characteristics I & II

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: In September, 2003 the DHS OIG conducted a thorough review of the program. Actions for improvement based upon OIG recommendations include a more robust monitoring program and refinements to the program metrics to ensure focus on the purpose. The U.S. Fire Administration led a cooperative study of comparing the needs identified by the U.S. Fire Service in its FY 2001 applications to the AFG Program to those of FY 2005 in order to help determine both program impact and whether or not the identified needs were being addressed. This report was published in October, 2006. Additionally, the National Academy of Public Administration (NAPA) conducted a comprehensive strategic review and assessment of the AFG Program; recommendations for modification will be included in the final report to be published by NAPA in March, 2007. The HHS Center for Disease Control and Prevention is currently conducting an independent longitudinal study of the effectiveness of AFG-funded Fire Prevention and Safety activities. This study is examining several key aspects of the effectiveness of safety training programs designed for elderly and children populations and smoke alarm installation effectiveness. Study findings will be published in multiple phases beginning in September, 2007 through September, 2009.

Evidence: "A Review of the Assistance to Firefighters Grant Program"; DHS OIG; September, 2003 Matching Assistance to Firefighters Grants to the Reported Needs of the U.S. Fire Service; U.S Fire Administration, Department of Homeland Security, & National Fire Protection Association Assistance to Firefighters Grant Program: Assessing Performance; National Academy of Public Administration; January 17, 2007 (currently in draft) Statement of work: Inter-Agency Agreement between HHS, CDC and DHS, AFG.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The AFG Congressional Justification is primarily a narrative of what the AFG program does and how the requested funds will be used, and does not specifically tie the level of funding requested to either annual or long-term performance goals. Further, AFG portion of the Budget does not tie its resource request in an organized manner to broader DHS goals or national preparedness priorities. Although the CJ includes information on AFG's current accomplishments and planned accomplishments, these accomplishments are tied neither to program outcome and performance goals, nor to broader DHS preparedness goals. Also, the program's budget does not adequately outline how resource levels requested will achieve accomplishments. AFG is a separate appropriation and has its own chapter in the CJ, so the resource requests are clear and transparent. The request for the AFG appropriation is on the OMB website, which the public can access.

Evidence: FEMA AFG FY 2008 Congressional Budget Justification AFG Annual Reports: www.firegrantsupport.com

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: Several significant improvements to the application scoring process have been instituted to ensure all types of departments are competing equitably. The result of these changes should be the scoring process better addresses the funding of major population centers. Similarly, the grant guidelines have been modified to allow for multiple vehicle awards to urban and suburban applicants and repeat vehicle awards to rural applicants. In keeping with the DHS focus on all-hazards, risk-based funding, greater emphasis has been placed on evaluating the risk confronted by applicants as determined by call volume and population served. In 2004 the program began requiring applicants to report civilian fatalities as a result of fire. Now that this data has been collected for three years it is beginning to offer meaningful analysis for trends as a result of grant awards. Historically the AFG program has reduced civilian deaths, but the rate of change has stabilized. Simultaneously, one of the recommendations anticipated in the upcoming NAPA assessment of the program is that greater emphasis should be placed on fire prevention and safety. Program data appears to support this recommendation and methods of enhancing the Fire Prevention and Safety activities will be incorporated during the 2007 Criteria Development Workshop for implementation in the FY 2008 program guidance. In 2006 the AFG Program office entered into an inter-agency agreement with the HHS, Center for Disease Control and Prevention to conduct a longitudinal study of the effectiveness of AFG-funded Fire Prevention and Safety activities. Results will be used for future criteria development and program guidance.

Evidence: Application evaluation algorithm (FOUO) AFG Program Evaluation Plan AFG Civilian Fatality Data Analysis Assistance to Firefighters Grant Program: Assessing Performance; NAPA; April, 2007 Statement of work for IAA between AFG, DHS and CDC, HHS.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: Explanation: The program utilizes its online grant management system to collect performance data from grantees. This performance data includes information on amendments (i.e., scope of work changes and period of performance extensions), payment requests, semi-annual performance reports, ancillary documents associated with the administration of afore mentioned activities and close-out activities. This system also allows the program to list the details of purchases for the use of other entities, including its federal and state partners. Data collected from this system is used in the annual development of funding priorities established each year through a criteria development workshop. In this workshop nine national fire service organizations and associations representing the complete spectrum of the fire service, are briefed by program staff on the most recent appropriation changes from Congress and any recommendations received from OMB or any other federal entities. The participants then examine these in relationship to the needs of the fire service and develop recommendations for revisions to program implementation. The data collected from applicants is reviewed each year to ensure it is providing the information necessary to award funds in an equitable, risk-based manner and to allow the effective management of the program. For example, the requirement to report civilian fatalities and firefighter fire-ground injuries provides the baseline data for the program's long-term and annual goals. This data is collected with each application period, and AFG also tracks the performance of jurisdictions who have received awards over multiple years, allowing for the performance evaluation AFG grants over time. AFG also collects information from all applicants and awardees about the demographic characteristics of their jurisdictions. This allows AFG to analyze performance trends among the three different types of awardees: Rural, suburban, and urban. AFG has used these analyses to help inform its policy decisions, such as the decision to allow for multiple vehicle awards to urban jurisdictions, and the reduction of the Urban cost share floor by 1/3. AFG also collects department call volume and population served statistics, which combine to help determine risk when making grant award decisions. The AFG Program is constantly re-evaluating its processes and making system improvements. Untimely grantee reporting is corrected through assertive monitoring of grants via programmatic desk reviews of grant files. Regional staff have been provided the additional tools of regular grantee project performance reports to facilitate their ability to follow-up for corrective measures. These monitoring actions have been linked via DHS's pay for performance personnel system known as Human Capital Operational Plan to program staff individual performance reports. These individual goals are not only linked to program goals but are also linked to the division and agency goals.

Evidence: Percentage of close-outs by fiscal year Department Characteristics I & II AFG Policy and Procedure for Monitoring Desktop Review Report Vehicle Review Report Site Visit Report Electronic communication from Headquarters to regional personnel Regarding late close-out reports.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: At the federal level, program accountability rests with the Assistant Secretary, Office of Grants and Training and the AFG Director. Funds must be expended within two years and awards are held to a one-year period of performance. The award recipients are formally accountable for their respective AFG funds and certify this accountability at the time of accepting their respective award. During the course of the grant period performance reports are submitted every six months providing a status of grant activity. Whenever a recipient requests to drawdown funds the request is scrutinized to ensure purchases are allowable within the scope of the individual award. In the rare instances where award recipients are found to not be in compliance with program guidelines funds are returned to the Program. Grant awards have been cancelled due to the discovery of misleading information and past poor performance with the administration of AFG awards has prevented applicants from receiving new awards. In FY 2006 the AFG Program office began requiring advance payment bonds whenever Program funds were used as a down payment for vehicle purchases and performance clauses that specify a delivery date with financial penalties imposed for vendor failure to meet that date. The advance payment bonds insure federal money against loss in the event of vendor bankruptcy or contract breach and the performance bond ensures timely delivery. This requirement was written into the FY 2007 AFG Program Guidance for all vehicle purchases. The program office expects to expand it in FY 2008 to other projects, other than simple procurements, that traditionally take a long time for completion.

Evidence: Copies of demand letters for unauthorized expenditures Copies of award cancellation letters Copy of letter notifying award recipient of frozen funds pending results of monitoring visit Copies of performance bonds Copies of performance clauses Documentation from GMS regarding notification of poor performance and its impact on later applications

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: Explanation: Generally, the AFG appropriation is signed into law on October 1st. Program applications are made available by March of the following calendar year - approximately six months later. This incorporates criteria development, guidance development, IT system revision development, internal and OMB approval, and preparation of the Federal Register notice. After the application solicitation in March, applications are received and undergo four levels of review. Awards are made beginning in June and approximately 95% of all program dollars are awarded by November 30th, approximately one year subsequent to appropriation. Starting in FY 2005 the AFG was required to publish a Federal Register (FR) notice prior to making awards. In FY 2006 a delay in receiving DHS senior management approval of the notice was experienced due to a lack of an established vetting protocol subsequent to DHS' "2 SR" reorganization. The notice was submitted in May and remained unapproved until September. In 2006, the FR notice and the program guidance were not submitted on the same track (i.e., program guidance being submitted months before the FR notice). Once the program guidance was approved by DHS the AFG published the guidance erroneously believing senior level DHS personnel had provided it to OMB for vetting. This was another source of the delays that were experienced in 2006. In order to mitigate delays in the future, AFG has pushed up the criteria development process by approximately 16 weeks. By moving up this process earlier in the calendar year all processes described above will also be moved up in schedule thus building in buffer time so that application periods can be conducted and awards made on schedule and earlier. Additionally the AFG now seeks approval of both the FR Notice and the program guidance concurrently instead of on separate tracks. This process helps eliminate the conflict that may arise from one document being approved prior to the other as was the case in 2006. The program ensures that funds are spent for the intended purpose through its review and approval of requests for funds, and then verifies that the proper purchases have been carried out during the grant close-out review process. If in the course of conducting the close-out review it is found that expenditures were made for purposes other than those that were allowable or intended (i.e., deviation from grant scope of work without program approval) the grantee is held accountable and must return to the program the amount of these expenditures. Site visits (see evidence 3.1) by the program or financial office staff also verify purchases At the local, award recipient level, six month performance reports are submitted providing a status of grant activity as well as requests for drawdown. As these are submitted through the online grant management system the Program office is able to monitor the obligation of funds by award recipients and take the appropriate action. For example, a grant with no activity nine months into its twelve month period of performance would receive a contact from a regional fire program specialist.

Evidence: See evidence in 3.2 & 3.1 above. Criteria Development Reports 2004-2007 Federal Register Notices

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: AFG measures its application and award processes through two automated systems. These provide data necessary to rank order applications received, track progress through panel reviews, and record entire life-cycle of each award. These systems allow AFG staff to track and manage the number of awards closed out within 15 months of award date. IT solutions provide additional efficiencies with the deployment of on-line tutorials. To encourage competitive procurement practices grantees are required to follow local procurement practices or, to acquire at least two bids and take the lowest one unless an exception is documented. To validate applicants cost proposals the AFG conducts its own annual market research to establish acceptable price ranges for equipment. Application costs falling outside must be justified. As mentioned in 3.2, performance clauses are required of all vehicle purchase contracts to ensure firm delivery dates. Internal performance measures tied to DHS' pay for performance system have been incorporated to reinforce all management efficiencies. AFG staff have stated goals of increased performance. This correlates directly with actions of measuring and improving program efficiency and cost effectiveness. At a federal level the AFG tracks the percent of awards that are closed out within the allowable 15 months . Over the last four years, AFG has improved from closing out 77% to 80.2% of awards. AFG has set ambitious targets for this federal efficiency measure of 90% by the FY06 grant cycle. While AFG is allowed up to 5% of its appropriation for program administration it has spent on average only 2.6% in fiscal years 01 through 05. M&A as a percent of appropriation has been variable and on the surface appears not to indicate a trend of improvement in efficiency. Several factors must be considered when trying to establish improvement of M&A efficiency. In beginning years of the AFG FEMA absorbed some of the M&A costs. These costs were not taken out of the appropriation and thus give the appearance of initially low M&A expenditures. Another reason for increased and variable program M&A cost is due to legislative creation of the SAFER program. An appreciable increase in M&A costs were necessary to stand up this program. Additionally in order to ensure that the grants are administrated properly we bore and maintained more M&A costs. Quantitatively M&A costs have increased from 02 - 05/06. Post award management of grants was continuing while at the same time grants that had been awarded in years prior were now entering close-out. At this point the award and close-out cycle had reached a point of saturation. M&A expenditures have now reached a plateau that has been sustained. Qualitatively grants have grown in complexity. As the AFG has evolved, protocols have been established that are qualitatively proportional to maintain and achieve a greater level of efficiency. The online system aids the AFG in minimizing number of staff needed to properly manage total number of grants and awards, as well as minimizing contractor support. AFG staff is comprised of 12 FTE in Headquarters and 12 FTE regionally located. The regional program staff is supplemented with 8 employees operating under intergovernmental personnel assignments. The financial staff is comprised of 18 FTE. With this staff of ca. 50, the AFG annually receives over 25,000 applications; conducts 400 applicant workshops; awards about 5,500 grants; processes approximately 3 payment requests per award; 0.4 amendment requests; performance reports every six months; carries out a monitoring effort including desk reviews and site visits; and pursues closure of open grants. AFG utilizes two major support contracts - one for program/financial work including support to peer reviews and one for the online system. Typically the M&A budget has been between $15-18M.

Evidence: Program Management and Administration Funds (M&A) statistics FY07 Grant award agreement articles Market research guidelines FY07 First Quarter FYSHP Report AFG 15-Month Closeout Report

Does the program collaborate and coordinate effectively with related programs?

Explanation: AFG Program activities are regularly coordinated within the National Preparedness Directorate with those of the State Homeland Security Grants (SHSGP) and the Urban Area Security Initiative (UASI) Grants through weekly leadership meetings and informal daily staff collaborations. Coordination with CEDAP regarding efforts to prevent duplication of effort has also occurred. Staff members working with these programs and the Citizen Corps program participate in the AFG and Fire Prevention and Safety panel reviews each year. Regional staff coordinate their activities with those of the respective FEMA regional offices and State Administrative Agencies for the SHSGP and the UASI programs and are provided copies of AFG regional applications and all applications that included requests either related to or directly for major communication systems or other State Homeland Security needs to ensure coordination of programs at the State and regional level. The AFG requires that a state technical review occur on all applications that request CBRNE-related equipment or training or interoperable communication equipment. The State official must certify that the requests are consistent with the State Homeland Security Strategy and do not duplicate assistance already provided or imminent. If a Sate representative determines that an application is either inconsistent with the State strategy or duplicates the State's assistance, that portion of AFG application will not be funded. In FY2004 and 2005 State officials came to Washington, DC where they reviewed and commented on the selected applications. In 2006 an online review system was created where the appointed State official reviewed and made comments online. For FY2004, 605 applications went through SAA review with a Federal share request of $35.8M. For FY2005, 1052 applications went through SAA review with a Federal share request of $98.7M. For FY2006, 703 applications went through SAA review with a Federal share request of $41.6M. Cumulatively, 80-90% percent of these requests were funded. It is worthwhile to note that only one time has a State Official objected to a particular request and, when AFG stated the request would not be funded based on the recommendation, the State official withdrew their recommendation. Additionally, AFG has an MOU with the Departments of Agriculture and Interior for the coordination of their fire department assistance programs. This cooperation brings wildland fire expertise to the AFG program which traditionally has a structure fire focus. Starting with FY2006 national wildland fire priorities have been provided via these agencies to be presented for consideration during the AFG criteria development session. This ensures a collaborative effort in addressing mutual concerns and helps to eliminate duplication of effort and funding between programs. As a result of the MOU the AFG sits, as a non-voting member, on the Interagency Wildfire Leadership Council. This council was established by the Bush Administration in April 2002, to help implement the National Fire Plan. Objectives of the council are to provide a coordinated seamless management structure to all aspects of wildland fire policy and to continue efforts to effectively cooperate with our partners at the Federal, state and local levels. Staff from these departments participate in the AFG and Fire Prevention and Safety panel reviews as well. The AFG Program has entered into an inter-agency agreement with the HHS, Center for Disease Control and Prevention (CDC) to conduct a study of the effectiveness of AFG Fire Prevention and Safety activities.

Evidence: Electronic communications between AFG and SSD regarding CEDAP. AFG Summary Scoring Sheet Results Electronic communication between Dept. of Interior and AFG. AFG Program Guidance. MOU with Departments of Agriculture and Interior. IAA between DHS, AFG and HHS, CDC.

Does the program use strong financial management practices?

Explanation: Explanation: The program's on-line grant system has enhanced financial management controls. Bank information is submitted with each application and is re-certified before any funds are transferred between AFG and the grantee. Individual payment requests are vetted through both the program office and the grants management office through a minimum of three levels of review: regional office review, program office review, and grants management office review (financial). All awards are vetted through a minimum of five levels of review prior to award: computer scoring, peer review, technical review, program review, and financial review.

Evidence: Desk Top Reviews; Site Visits; Close-out Reports Sample copy of award articles of agreement Sample copy of Reviewer Comments section of award DHS OIG FY2006 Audit DHS Management Directives

Has the program taken meaningful steps to address its management deficiencies?

Explanation: Several of the steps taken include enhancing the role of the regional offices and the support the headquarters staff provide them. The regional staff members are now the first point of review on all requests for amendments and performance reports. An enhanced reports function is soon to be deployed to assist them with the analysis of applicant and award recipient activity trends. Specific regional and programmatic performance targets have been incorporated into all staff individual performance evaluation plans. The program office is examining improvements such as transferring the helpdesk function to regional offices in order to enhance their roles as the primary point of contact with the field. To usher vehicle awards toward a more expedient grant completion monitoring tools have been tailored specifically for these grants. As a result of more assertive monitoring these historically slower performing grants should be expedited during 2007.

Evidence: Individual Performance Plan for Regional Specialists Sample of Vehicle Desk Review Report

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: Explanation: The AFG Program makes extensive outreach efforts through hands-on workshops, on-line announcements, and advertisements in professional journals. The grant program guidance clearly explains the competitive process of the application and award reviews. This is further emphasized through approximately 400 workshops that are conducted in communities around the Nation each year and the on-line grant application tutorial. This tutorial is available at www.firegrantsupport.com and is accessed an average of 882,717 number of times each year, for a total of 1,765,435 to date. Throughout the year the program office maintains the AFG Helpdesk at 1-866-274-0960 to assist applicants and approximately 19 regional staff members are co-located with each of the ten FEMA regional offices to ensure program accessibility and provide technical assistance throughout the application and award processes. All applications are reviewed first through a computer-generated scoring of submitted answers, then through a thorough review by independent panels comprised of fire fighters and fire service experts representing the complete spectrum of the fire service. Recent modifications to the prioritization algorithm process have ensured all types of departments are competing on the same merits by eliminating separate algorithms for rural, urban, and suburban departments. Now, all departments are evaluated based upon the same algorithm with scoring mechanisms which account for population protected and call volume. The factors representing the Program priorities are clearly explained in the program guidance and online tutorial. That being said, the AFG for the first time may have to go to lower scoring applications in order to achieve a newly established funding "floor" for combination departments. The AFG has a statutory "floor" of 55% of the appropriation for both combination and volunteer departments. Historically, meeting the 55% "floor" has not been an issue as it is exceeded easily due to the make up of applications relative to the type of departments applying. The awarding process starts with the highest scoring applications and terminates when the available funds are exhausted. As a result of recent negotiations with OMB the 55% level has been allotted in a fashion that provides combination departments with 33% and volunteer departments with 22%. As a result of this alteration in distribution it is now likely that higher scoring applications from both career and volunteer departments will be passed over in order to meet the 33% imposed "floor" for combination departments which historically receive 27% of the available funding. A large portion of funds have been awarded to rural departments due to a variety of factors.

Evidence: Program guidance document; workshop presentation, workshop schedule, regional points of contact information, and online tutorial are all found at www.firegrantsupport.com. U.S. Fire Department Profile Through 2005, NFPA Copies of tutorial usage statistics.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: Explanation: In addition to the five-tier review of applications which incorporate computer scoring, panel review, technical review, program office review, and financial review, all awards undergo programmatic and financial monitoring, amendment and payment request review, performance reporting review, and closeout review. The monitoring function has been enhanced to more assertively gather indicators of project implementation compliance. In addition to the on-site visits performed by program and financial staff, desk reviews of grants have been implemented which incorporate a static review of the grant files and telephonic contact with award recipients. Further actions are taken as the individual case requires. To date 2003, 2004, and almost all 2005 grant awards have benefited from a programmatic desk review. Selection criteria are modified each year to ensure these reviews are effectively focused on the concerns the program office wants addressed. In FY 2006 desk reviews for vehicle awards were begun and will be continued for future fiscal years. The financial monitoring functions are performed in a manner that is coordinated with the programmatic monitoring, yet independent of the programmatic efforts, to ensure a more thorough overall monitoring of the national program. The on-line grant management function is being expanded to include all monitoring reporting to enhance the ability to examine trends in outcomes from the monitoring efforts.

Evidence: Statistics on closeouts AFG Monitoring Plan E-mail message prescribing desk review criteria Reports Site Monitoring visits Site Visit Report Vehicle Review Report

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: Explanation: The program actively collects data on applicant requests and grant recipient activities. While grantee-specific performance data is not widely available to the public, data on grantee's funded activities are made available via the internet. Besides reporting on award distribution and identifying those grantees receiving awards, success stories are regularly presented. These stories clearly demonstrate how the program is impacting not only the grantees themselves but the communities they serve thus imparting to the public the important impacts of the program. Due to the very large pool of grantees the AFG serves, (20,000 +), it is not feasible for the program to provide to the public more specific performance information than it currently does. The program does not have the staffing it would require to pull such a tremendous quantity of information and then distribute to the public in a transparent and meaningful manner. Additionally, the lower the M&A expenditure level is kept the higher the eventual awarding rate will be (i.e., making additional awards).

Evidence: Annual Reports found of AFG website: http://firegrantsupport.com/

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: 1) The program is making steady progress on its first long-term outcome measure of the percent reduction in fire-fighter injuries in jurisdictions receiving AFG grants, compared to national average, reducing this rate by 44.3% between 2004 and 2006. For 2004, the program had to use the year 2003 as the comparison period instead of 2002 because AFG did not collect firefighter injury data in 2002. The September, 2003 DHS OIG report corroborates the trend of this analysis with its survey of 116 departments showing that 90% operate more safely as a result of AFG training awards and 99% operating more safely as a result of equipment awards. If the program continues to achieve this ratio of reducing firefighter injuries the overall effect will be to increase its impact over time as more departments receive awards. Since 2003 fewer than 400 departments have received an award each year. Percent reduction in fire-fighter injuries in jurisdictions receiving AFG grants, compared to national average 2004 3.5% / 4.0% 2005 -1.3% / 1.8% 2006 44.3% / N/A 2) The Program is also achieving its second long-term performance measure of increasing the percent reduction of fire-related civilian deaths in jurisdictions that received Assistance to Firefighter Grant (AFG) funding compared to the national average. AFG has achieved strong outcomes for this measure since 2004. The national average rate of reduction as calculated by the CDC is not available for 2005 and 2006 yet and will be updated once that data becomes available in the coming years. From 2002 to 2006, the average rate of reduction in AFG jurisdictions is 7.8%. AFG plans on beating this rate in all future years. One of the tasks of the 2007 criteria development workshop will be to examine ways the AFG Program can be modified to create a more robust fire prevention and safety activity. When these changes are agreed upon, the aggressive targets for annual and long-term performance measures that demonstrate a direct linkage between program activities and civilian fatality rates among grant recipients can be met. percent reduction of fire-related civilian deaths in jurisdictions that received Assistance to Firefighter Grant (AFG) funding compared to the national average 2004 7.1% / -0.3% 2005 9.0% / N/A 2006 7.2% / N/A

Evidence: DHS OIG Review of AFG Program; dated September, 2003 Applicant-Reported Firefighter Injuries Report: 2002 - 2006. Civilian deaths as reported by AFG applicants: 2004 - 2006.

Does the program (including program partners) achieve its annual performance goals?

Explanation: 1) The program is showing strong success in the annual outcome measure percent of AFG awards with a training element. AFG has shown a substantial improvement from 2005 to 2006, but this is due to availability of training for equipment grants for the 2006 data set. Training for equipment data was not available prior to 2006. AFG calculated an estimate of what the 2004 and 2005 results would have been by extrapolating the percent of total awards that received training in 2006 onto 2004 and 2005 grants. The program showed an improvement of 5% from the 2005 estimate to 2006. The long term targets for this measure use 2006 as a baseline and the 5% improvement rate estimated prior years; in future years all equipment-based training will be captured in close-out reports. Current results for 2004 and 2005 do not capture this large portion of training activities. Year Percent of Grants with a Training Element 2004 25.4% 2005 22.2% 2006 40.2% 2) The program is showing strong improvement in the annual outcome measure AFG grant dollars spent per firefighter trained. However, as previously mentioned, 2004 and 2005 results do not include the number of firefighters trained on as part of equipment grants. AFG calculated the estimated total cost per firefighter trained for these years, and estimates that the program showed an improvement of 2% in 2006 over the estimated average of the previous three years. The long term targets for this measure use 2006 as a baseline and the 2% improvement rate to calculate out-year targets. Year AFG Grant Dollars Spent per Firefighter Trained 2004 $7,538 2005 $7,842 2006 $4,081 3) The Program has implemented several steps to ensure that it continues to attain its annual target for the percent of grants for which a close out report was received within 15 months of the award date. To date, AFG has achieved positive results for this measure. Aggressively conducting programmatic desk reviews of grants is reaping benefits in solving problems during close out before they actually occur. Using this same methodology to specifically target vehicle award desk reviews is helping ensure grant recipients minimize the historical lag time associated with procuring firefighting vehicles. And enhancing the program management tools available to the regional fire program specialists is giving them the means to meet the higher expectations that have been placed on them through their performance appraisals. Using the 2002 figure of 77% as the baseline for this measure improvement is characterized as an increase over that percentage. Year Percent of AFG Grants Closed-Out within 15 Months of Award Date 2003 77.2% 2004 75.5% 2005 80.3% 4) The program has shown mixed results on the measure ratio of AFG funds awarded to population served over the last three years. From 2003 to 2005, the program achieved steady progress, showing a decrease of 9%. This number has increased in 2006. AFG expects this number to improve in the future due to the expansion of regional applications, as well as a change in program policy that allows high density districts to apply for grants for multiple vehicles. The statutory caps on this program do not limit AFG's effectiveness on this measure: the demographic profile of all applicants is such that qualified high-density jurisdictions (urban and suburban) are not turned away due to any limits on award funding. Year AFG Grant Dollars Spent per Firefighter Trained 2004 $4.62 2005 $4.57 2006 $5.72

Evidence: Mapping Critical Task Exercise Analysis to Firefighting Operations / Support Capability Close-out Report Submission Analysis AFG Cost Benefit Summary Report DHS OIG Review of AFG Program; dated September, 2003

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: Explanation: The program has shown consistently strong performance on the efficiency measure the percent of grants for which a close out report was received within 15 months of the award date. This consistently strong performance indicates that AFG has improved its own program efficiency by ensuring that a higher number of grantees submit close out reports, thereby providing AFG with data that informs future decision making and more informed, and efficient, program operation. Web-based online system with improvements to include reports functions, de-obligated funds re-programmed to other applicants previously not receiving awards. Since the inception of the AFG Program $45,550,834.00 has been saved through programs' office market research, de-obligation of funds due to unallowable expenditures, and un-spent administrative costs. These funds have been re-directed back into project awards to grant recipients who did not receive funding during the initial round of awards. During the first five years of the program, 2001 - 2005, it has received $2,665,700,000.00 in appropriations and awarded $2,594,321,904.00 to grant recipients. Subsequently, it has spent less than $71,378,096.00 in program administration. This is only 2.6% of the authorized 5%.

Evidence: Cost comparison of paper applications to on-line applications Expenditure totals for past fiscal years AFG Expenditure Overview

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: Explanation: There are no other programs for the specific purpose of awarding funds to fire departments to enhance their ability to protect firefighters and the public they serve from fire-related incidents. There are, however, some programs with the similar purpose of enhancing public safety. The Local Law Enforcement Block Grant, for example, provided funding to local governments for the purpose of assisting law enforcement agencies in the procurement of equipment, training, and personnel. Using an online application and grant management system similar to that of the AFG Program, the LLEBG Program received 17,581 applications over the course of six years and awarded 17,385 grants. The program's last year of funding was 2004 and it has closed out 93% of its awards. By comparison, the AFG Program received 137,302 applications from 2001 - 2005, for a total of 31,398 awards. Of these, 78% have been closed (24,490). In terms of the sheer volume alone the AFG Program compares very favorably, especially when it is considered that these figures do not represent the number of grant actions required to manage an application through to award close-out.

Evidence: Close-out reports of LLEBG, 1999 - 2004.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: Explanation: The Department of Homeland Security Office of the Inspector General report of the AFG Program reported many favorable achievements. Of 999 fire departments that received an AFG grant award and responded to the DHS OIG survey, responses included: 97% said the AFG program grants have had a significant or moderate positive impact on their department's ability to handle a fire or fire-related incident; 99% said the AFG program allows their department to present initial and/or regular training to firefighters; 90% believed their fire department operates more efficiently and more safely; 98% reported significant or marginal improvement in the operational capacity for firefighting; 99% reported a significant or marginal improvement in the safety of their firefighters during emergency operations; and, 62 percent reported a reduction of injuries during firefighter operations. The National Academy of Public Administration (NAPA) conducted an assessment and strategic review of the program that indicates several areas of achievement. For example, regarding the question of the AFG Program impact on community fire protection capabilities, the report states: "While there are many assumptions made in review of this data, there is an indication that over 50% of the jurisdictions that received AFG money for apparatus and PPC (public protection classification) improved their classification, and many by two or more classifications." In October, 2006 the U.S. Fire Administration published its second report comparing the reported needs of the fire service among FY 2005 AFG grant award recipients to those reported in 2001. This review points out multiple areas of achieved results, such as: the percentage of departments without enough SCBA to equip all emergency responders on a shift declined by 10%; the percentage without enough PASS devices to equip all emergency responders on a shift declined by 14%; and, the percentage of departments where there were not enough portable radios to equip everyone on a shift declined by 13%.

Evidence: A Review of the Assistance to Firefighters Grant Program; DHS, Office of Inspector General, September, 2003 Assistance to Firefighters Grant Program: Assessing Performance; National Academy of Public Administration, April, 2007 Matching Assistance to Firefighters Grants to the Reported Needs of the U.S. Fire Service; U.S. Fire Administration, October, 2006