Detailed Information on the
Trauma-EMS Systems Program Assessment

Program sunset in FY 2006.

Develop baselines and targets for all long-term and short-term goals.

Measure: Reduce the number of patients who die per year as a result of not being treated at a designated trauma center.

Explanation:Some targets may be shown as NA (not applicable) because the program is not proposed for funding.

Measure: Among the States that collect trauma data from all acute care facilities, increase the proportion of patients with an Injury Severity Score (ISS) Score of greater than or equal to 15 who are transferred to level I or level II trauma centers within 48 hours.

Explanation:There are 12 States that have mandatory submission for the collection of trauma data from all acute care hospitals (both trauma and non-trauma centers). Of those twelve States, the program was able to obtain information from ten States (83%). To address this annual performance measure, each state with a statewide trauma registry data collection system was asked to tabulate the total number of patients contained within the registry in 2003 meeting the following criteria. Some targets may be shown as NA (not applicable) because the program is not proposed for funding.

Measure: Among States that collect trauma data from all acute care facilities, increase the proportion of patients with an Injury Severity Score (ISS) score of greater than or equal to 15 who are transferred to a level I or level II trauma center within 48 hours, while maintaining a constant Federal expenditure.

Explanation:Some targets may be shown as NA (not applicable) because the program is not proposed for funding.

Measure: Increase annually the number of States that have adopted statewide standardized triage protocols.

Explanation:Some targets may be shown as NA (not applicable) because the program is not proposed for funding.

Measure: Increase annually, the number of States that designate trauma centers

Explanation:Some targets may be shown as NA (not applicable) because the program is not proposed for funding.

Measure: Increase annually the number of States that have traige training programs.

Explanation:Some targets may be shown as NA (not applicable) because the program is not proposed for funding.

Is the program purpose clear?

Explanation: The purpose of the Trauma-EMS Systems Program is to assist State governments in the development, implementation, and improvement of systems of trauma care so as to decrease the morbidity and mortality of individuals that present with moderate-severe traumatic injuries.

Evidence: Evidence for this answer can be found in the following documents: 1) Public Health Service Act, Title XII- Trauma Care, Section 1201, and P.L.101-590 2) 2005 Program Grant Guidance

Does the program address a specific and existing problem, interest, or need?

Explanation: The program does address the specific and existing problem of high mortality and morbidity due to injuries/trauma in the United States. In 2002 aloone, 160,000 Americans died from injuries, and approximately 15,000 to 20,000 children die each year as a direct result of pediatric trauma. Studies have demonstrated that these numbers could be lowered significantly if trauma systems were to be fully implemented in every State and region of the Nation. It is estimated that 20,000 to 25,000 lives per year could be saved in the U.S. if States succeeded in fully implementing and improving their trauma care systems.

Evidence: Evidence for this answer can be found in the following document: Institute of Medicine. ""Reducing the Burden of Injury"", The Institute of Medicine, Chapter 6, p.151, Section: Outcomes of Trauma Care Systems. (On page 150 of Chapter 6 there begins a section called "Outcomes of Trauma Care Systems".)

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: The program is designed in a manner so that it is neither redundant nor duplicative of any other Federal, state, local or private effort. The Trauma/EMS Systems Program is the only Federal program that focuses on, and is specifically authorized for the development, implementation, and improvement of statewide systems of trauma care and aims to reduce mortality and morbidity secondary to trauma. The Program is designed to avoid redundancy.

Evidence: Evidence for this answer can be found in the following documents: 1) Public Health Service Act, Title XII- Trauma Care, Section 1201, and P.L.101-590 2) 2005 Program Grant Guidance 3) The FY 2006 Congressional Budget Justification for the Trauma-EMS Systems program

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The program appears to be free of major flaws that limit its effectiveness or efficiency as there exists no evidence that a different approach or mechanism would be more effective or efficient given goals of the program and the changing conditions in the field.

Evidence: The program delineates 80% of its annual appropriation to States in the form of competetive grants. It also (as directed by statute) spends 10% of its budget on rural initiatives, with the remaining 10% going to Program administration/discretionary funds. For those States that do not receive funding the program continues to provide technical assistance (through its Technical Assistance Center).

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: The program is effectively targeted as its resources are targeted to each of the 50 States, often in the form of competitive grants provided to State Trauma Systems Managers and State EMS Directors. Through these organizations, and with the help of Trauma/EMS program funding, States work to promote trauma system development, implementations and improvements. Improved State trauma systems then benefit those individuals who experience moderate to severe injury and/or at risk for moderate-severe injury in the U.S. and its Territories.

Evidence: Evidence for this answer can be found in the following documents: 1) 2004 Map of State Funding 2) 2003 and 2004 Program Grant Guidance 3) Technical Assistance Center Scope of Work 4) American College of Surgeons Scope of Work 5) Model Trauma Systems Planning and Evaluation (Draft) (See Resource # 5)

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The program has developed a long-term performance measure that focuses on an outcome that meaningly reflects the purpose of the program. As the purpose of the program is to decrease the mortality of individuals who experience moderate-severe traumatic injuries, the program's long-term measure focuses on this goal: By 2017, reduce the number of patients who die as a result of not being treated at a designated trauma center.

Evidence: This measure takes into account several aspects of the program's primary activities: 1) helping an increased number of States establish formal systems of trauma care and 2) helping States further develop and improve these trauma systems and newly designated trauma centers once they are in place.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The program has developed an ambitious target and timeframe for its new long-term measure. The program seeks to reduce the number of patients who die as a result of not being treated at a designated trauma center by 10% by 2017 (from 45% in 2002 to 35% in 2017).

Evidence: Several experts in the fields of Emergency Medicine and Emergency Medical Systems provided guidance to the program in its development of this long-term measure. State Trauma Care Managers across the country have attested that the target set for this measure is in fact ambitious. An article in the Journal of the American Medical Association (JAMA) entitled: "The Effect of Organized Systems of Trauma Care on Motor Vehicle Crash Mortality" found that the effect of implementing an organized system of trauma care does not appear for 10 years; and in this particular study they found an 8% reduction in mortality secondary to motor vehicle accidents after 10 years of implementation (which is less than the 10% reduction in mortality the Trauma/EMS program has set for its 10 year/long-term measure).

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The program has developed four annual measures three of which have updated baselines and targets, and two of which have historical data with which the program's progress can be measured. For the fourth annual measure (listed to the right in the evidence column), the program will have baselines and targets developed by October 2005.

Evidence: The four annual measures are as follows: 1) Increase annually the number of States that have adopted statewide standardized triage protocols. 2) Increase annually the number of States that have triage protocol training programs 3) Increase annually the number of States that designate trauma centers 4) Among the States that collect trauma data from all acute care facilities, increase the proportion of patients with an ISS Score of greater than or equal to 15 who are transferred to level I or level II trauma centers within 48 hours.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: At the present time, out of 4 measures, the program has baseline data/targets for 3 of them. The program has indicated that for their newest and most useful measure that baseline and target information would be available by Nov. 1st, 2005. The PART process will be opened again in the Fall, at which time the program will have an opportunity to provide this information (for both remaining measures) and, if determined to be sufficiently ambitious, have an opportunity to change this answer to a "yes".

Evidence:  

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: The Trauma-EMS Systems program has many partners, public and non-governmental, who are committed to developing and improving States' systems of trauma care. Some of the ways this commitment has been demonstrated are: 1) Both private and non-profit stakeholders and Federal partners were involved in program goal development. 2) Through the Grant process, the States have shown evidence of major theme progress on programmatic goals, 3) The focus of Annual State Trauma Leadership Meetings (that all States participate in) has been the program's initiatives and goals, 4) Researchers communicate frequently with the program regarding their research results and program's current and long-term focus. In addition, each of these partners are held accountable for demonstrating progress towards the goals of the program through reviews of their annual Workplans, progress and division reports. Failure of a grantee to demonstrate progress as defined in their guidance can result (and has resulted) in funds withheld from the grantee the following fiscal year.

Evidence: Evidence for this answer can be found in the following documents: 1) 2005 Program Grant Guidance 2) Model Trauma Systems Planning and Evaluation (draft document) 3) National Trauma-EMS Systems Stakeholder Group membership listing 4) Federal Partners for Trauma and EMS listing and initiatives document 5) Annual State Leadership Meeting attendees listing and agenda 6) American Trauma Society Contract Scope of Work; public info./educ.outcomes 7) Society of Trauma Nurses Contract Scope of Work; training outcomes 8) Trauma-EMS Technical Assistance Center Contract Scope of Work. 9) Trauma-EMS Systems Program Workplan 2005 10) Trauma-EMS Systems grantee progress and annual division reports.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: The program has not had any formal independent evaluations.

Evidence:  

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The program/ administration has not requested a budget for this program from FY 2003-FY2006.

Evidence:  

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: As stated above in the answer to question 2.4, the program is in the process of developing a baseline and targets for a new annual measure that will provide valuable information on the program's annual progress in achieving its goals.

Evidence: The program has indicated that baseline and target information would be available for their newest and most useful measure by Nov. 1st, 2005.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The program does collect timely and credible performance information from key program partners through grantees' progress reports and their annual State Trauma Abstracts and State Trauma Funding Profiles.

Evidence: Evidence for this answer can be found at the following HRSA/Trauma-EMS web-sites: 1) State Trauma Projects Abstracts ftp://ftp.hrsa.gov/hrsa/trauma/04projectabstracts.pdf 2) State Trauma Funding Profiles: ftp://ftp.hrsa.gov/hrsa/trauma/04fundingprofiles.pdf

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: Federal managers of the program are held accountable for the performance of the program through the HRSA Performance Appraisal Program's Performance Evaluation Plan. This evaluation specifically inquires as to whether or not the manager's program is achieving its desired results. The program holds grantees accountable through their annual Program Workplans and through their grantees' progress reports and annual division reports. Failure of a grantee to demonstrate progress toward the goals of the program, or as defined by their guidance, can result (and has resulted) in funds being withheld from the grantee the following fiscal year.

Evidence: Accountability for cost, schedule, and performance results are maintained through: 1) HRSA's Performance Appraisal Program Performance Evaluation Plan 2) HRSA's Functional Grants Process Flow Chart 3) Trauma-EMS Systems Program Workplan 2005 4) Trauma-EMS Systems grantee progress reports and annual division reports.

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: Federal funds from this program are obligated in a timely manner and mechanisms are in place to ensure that the funds are spent for the intended purpose. The Grant Management Specialists within HRSA monitor grant budget expenditures and inform grantees if the funds are not being expended on schedule. Grant management staffs are primarily responsible for the business and other non-programmatic areas of grant award and administration. They are also repsonsible for ensuring that, for grants under their cognizance, both federal staff and grantees fulfill applicable statutory, regulatory, and administrative policy requirements.

Evidence: Evidence for this can be found in the following documents: 1) HHS Grant Policy Directive 2.04 2) Public Health Service Act, Title XII- Trauma Care, Section 1201, and P.L. 101-590 3) Example Highlighted in Trauma-EMS Technical Assistance Center Contract

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The program has in past years used the percentage of grantees that submit mid-year reports electronically to measure its increased efficiencies and cost-effectiveness in the program's execution. The program has recently developed a new efficiency measure that will provide a greater amount of and more detailed information regarding the program's ability to achieve efficiencies and cost effectiveness in program execution.

Evidence: The program's newley developed efficiency measure is as follows: Among States that collect trauma data from all acute care facilities, increase the proportion of patients with an ISS score of greater than or equal to 15 who are transferred to a level I or level II trauma center within 48 hours, while maintaining a constant Federal expenditure.

Does the program collaborate and coordinate effectively with related programs?

Explanation: The program effectively collaborates and coordinates with multiple related programs that have related purposes through its Federal Partners Meetings, Annual State Trauma Leadership Meeting, and through inclusion on the Program's e-mail and listserv correspondence. The Federal agencies with which the Trauma/EMS Systems program collaborates are the Centers for Disease Control and Prevention (CDC), National Highway Traffic Safety Administration (NHTSA), Emergency Medical Services for Children (EMSC), and Office of Rural Health Policy (ORHP).

Evidence: Examples of specific collaborative efforts include: 1) CDC/HRSA Funded Project: The Trauma Program went together with the CDC Injury Center to support a mutually important May 2003 Alcohol and Drug/Trauma Conference. There were 29 separate papers in proceedings, which will be published mid 2005 (in the next few months) as a special supplement to the Journal of Trauma. 2) In 2004, HRSA's Emergency Medical Services for Children and Trauma-EMS Systems programs were contacted by the Delaware Office of Emergency Medical Services to facilitate discussions and provide consultation for improvements of the State's Trauma System.

Does the program use strong financial management practices?

Explanation: In FY 2004, HHS OIG conducted an HHS financial statement audit. The audit reported that the Department had serious internal control weaknesses in its financial systems and processes for producing financial statements. OIG considered this weakness to be material. The audit recommended that HHS improve their reconciliations, financial analysis, and other key controls. The September 30, 2002 HRSA independent auditor's report found that the preparation and analysis of financial statements was manually intensive and consumed resources that could be spent on analysis and research of unusual accounting. The audit also found that HRSA's interagency grant funding agreement transactions were recorded manually and were inconsistent with other agencies' procedures. Finally, the audit found that HRSA had not developed a disaster recovery and security plan for its data centers.

Evidence: Evidence for this answer can be found in the following documents: 1) HHS FY 2004 Performance and Accountability Report 2) HRSA's 2002 audit report

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The DHHS' long-term strategic plan to resolve the internal control weaknesses is to replace existing accounting systems and other financial systems within HHS with the Unified Financial Management System (UFMS). HHS plans to fully implement the UFMS Department-wide by 2007. HRSA developed a corrective action plan to address the reportable conditions identified in the September 30, 2002 independent auditor's report. For each aspect of the five reportable conditions, HRSA assigned an office responsibility. The plan also outlines milestones and target completion dates. When the new performance measures are in use, results from these measures will be available to public via the HRSA website.

Evidence: Evidence for this answer can be found in the following documents: 1. HHS FY 2004 Performance and Accountability Report www.hhs.gov/of/reports/account/acct04/pdf/section4.pdf 2. HRSA Corrective Action Plan for FY 2002 Financial Statement Audits

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: Trauma/EMS System program grants are awarded based on a clear competitive process that includes a qualified assessment of merit. HRSA's Division of Independent Review is responsible for managing objective reviews within the agency. Application competing for federal funds that are received by this Division undergo an objective and independent review by experts qualified by training and experience in the field of Trauma and EMS services. In addition, in FY 2005, the Trauma/EMS Systems program will undergo a Grant Application Field Review, which is defined as an objective review approach in which reviewers review and score assigned applications, but not as a committee.

Evidence: Evidence for this answer can be found in the following documents: 1) HRSA Division of Independent Review Objective Review Manual 2) Grant Application Field Review 3) 2005 Program Grant Guidance

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: The program does have oversight practices that provide sufficient knowledge of grantee activities. All Trauma/EMS grantees must submit a Payment Management System Quarterly Report, Financial Status Reports (which are due within 90 days of the end of each grant year), Progress Report for Program Reporting (which are due at the time of the continuation application and 90 days post closeout of the project), and Final Closeout Reports (which are due within 30 days following the end of the project period). Failure to produce these reports may result in the inability of the grantee to access future funding.

Evidence: Evidence for this answer can be found in the following documents: 1) 2005 Program Grant Guidance 2) 2004 State Trauma Project Abstracts 3) Office of Performance Review (OPR) Performance Review Protocol Guide 4) 2004 Grant Matrix

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: The program does presently collect grantee performance data on an annual basis through their State Trauma Project Abstracts and State Trauma Project Profiles. This information is easily accessible on the program's web-site for any of the public to see.

Evidence: Evidence for this answer can be found at the following HRSA/Trauma-EMS web-sites: 1) State Trauma Projects Abstracts ftp://ftp.hrsa.gov/hrsa/trauma/04projectabstracts.pdf 2) State Trauma Funding Profiles: ftp://ftp.hrsa.gov/hrsa/trauma/04fundingprofiles.pdf

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: Because the program's long-term measure is newly developed, there is no historical data that demonstrates the program's progress towards achieving this specific long-term goal.

Evidence:  

Does the program (including program partners) achieve its annual performance goals?

Explanation: Of the four short-term goals newly developed by the program, three have historical data that indicate that the program has been effective in assisting States in adopting statewide standardized triage protocals and designating trauma centers. Studies have indicated with some consistency that improving organized systems of trauma care (i.e. States designating trauma centers, adopting standardized triage protocals) does lead to measurable decreases in mortality doe to trauma.

Evidence: During the period of 2001 through March 2005 the following progress has been made by States in implementing or updating their Trauma-Triage protocols: 1) Implementation: Between 2001 and 2005 eight States have implemented Trauma Triage Protocols and two States plan to implement such protocols in 2005. 2) Protocol updates: Between 2001 and 2005 fifteen States have updated their Trauma Triage Protocols and two States plan to update these protocols in 2005. The source of this data was the State Grant applications and Technical Assistance (TA) Center.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: While the program's efficiency measure is currently under development and has no historical data to which to refer to answer this question, the program has been able to demonstrate progress and increased outputs while experiencing small decreases in funding. Each year, between FY2003 and FY 2005, the appropriations received demonstrate a downward trend. As you can see from the information provided in question 4.2 there is an upward trend in State progress. For the low dollar amt. of State funding received from this program, there has been much work accomplished by the States.

Evidence: Evidence for this answer can be found in the following documents: 1) The FY 2006 Trauma-EMS Systems Congressional Justification 2) The State Grant applications and Technical Assistance Center (see data above) 3) FY 2004 State Funding Profiles

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: There is no other Federal, State, local, private or non-profit organization that is currently engaged in activities similar to those performed by the Trauma-EMS Systems program. No other public program or non-government organization has committed itself to developing, implementing and improving statewide systems of trauma care in every State of the Nation.

Evidence:  

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: The program has not had any formal independent evaluations that have assessed its effectiveness or impact.

Evidence: