Skip Navigation HRSA - U.S Department of Health and Human Services, Health Resources and Service Administration U.S. Department of Health & Human Services
Home
Questions
Order Publications
 
Grants Find Help Service Delivery Data Health Care Concerns About HRSA
Performance Review
 

Performance Review Protocol Guide: Principles and Implementation


Dear Colleagues:

Dear Colleagues:

We are pleased to share with you the attached Performance Review Protocol Guide: Principles and Implementation. We hope you find this guide helpful and informative, as you proceed with the performance review process.

As part of the Health Resources and Services Administration’s overall efforts to improve the nation’s health and access to quality health services, the Office of Performance Review (OPR) plays a central role in achieving the HRSA mission by reviewing and enhancing the performance of HRSA-supported programs within communities and States. The purpose of performance reviews is to improve performance of HRSA funded programs by working collaboratively with grantees and HRSA Bureaus/Offices to measure program performance, analyze the factors impacting performance, and identify effective strategies and partnerships to improve performance, with a particular focus on outcomes.

Performance reviews also create the unique opportunity to assess HRSA funded programs within the context of the grantee’s entire organization. Viewing programs from such an integrative and comprehensive perspective provides a deeper understanding of a grantee’s performance and the impact or effectiveness of HRSA funded programs on the public’s health. The Performance Review Protocol is a document that sets forth the basic framework (four primary components) for conducting the performance reviews. As designed, the Protocol is meant to be dynamic and flexible to ensure its applicability across the wide range of HRSA funded programs.

The Performance Review Protocol Guide – Principles and Implementation is a companion to the Performance Review Protocol. The Protocol Guide was crafted to facilitate and direct the analytical and implementation process, by providing an in-depth explanation of the principles and methodology for conducting performance reviews. The guide is meant to build upon the performance review framework presented in the Protocol and should serve as a resource for OPR’s internal and external stakeholders to illustrate our approach to performance reviews.

OPR is committed to continuing our collaboration with grantees and our fellow HRSA colleagues through this performance review process. By working in partnership, OPR will best achieve its mission of improving and enhancing the performance of HRSA-supported programs within communities and States.

Thank you for your time and interest.

Rebecca Spitzgo
Associate Administrator
Office of Performance Review

Performance Review Protocol – Principles and Implementation

Key Principles and Concepts

The Protocol is a framework which may be applied to the wide variety of HRSA funded programs and grantees. To apply the Protocol effectively, one must understand its underlying principles and concepts, described below.

  • Essential Characteristics of a Good Performance Review System

    A good performance review system should have the following characteristics:
    • Useful
    • Common sense
    • Simple
    • Uses plain language
    • Uses minimum paper
  • Data-driven Decision Making
    Effective performance management systematically accesses and uses the best quantitative and qualitative data and data analysis available at every stage of decision making. For example, the ability to manage performance and, in particular, to achieve an impact for the population served by a program is inseparable from the ability to assess whether such an impact has been achieved. Progress on performance measures and/or goals and objectives, provide such an assessment – a “bottom line” – in managing for performance.
  • Language Discipline
    Too often terms are used interchangeably and with a multiplicity of definitions in talking about performance. The Protocol recognizes the importance of the disciplined (clear, consistent, and jargon-free) use of language.
  • Population-level Accountability vs. Program Performance Accountability
    In thinking about accountability, it is important to distinguish between accountability for achieving a condition of well being for (1) a whole population (e.g., all of the children in a county who are under six years old) and (2) the population or clients served by a program, agency or service system (e.g., the clients of a HRSA grantee). The significance of this distinction lies in the recognition that a single program, agency or even service system cannot take sole responsibility (or credit) for achieving a desired condition of well being for a whole population.
  • Talk to Action
    Importantly, the Protocol gets from talk to action quickly.

    (1) Adapted from materials authored by Mark Friedman
  • Ends to Means
    The Protocol starts with ends defined as either Performance Review Measures or selected Program Goals and Objectives and works backwards to means (Performance Analysis, Key Area Discussion, Performance Improvement Options, and Action Items). Starting with ends – asking what we want for whom at the outset – saves countless hours and reams of paper typically expended in most review processes. It also provides focus: we know why we are here. The progression of the Protocol from ends to means involves four basic steps (these steps mirror the four components of the Protocol) which should first be understood in general terms:

    Step 1. First, define what we want for whom (e.g., babies born healthy) in measurable terms when there are adequate data (e.g., % of prenatal clients who had babies with low birth weights) or what goal or objective is desired. In the cases where a measure is chosen, the data for the measure is graphed to show both historic and projected trends. Following a review of the performance trend, the question becomes one of means: how to change (or accelerate, if positive or the same) the trend line (i.e., how to “turn (improve) the curve”) of the trend line. In the cases where there is a dearth of measurable data or an inability to trend data, an examination of the grantee’s application should yield a set of three to five significant goals and objectives on which to focus the performance review.
    Step 2. To turn the curve for a specific performance measure or goal or objective, start by analyzing the performance of the grantee with respect to the particular performance measure. Identify the most important factors that must be addressed to improve the performance trend. This is also an opportunity to explore what other issues that may not be directly aligned with the performance measure but are important to relevant to the grantee’s overall performance and operations.

    Step 3. Once factors associated with performance have been identified, determine what works to address those factors, and what the available options are for actions that will turn the curve.

    Step 4. Finally, deciding on a specific action plan to turn the curve and improve performance.


Following the implementation of the Action Plan, the progression begins again, starting with determining whether there has been progress in turning the curve of the performance trend line. The circle chart (2) on the
following page illustrates the four steps

(2) This circle chart is an adaptation of the circle chart developed by Roger Fisher and discussed in Getting to Yes, by Roger Fisher, et. al.

Performance Review Protocol:
Ends-to-Means Circle Chart

Ends-to-Means Circle Chart

  • Broadening the Focus

    The protocol encourages the focus of the analysis be broadened beyond the individual performance review measure or goal or objective. This should include a discussion of “Key Areas” relating to grant implementation, best or promising practices, or the challenges and opportunities the grantee faces in achieving its mission.

  • Collaboration

    Finally, before moving ahead with a detailed discussion of the four components of the Protocol, it is important to note that the Protocol – and the performance review – is designed to be conducted in collaboration. The Protocol is, from this perspective, a tool for collaboration: it provides a framework from which the review team, in collaboration with the HRSA project officer(s) and the grantee work together in selecting key Performance Review Measures or Goals and Objectives, analyzing the factors affecting performance, and developing options and actions to improve the grantee’s performance.

    More specifically, the Protocol requires systematically working through each of the four components, making performance data and analysis explicit, applying certain decision making criteria, and ultimately developing a set of actions that will improve performance. The Protocol both provides for efficient participation and garners the best thinking of those involved.

  • Performance Review Approach

    The Performance Review Protocol is a performance improvement tool used with each grantee to review its HRSA funded programs. Due to the wide variability in the scope and purpose of HRSA grant programs, the Protocol provides for two approaches to accommodate these variances and best facilitate improvement for each type of grant program.

    All HRSA grant programs (including cooperative agreements) eligible for a review are designated by OPR as either a Category 1 or Category 2 grant program for the purpose of conducting a performance review. (See Matrix of HRSA Grant Programs for listing of grant programs by category).

      • Category 1 includes: health services delivery activities and professional training programs with measurable data.
      • Category 2 includes: grant programs funding professional training and public health infrastructure development that do not lend themselves to using the same approach used for Category 1 grant programs typically due to small numbers of recipients/clients, limited term funding, or the scope of the activity.

The review of Category 1 grant programs measures program performance, analyzes the factors impacting performance, and identifies effective strategies and partnerships to improve program performance. The review of Category 2 grant programs analyzes the factors impacting the ability of the grantee to meet selected goals and objectives of the grant program and/or achieving sustainability beyond the end of HRSA support, and identifies effective strategies and partnerships to improve performance/progress on program goals and objectives.

For Category 1 grant programs, the four primary components of the protocol are:

  • Performance Review Measures
  • Performance Analysis and Key Area Discussion
  • Performance Report
  • Action Plan/Technical Assistance/Follow-up

For Category 2 grant programs, the four primary components of the protocol are:

  • Goals and Objectives
  • Discussion about Progress/Impact/Sustainability
  • Performance Report
  • Action Plan/Technical Assistance/Follow-up

The Four Components of the Protocol
The Protocol, incorporating the principles and concepts discussed above, consists of the following four components, which are embedded in the previous Circle Chart.

Four Components of the Protocol and the Ends-to-Means Circle Chart

The following is a detailed description of the process for implementing each of the four
components.

Performance Review Measures Goals or Objectives
The Protocol begins with selecting Performance Review Measures or Goals and Objectives for each HRSA funded program based on the grant program’s categorization – Category 1 grant programs being those with measurable data versus Category 2 grant programs being those that do not lend themselves to the development of performance measures due to small numbers of recipients/clients, limited term funding, or the scope of the activity. Please see the Matrix of HRSA Grant Programs on page 32 for a list of grant programs by category. As will be shown, the Performance Review Measures or Goals and Objectives in many respects define the scope and focus of the performance review because the performance review and its two primary documents, the Performance Report and the Action Plan, all “begin at the end” with the Performance Review Measures or Goals and Objectives.

Performance Measures

In order to discuss Performance Review Measures, we must first define performance measures. For the purposes of the Protocol, a performance measure is a set of data that answers one of the three following questions with respect to a program or major activity:

  1. How much service (or resources) is delivered?
  2. How well is it delivered? In the Protocol, these kinds of measures are called “effort measures.”
  3. What impact (3) is produced? In the Protocol, these kinds of measures are called “outcome measures.

(3) The term “impact” is not meant to imply an “impact analysis,” which would involve isolating variables to scientifically verify a causal relationship between a specific activity and a corresponding result. Such impact can, at best, be inferred from the knowledge that certain activities occurred and certain results followed.

The following table provides some common examples of these three different kinds of performance measures:

Common Examples of the Three Different Kinds of Measures

1. How much? Dollars Spent
Number of target population/clients/customers served
Number of activities (by type of service)
2. Effort Measures
(How well?)
Administrative efficiency: overhead as a percentage of total cost; unit cost
Staff measures: client staff ratio; workload ratio; staff turnover rate; staff morale; % staff fully trained; % clients seen in their own language
Access measures: waiting time; waiting lists
Activity-specific measures: % timely; % clients completing activity; % correct and complete; % meeting standard
3. Outcome Measures
(Impact?)
Client results or client outcomes: The % of clients achieving certain skills/knowledge (e.g., nursing skills)
The % of clients achieving a desired behavior (e.g., preventative behaviors, adherence to prescribed medical regimens)
The % of clients achieving a desired circumstance (e.g., access to health care, a desired health status)

The next three tables provide examples of the three kinds of performance measures for different programs:

Nursing Education Program
1. How much? Number of students
2. Effort Measure (How well?) Student-teacher ratio
3. Outcome Measure (Impact?) % of nursing students who enter nursing upon graduation

Health Clinic
1. How much? Number of patients treated
2. Effort Measure (How well?) % of patients treated in less than one hour
3. Outcome Measure. (Impact?) % of babies born healthy (e.g., birth weights)

Drug/Alcohol Treatment Program
1. How much? Number of persons treated
2. Effort Measure (How well?) % of staff with training certification
3. Outcome Measure (Impact? % of clients off alcohol or drugs: (a) at exit;
(b) 12 months after exit

For outcome and effort measures, a measure that is a percentage is generally a more useful than an absolute number. The percent tells the impact with respect to the total population served; a number will not provide that information or context. For example, it is more meaningful to know what percent, rather than what number, of women who were provided prenatal services had babies with low birth weights. It is also extremely powerful to take the data for a Performance Review Measure collected over time and present it in a graph to portray performance trends.

The following description of the steps by which Performance Review Measures are selected explains the role of each of the three kinds of performance measures and shows how Performance Review Measures are graphed.

Goals or Objectives

In order to assess progress in attaining a grant program’s goals or objectives the reviewer must be familiar with the grantee’s recent grant applications and progress reports. The grant application, in response to the HRSA Bureau or Office’s funding guidance, will detail specific goals and objectives that it will achieve to advance HRSA’s mission and priorities. Periodic reports submitted by the grantee will highlight progress in attaining its stated goals and objectives.

Selecting Performance Review Measures and Goals and Objectives: HRSA Funded Programs and Activities

It is essential to understand that a Performance Review Measure or Goal or Objective may apply to an entire HRSA funded program or a major activity within the HRSA funded program. Therefore, before selecting a Performance Review Measure or Goal or Objective, the performance review must, as a prerequisite, be clear as to whether it is measuring performance with respect to an entire HRSA funded program or a major activity within the HRSA funded program. As a general rule, Performance Review Measures or Goals or Objectives should focus on the core HRSA funded program activities of the grantee (i.e., program activities where the majority of a program’s resource, time, effort and/or staff are allocated.)

In the case of very complex grantee programs, however, it may be necessary to examine several major activities within the core program. Generally, these choices will be dictated by the size and complexity of the HRSA funded program and the nature of the HRSA funding. For example, if the HRSA funded program is large and complex and the HRSA funding is applied broadly across the program, such as a community health center (CHC), it will usually be necessary to measure performance for the major activities or subunits within the CHC, as well as for the entire CHC. Typically, where activities or subunits are measured, the performance review should focus on those subunits or activities that are most significant (i.e., where the most resources and staff are allocated) within the scope of the larger program.

The selection of Performance Review Measures or Goals or Objectives begins with answering three primary questions.

  • What are the grantee’s goals and objectives for the activity?

    When selecting Performance Review Measures or a grantee’s Goals or Objectives for review, it is important to consider the goals and objectives and program priorities of the awarding HRSA program.

  • Who is the population served (e.g., patients, students, customers)?
    For the purposes of selecting a Performance Review Measure, the “population served” will differ depending upon the program or major activity being measured. For example, if outreach activities are being measured for a health center (the term “outreach” is used here to mean activities designed to increase access to health care by underserved populations), then the “population served” will typically be the entire underserved population within the geographic area served by the program. If the outreach activities are for a certain type of care, such as a diabetes clinic, then the “population served” might be all underserved diabetics within the geographic area served by the clinic. (4)

    By contrast, if the delivery of health services is being measured, then the “population served” will typically be the clients actually receiving services and/or enrolled in a program (e.g., the diabetics enrolled in a diabetes clinic). In each case, if the program does not collect exact data on the number of people being served, then the number should be estimated based upon the best available data.


  • What are the specific services or resources provided to this population?

    This question refers to the first type of performance measures described above. (“How much service or resources is delivered?”). It should, therefore, be answered in concrete, quantifiable terms, indicating not only what are the services or resources provided to the “population served”, but also how much of those services are provided. For example, a financial aid program for nursing students might provide 100 students annually a total of $200,000 in financial aid and 2 hours per student of financial aid counseling. Note that the description of services or resources provided will often include an indication of the number of clients to whom the services or resources are being provided.

(4) If the population served is an estimated number, then the basis for the estimate should also be provided.

With the information provided by the answers to these three questions, the selection of Performance Review Measures or Goals or Objectives for a HRSA funded program or major activity can begin. In initial discussions with the grantee and fellow HRSA colleagues, the review team should discuss these questions to assure that all parties begin with the same basic assumptions.

Selecting Performance Review Measures: Identifying Outcome and Effort Measures

Outcome Measures and Effort Measures
Performance Review Measures should, where such measures are available, consist of a combination of outcome measures and effort measures, for several reasons. First and foremost, not all performance measures are created equal. It is more important to measure impact (outcome measures) than it is to measure how well a service is delivered (effort measures), and it is more important to measure how well a service is delivered (effort measures) than it is to measure how much of a service is delivered. The HRSA Performance Plan not only states that outcomes are the most important performance measures at HRSA, it also states that HRSA will work to increase the use of such measures.

Not All Performance Measures Are Created Equal

Second, even though these three kinds of performance measures are not equally important, each serves an essential role. As discussed above, measuring “how much” service or resources are provided is a necessary first step in selecting effort measures and outcome measures. Effort measures are necessary to understand how well a program is operating (for example, how efficiently it is run), but these types of measures do not provide data on the impact of the program. Likewise, outcome measures will provide data on impact, but not on how well the program is run.

Finally, the HRSA Performance Plan notes that, in implementing new programs, the first data available often are only data for measures of “how much” or “how well” service or resources are provided. The plan states that these types of measures are “frequently the most realistic and routinely measurable indicators of performance … are often the only indicators currently available on an annual basis and reflect the level of control an agency can bring to bear through particular programs.” The data for outcome measures remains essential; but often such data is not available until after the initial program start up is complete. It should be noted, however, that data on outcome measures are often most helpful and easier to collect if they are established at the outset of program design.

Identifying Performance Review Measures for Category 1 Grant Programs

The answers to the first three questions discussed above provide the essential parameters for the identification of the selected Performance Review Measures for a HRSA funded program or major activity: goals and objectives, the population served, and the services or resources provided. As suggested earlier, it is also important to discuss what the grantee is trying to accomplish (especially with respect to the impact on the population served) and how the grantee knows if it is making progress. In addition, when gathering data and information during the pre-site visit phase prior to the selection of performance review measures, the review team must assess the environment within which HRSA funded programs operate as well as the financial viability of the grantee.

With this information, the review team in collaboration with the HRSA project officer can, on a preliminary basis, identify Performance Review Measures (both outcome measures and effort measures) from the grantee’s pool of available performance measures for a program or major activity.

The review team in collaboration with the HRSA project officer(s) and other appropriate HRSA staff, contact the grantee to describe the performance review process and select the performance review measures. In selecting the performance review measures, resources that should be consulted include:

  1. HRSA program performance reporting requirements; and
  2. The grantee’s own measures and data, identified as program (including business management) goals and objectives in grant applications and progress reports, and often connected to a grantee’s strategic plans, annual plans, and/or work plans.
  3. Performance Review Measures required by several HRSA Bureaus and Offices

The next step in selecting Performance Review Measures for a HRSA funded program or major activity is to answer the following question:

From the pool of measures, what are the key outcome and effort measures?

It is essential in ensuring the usefulness of the selected Performance Review Measures to limit the total number of Performance Review Measures used. This allows the performance review to focus and develop a more in-depth analysis and set of options to improve program performance. Select no more than 5 Performance Review Measures per HRSA funded program or major activity (in total, both effort measures and outcome measures) from the pool of performance measures. Overall, to ensure that Performance Reports are concise and useful, the following limits on the number of Performance Review Measures are provided below:

# of Grant Programs
Max # of Performance Measures*
1
5
2
8
3
9
4
12
* For multiple funded grantees, please select cross-cutting measures whenever possible.

Once the pool of Performance Review Measures (outcome measures and effort measures) have been identified for the HRSA funded program or major activity in question, the next step is to select the program or major activity’s Performance Review Measures. To do so, the pool of Performance Review Measures should be rated using the following three criteria:

Communication Power

  • Are these measures of outcomes and effort communicated easily? Would those who pay attention to your work (e.g., taxpayers, legislators, HRSA project officers, clients) understand what this measure means?

Central Importance

  • Do they measure the most important outcomes and efforts? Is this a measure of something of central importance with respect to the program’s goals and objectives (e.g., the health of a baby at birth would be of central importance to a prenatal clinic)? Is this measure a good corollary for the many other measures that might also be used (i.e., would the trend represented by the data collected for this measure be similar to the trend represented by data collected for those other measures)?

Data Power

  • Are the data available and credible? To be credible, the data must be consistent and reliable. In addition, timeliness is necessary to track progress.

The process for applying these criteria is outlined on the diagram below: “Criteria for Selecting Performance Review Measures.” The measures that have high ratings for all three criteria are good choices for Performance Review Measures for the HRSA funded program or major activity. Of course, sometimes the best available data in the pool will not meet all three criteria. Following the graphic is a worksheet for applying the criteria, titled “Criteria for Selecting Performance Review Measures Worksheet.”

Grantee Performance Review Protocol:
Criteria for Selecting Performance Review Measures

Directions: List potential Performance Review Measures for the HRSA funded program or major activity in question and rate each as High, Medium, or Low on each criterion: Communication Power, Central Importance, and Data Power.

 

Criteria for Selecting Performance Review Measures

Criteria for Selecting Performance Review Measures Worksheet

Directions:

  1. List the Performance Review Measures for the program or major activity.
  2. Rate each measure as High, Medium or Low for each of the three criteria.
  3. . Rank the measures to select up to 5 Performance Review Measures: the ones that rank high or highest on all three criteria.
  4. Note the measures that rank high on communication and importance but low on data. These may be candidates for a “data development agenda”: (i.e., measures we would use if the data were available).

Suggested criteria for selecting Performance Review Measures:

Communication: Would an average resident, client or taxpayer understand what this measure means?
Central Importance: Does the measure say something of central importance about the outcome/effort the grantee is trying to measure? Is it a good corollary for similar measures?
Data: Does the grantee have quality data available on a timely basis?

Performance Review Measures
Criteria:
H, M, L
Rank
C
CI
D
         
         
         
         
         
         
         
         
         
         
         
         
         
         
         
         
In summary, Performance Review Measures should include a mix of outcome and effort measures, whenever possible. They should each represent a grantee’s efforts in a broader key area. Up to 5 Performance Review Measures may be presented for each HRSA funded program and/or major activity. All of these measures should reflect the core work of the grantee. Lastly, given the critical importance of the financial viability of HRSA funded programs, at least one financial measure should be considered in the mix of Performance Review Measures based on the pre-site visit financial analysis of the program(s).

A set of national performance review measures have been identified for use with several HRSA programs. For a listing of these programs as well as additional information on the measures, please visit http://www.hrsa.gov/performancereview.

Identifying Goals or Objectives for Category 2 Grant Programs

Similar to identifying Performance Measures for Category 1 Grant Programs, the identification of selected Goals or Objectives for the review of Category 2 Grant Programs should consider the communication power and the central importance of the goal or objective, i.e., by reviewing the grantee’s progress in attaining the stated goal or objective the public, the project officer, and stakeholders could generally determine the grantee’s success in implementing the approved grant program.

Graphing Performance Review Measures

Once the Performance Review Measures have been selected, the next step is to assess performance. The first step in assessing performance is to graph the data for a Performance Review Measure (5) .

In grant types where goals and objectives will be used in lieu of performance review measures, typically no data or very limited data will be available. Therefore, graphs will not be created.

Graphing Performance Review Measures

Once the Performance Review Measures have been selected, the next step is to assess performance. The first step in assessing performance is to graph the data for a Performance Review Measure.

Graphing Performance Review Measures
Performance Review Measure #1 Data Source: < >

The graph provides a concise, visual representation of a grantee’s performance on each selected Performance Review Measure. Each graph should include units of the measure on the vertical axis and units of time on the horizontal axis. The graph should also include a title and legend. When available and appropriate as comparisons, the graph may include the grantee’s goal/target, HRSA standards and/or national performance data. The graph should include at least three actual data points. Each graph should also provide the source of the Performance Review Measure data.

Assessing Performance

Once the data are graphed for a Performance Review Measure, performance may be assessed. Assessing performance means answering the following questions:

  • How is the grantee performing on each Performance Review Measure?
  • What are the grantee’s performance trends for each measure?
  • What is the gap from current to desired performance on each measure?

Performance Review Measures should be used first and foremost to assess the improvement of the grantee’s program against its own history. To this end, the graph should also include a trend line of the history of the program’s performance and, to the extent possible, a projection of future performance (i.e., the projected trend assuming there is no change in the grantee’s management of the program). These projections of future performance should be based on facts (i.e., past performance history) or informed opinions based on facts (i.e., similar grantee performance trends; etc.) – not just the grantee/review team’s specified goal.

Trend lines may also be used to compare the grantee’s performance with (1) a goal, (2) a standard in the field, or (3) the performance of other similar programs. These uses, however, should be approached with caution. Goals are often important for managing a program, but are not inherently a reliable reflection of a grantee’s performance potential; they are sometimes set unrealistically high or low based on reasons other than an objective assessment of a grantee’s potential performance. The reliability of standards also varies. Standards for effort measures are typically more well established and, therefore, more reliable than standards for outcome measures. Examples of standards for effort measures include: staffing ratios; application processing time; handicap accessibility; and waiting time. Standards for outcome measures are almost always more experimental, such as hospital recovery rates and morbidity/mortality rates.

Similarly, comparisons to other programs can lead to unfair and misleading conclusions when such comparisons are cases of “apples and oranges.” In the case of complex service systems, for example, the variables are usually so dynamic that it is hard to imagine a scenario in which the comparison of outcome measures between two such systems would, alone, be a reliable indication of program performance. Similar programs may have significantly different clients or environmental circumstances.

In the final stage of this component, the performance review answers the question “Is this acceptable?” The Performance Analysis that follows will either focus on improving performance if it is “not acceptable” or sustaining performance if it is “acceptable.” To the extent that a performance review can reach such a judgment about the grantee’s performance, the basis for that judgment, such as a comparison to a goal or standard should be stated clearly. By contrast, at this point in the Protocol, it is not necessary or even desirable to explain the underlying reasons for a grantee’s performance, which is the subject of the second component of the Protocol: Performance Analysis.

Performance Analysis

In order to develop a plan to improve performance, it is necessary to understand the factors that impact performance (i.e., those factors that contribute to and/or restrict performance with respect to each Performance Review Measure), answering the question:

  • What are the factors behind the grantee’s performance on each selected Performance Review Measure?

In answering this question, the grantee, in particular, has the opportunity to step back from its day-to-day operations and consider the larger issues it must deal with to improve performance. This question also presents the grantee with the opportunity to explain to others (including funders) the nature of the challenges it faces.

It should be noted that this analysis is important whether or not a grantee’s performance has met expectations. The analysis will: (1) identify those contributing factors that should be preserved to sustain performance; (2) identify restricting factors that are hindering performance; (3) identify factors beyond the control of the grantee that might have influenced performance (e.g., economic trends); (4) identify those effective elements of the grantee’s program that may be useful to other grantees engaged in similar work; and (5) determine whether performance can still be improved (for example, if a goal was, in retrospect, too low).

The worksheet below is a “force field analysis,” which can be used to organize the factors according to their contributing and restricting influences on efforts to turn or improve the curve of the performance trend.

Force Field Analysis Worksheet

Force Field Analysis Worksheet

In analyzing performance, the performance review should consider the full range of factors that might influence performance. The following questions address different possible factors:

  • How do the grantee’s internal systems and processes contribute to or restrict the grantee’s performance on each measure?
  • This question should include a review (when applicable) of one or more of the four organizational areas described in Appendix A of the Protocol:

    a) Capacity and Systems;
    b) Outreach and Consumer Feedback;
    c) Business and Financial Management; and
    d) Leadership and Staffing.

    In addition, the review team should review the relevant HRSA program expectations and application guidance for additional insight and inquiry points for the grantee, HRSA project officer(s) and other appropriate HRSA staff on the grantee’s internal systems and processes.

  • What external issues, including HRSA policies and environmental, political, demographic, and economic trends, are influencing the grantee’s performance on each measure?
  • In this area, it is important to note that external factors should not be immediately dismissed because they are perceived to be beyond the control of the grantee. A succinct Performance Analysis that demonstrates the significant impact of an external factor on a grantee’s performance may well serve as a catalyst for actions on the part of HRSA, HRSA program policy, and/or the grantee.

  • What are the grantee’s successes and challenges in forming and sustaining partnerships that support its performance on each measure?
  • The question of partnerships is critical to achieving outcomes. It also leads to one of the great temptations in working with outcome measures, (i.e., to use lack of control as an explanation for not accepting responsibility for those client outcomes).

    To some extent, this reaction is understandable. A prenatal counseling program does not control factors like drug dependence, domestic violence, unemployment, or homelessness that might influence whether or not a pregnant teenager who has gone through the counseling program has a healthy birth. Therefore, the question becomes: is it fair, to have healthy births as a performance measure for a prenatal counseling program?

The Matter of Control

Therein lies the paradox at the heart of performance accountability: the measures that matter the most (i.e., outcome measures) are the ones over which a grantee will have the least control. However, if complete control was the litmus test for choosing performance measures, there would be few, if any, outcome measures. By contrast, it is because client outcomes are so important that they are indispensable to the performance review process. Furthermore, most managers and front-line workers went into their fields of work because they wanted to make a difference in their clients lives, which usually translates into outcomes.

Achieving such outcomes, however, often inherently involves partnerships – engaging and working with those who have a potential role to play in achieving the desired outcomes. When a program becomes committed to client outcomes, it expands its influence by partnering with those who also have a stake in those client outcomes. The identification of factors impacting client outcomes points the way to the type of partners who need to be engaged and accountable. Therefore, a prenatal counseling program might well partner with domestic violence counselors, drug treatment programs, or an employment training center to better ensure that teenage moms have healthy births.

In addition to considering the full range of factors influencing the trend line for a performance measure, the performance review should seek to prioritize those factors, answering the question:
  • What is the order of priority among the factors examined – which are the most important to address to improve the grantee’s performance on each measure?

To identify those factors with the greatest influence or “leverage” with respect to performance, performance factors should be analyzed from two perspectives. First, it is important to not just identify the most immediate and easily observed factors impacting performance (i.e., the “proximate causes”), but to engage in the kind of rigorous analysis that will identify the underlying or more systemic factors (i.e., the “root causes”) that impact the grantee’s performance. One simple way to surface these root causes is to use the question “Why?” For example, if patient satisfaction is poor, as measured by a patient satisfaction survey, and the reason identified is that staff members are not adequately trained, ask “Why are staff members not adequately trained?” If the answer is that there is not enough time to train the staff, ask “Why is there not enough time to train staff.” Continue asking “Why?” until you have identified the root causes. Another successful method is to review on-site the identified key factors and/or processes (i.e., follow the patient tracking system; billing procedures; etc.) to determine how and whether, in fact, these factors/processes lead to the selected Performance Review Measure.

Second, once the root causes have been identified for the trend line for each Performance Review Measure, the performance review should prioritize those root causes based on their relative influence. The root causes that have the biggest impact (or “leverage”) on performance and, therefore, are the most critical to turning or improving the curve of the trend line, should be given the highest priority.

One concern at this juncture in the Protocol is that the Performance Analysis will almost inevitably identify the highest leverage performance factors as those that are also truly beyond the control of the grantee and its partners. For example, pervasive poverty resulting in a lack of income may be a root cause contributing to poor nutrition, which in turn is a proximate cause for poor birth outcomes. In this instance, even though pervasive poverty has a very big impact on birth outcomes, it may not be possible for either the grantee or its partners to have much impact on this economic condition.

The key at this juncture in the Protocol is not to try to determine whether or not solutions are available; rather, the performance review should simply record in its entirety their root cause analysis. In the example given, the performance review would record the links from pervasive poverty to income to nutrition to birth outcomes. This information will then, in turn, be used in the next step in the Protocol: generating options for improving program performance. It is in this next step, which takes place in the third component of the Protocol, discussed below, that the performance review will determine whether the grantee can only act on a more proximate cause or, alternatively, or in addition, whether there are any feasible actions that can be taken with respect to a root cause.

Using the example discussed above, if pervasive poverty cannot be readily addressed, the performance review might generate options such as engaging partners to help improve nutrition, such as the agency that administers the Woman Infant and Children (WIC) feeding program. On the other hand, or at the same time, the performance review might identify potential roles the grantee can play in efforts to address the issue of pervasive poverty systemically, such as participating in a local economic development initiative. While these examples of options to improve program performance are provided here to illustrate the importance of a rigorous and thorough root cause analysis; they are the subject of the next part of the Performance Analysis: Performance Improvement Options.

Key Area Discussion

During the course of the analysis of the performance review measures or goals or objectives, the review team, grantee, and project officer(s) if available, may identify additional facts regarding the implementation of the grant program that are not directly aligned with the measure but are worth discussing and recording. The discussion may include broader internal systems issues; external issues, including HRSA policies; identification of cross-cutting factors; as well as significant accomplishments and best or promising practices. Including the highlights of the discussion in the report will broaden HRSA’s understanding of the grantee and the grant program and will enhance HRSA’s understanding of the challenges and opportunities the grantee faces in achieving its mission.

Program Requirements

The focus of performance reviews is to measure program performance, analyze the factors behind performance and develop options and actions for performance improvement. In the course of a review, however, a review team may identify and review key compliance issues/program requirements. If identified, these issues should be discussed with the grantee and documented in the Performance Report. When documenting such an issue, the review team should refer to the current HRSA program expectations and application guidance for the applicable statutory and regulatory Program Requirements, where appropriate. In addition, during the pre-site conference calls with the HRSA Project Officer(s) and other appropriate HRSA staff, the review team should ask if there are any concerns with respect to Program Requirements that the review team should review with the grantee during the site visit.

Performance Report

The third component of the Protocol concerns the production of the Performance Report. The Performance Report includes an Executive Summary, a Table of Contents, Grantee Information, HRSA Grant Program Information, a Grantee Overview, Grantee Performance Information, and Program Requirements. The report records the contributing and restricting factors for each of the Performance Measures or Goals or Objectives and lists the Performance Improvement Options identified collaboratively between the review team and grantee. The development of performance improvement options and a discussion of actions the grantee can take to turn or improve performance on the Performance Review Measure (i.e., Performance Improvement Options) is the final step of the onsite review. Once the Performance Improvement Options have been developed, the review team drafts a Performance Report in which the Performance Improvement Options are documented for the grantee. Based on this on-site discussion or a subsequent conference call between the grantee, the review team and the project officer(s), if the project officer(s) was not part of the on-site review team, the grantee initiates the development of the Action Plan, which is the subject of the fourth and final component of the protocol.

Performance Improvement Options

While many options for actions to improve performance may be developed, the selection of Performance Improvement Options should be limited to ensure that the most important options are identified and addressed. On the other hand, enough Performance Improvement Options should be selected to present the grantee with a variety of choices for action. The following considerations are particularly important in developing Performance Improvement Options:

  • Does the Performance Improvement Option correspond with one or more of the root causes identified in the Performance Analysis?
    A key discipline in developing Performance Improvement Options is the “alignment” of those options with the root causes identified in the Performance Analysis. This alignment provides the rationale for selecting the Performance Improvement Option. Without such a rationale for the choice of a Performance Improvement Option, there is no tie to improving performance: the connection between the “ends” (as measured by the Performance Review Measure) and the “means” is lost.
  • How strongly will the Performance Improvement Option turn or improve the performance on the Performance Review Measure?
    In addition to alignment with the root cause(s) identified in the Performance Analysis, the Performance Improvement Options should also be assessed with respect to the prioritization given to those root causes. Given that resources are finite, decisions with respect to the dedication of resources (including staff time) to a Performance Improvement Option must be based on the expected impact of those resources on performance. In other words, the Performance Improvement Options presented should address the most important root causes identified in the Performance Analysis and, therefore, be geared to having the greatest potential impact on the performance trend for the corresponding Performance Review Measure. As noted in the discussion of the Performance Analysis, this concept is sometimes referred to as “leverage.”
  • Is the Performance Improvement Option feasible?
    Can it be done? This question is the necessary counterpart to the question of leverage. While the final determination of feasibility must be made by the grantee in selecting from among Performance Improvement Options for its Action Plan, feasibility should also be considered at least in broad terms in developing the Performance Improvement Options. Questions of feasibility, however, should be handled so as not to limit creativity. Sometimes the proposal of a potentially infeasible option will be the catalyst in the thinking process that leads to a highly creative and feasible solution to a problem. The importance of creativity is further discussed in the context of “no-cost/low-cost solutions,” below.
  • Are the Performance Improvement Options evidence-based?
    In weighing both leverage and feasibility, research is indispensable. Performance Improvement Options should be based to the extent possible on research or evidence-based practices (i.e., practices that have been documented to work elsewhere). There may, of course, be times that data is limited and a program must move forward with the best judgment of its experienced professionals who are working to improve performance. In most cases, however, the review team should challenge the grantee in its Performance Improvement Options to implement only practices that are research or evidence-based. Correspondingly, the review team should document the relevant source when noting the use of research-based or evidence-based practices.
  • Have “no-cost/low-cost” options been developed?
    As noted above, creativity is also a key to this work. Have all the best options available been identified? Often, however, such a discussion of what can possibly be done will turn to and get stuck on the need for additional funding. Funding is almost always a critical need and careful thought must be given to ways to increase funding where needed. However, the performance review should also ask the grantee to explore “no-cost/low-cost” options (i.e., options the grantee can pursue with existing staff and/or resources) to make sure the most potentially creative options are identified and explored. What assumptions does the grantee need to challenge about the current way things are done? Has the grantee considered ways it could work differently with existing resources and staff?
  • What partners may have a role to play in improving program performance?
    The critical role of partners was explored in the discussion of Performance Analysis, above. An important consideration in developing Performance Improvement Options is whether there are partners who may have a role to play in improving program performance, especially with respect to outcome measures. If there are partners with a role to play, then a Performance Improvement Option should include identifying the partner and providing a brief description of the contribution they can make to performance improvement.

Action Plan

Action Items

Finally, a grantee must decide specifically what it will do to improve performance (i.e., the Action Items). This process which should involve the grantee, the project officer(s), and the review team, culminates in the Action Plan, provides an opportunity for the grantee to reflect upon the Performance Report, including the Performance Improvement Options and Key Area Discussions, and describe its Action Items. The grantee is asked to describe the specific action items to be completed for each Performance Review Measure or Goal or Objective and, when necessary, describe the specific actions to be completed to address any identified Program Requirement issues. The grantee has the opportunity to request technical assistance to support the implementation of the Action Items.

The Action Plan builds on the work done in developing the Performance Improvement Options. The grantee should consider alignment with program purpose and priorities, leverage, and feasibility. With respect to leverage and feasibility, the grantee will often need to balance these two considerations in choosing the actions it will implement. A Performance Improvement Option that has high leverage and high feasibility will, of course, be a prime candidate for the Action Plan. The choice among other Performance Improvement Options, however, may involve some trade-offs between leverage and feasibility and will need to be weighed accordingly.

In addition, once the grantee has chosen a Performance Improvement Option, the Action Plan should provide greater specificity about what the grantee is intending to do. In other words, once the Performance Improvement Option becomes an action item in the Action Plan, the grantee should provide the details necessary to indicate by whom, what, where, when, and how the action will be implemented.

Action Plan Development

Action items should be a consolidation of both short-term and long-term priorities that will assist the grantee in achieving substantial improvement in overall performance. The 3-5 action items address concerns discovered during the performance review process that may relate to the program goals, program objectives or performance measures.

Action Item: The action item is a specific activity of central importance to the program that will improve the performance of the grantee. State how the action item is linked to a performance measure, performance improvement option, grant goal or objective, cross cutting key area or internal systems/process. Action items should be implemented within the lifecycle of the HRSA grant project period. Focus on low-cost and no-cost action items that can be accomplished within existing resources.

Expected Outcome: State the quantitative or qualitative outcome anticipated upon completion of the action item.

Key Steps with timeframes and parties responsible:

  • Key steps: List the specific steps the grantee plans to take to achieve its goals and objectives. The steps should be completed within the HRSA grant project period.
  • Timeframes: State the timeframe needed to complete the step.
  • Parties responsible: State who will assure that the action step is accomplished.

Action Plan Review: The review team and project officer conduct a joint review of the plan and provide feedback to the grantee on the specificity, feasibility and potential impact of the action items.

Status/progress/update: Report on the progress of the action items; including barriers encountered, needed revisions and technical assistance updates.

Action Plan Follow-up

Upon submission of the Action Plan, the review team and the project officer(s) review the Action Items to ensure that they are aligned with program purpose and priorities, that areas of concern will be addressed, and that specific outcomes are identified. The performance review concludes with the adoption of an agreed upon Action Plan to be implemented. The project officer has the lead responsibility for monitoring the action plan.

Please see the Action Plan/Technical Assistance Request Template on page 35 for further information on the recommended format.

Technical Assistance Guidance

Goal of Technical Assistance: HRSA grantees participating in performance reviews create an Action Plan to reflect and leverage the discussions and findings of the review and identify actionable items that will improve the grantee’s performance. The grantee may submit a technical assistance (TA) request in support of their Action Plan.

Technical Assistance Proposal Development: The TA request may address Program Goals, Program Objectives or Performance Measures. Types of TA requested include, but are not limited to: billing and collections; educational material development; medical records; financial management; marketing; patient flow; and practice management.

TA may be identified during the pre-site and/or on-site. The TA request will be submitted by the grantee to the OPR Review Team within 30 days of the on-site review along with the Grantee Action Plan. TA requests will range from 2 days to a maximum of 5 days.

Once the TA request has been submitted and approved by the OPR Team and Project Officer the OPR Team will forward it to the OPR Headquarters. The Headquarters office will submit a consolidated list of TA requests to the Bureaus/Offices on a monthly basis for approval and a decision regarding who will have the lead role of the TA. If the Bureau has the lead role in the TA process they will notify OPR when the TA has been completed and provide a summary of TA.

Other Periodic Reports

Review teams will submit items for the Environmental Snapshot Report and the Policy and Trends Report that document the key environmental trends/issues (i.e., regional, State and/or local) and/or policy issues (including HRSA policy issues) impacting program implementation and/or performance that were identified during the performance review process, describing their specific impact and providing recommendations to HRSA for addressing these issues, as appropriate.

Summary

In summary, the review team will work with grantees, HRSA project officer(s) and other appropriate HRSA staff in the performance review process to systematically and collaboratively apply the Protocol. They will “begin at the end” by selecting and assessing progress on Performance Review Measures or selected Goals and Objectives, analyzing performance and discussing Key Areas regarding the implementation of the grant, and generating Performance Improvement Options. This is documented by the review team in the Performance Report. Finally, the grantee, in collaboration with the review team and project officer(s) will develop Action Items for an Action Plan to improve performance.

2008 OPR Matrix of HRSA Grant Programs

HRSA Activity Code Bureau or Office CATEGORY 1 Services and Training with Measures (or required measurable goals) that have data CATEGORY 2 Training, Infrastructure, other programs not easily reviewed using Category 1 approach Program
B04 MCHB
X
MCHB Title V Block Grant
D04 ORHP
X
Rural Health Care Services Outreach Grant Program
D06 ORHP
X
Rural Health Network Development Program
D11 BHPr
X
Nurse Education Practice and Retention
D13 BHPr
X
Dental Public Health Residency Training Grants
D19 BHPr
X
Nursing Workforce Diversity
D62 BHPr
X
Comprehensive Geriatric Education Program
D66 BHPr
X
Nurse Education, Practice and Retention: Enhancing Patient Care Delivery Systems
D70
MCHB
X
State Implementation Grants for Integrated Community Systems for CSHCN
G22 OHIT
X
Telehealth Resource Center Grant Program
G96 MCHB
X
Improving Understanding of Maternal and Child Health and Health Care Issues
G97 MCHB
X
Partnerships to Promote Maternal and Child Health
H12 HAB
X
Ryan White Part D: Women, Infants, Children, Youth and Affected Family Members AIDS Healthcare, service delivery
H18 MCHB
X
State Systems Development Initiative
H1C BPHC
X
Native Hawaiian Health Care
H1M HAB
X
Licensure Portability Grant Program
H21 MCHB
X
Traumatic Brain Injury
H25 MCHB
X
Community Based Integrated Service Systems
H33 MCHB
X
EMSC Partnership Grants
H3H ORHP
X
Small Rural Hospital Improvement Program
H49 MCHB
X
Healthy Start-Eliminating Racial/Ethnic Disparities
H4A HAB
X
National AIDS Education and Training Centers
H57 MCHB
X
Integrated Health and Behavioral Health Care for Children And Adolescents
H59 MCHB
X
Healthy Behaviors in Women
H61 MCHB
X
Universal newborn hearing screening
H67 MCHB
X
Disparities in perinatal health - border initiatives
H76 HAB
X
Ryan White Part C Outpatient EIS Program
H80 BPHC
X
Health Center Cluster
H87 MCHB
X
Thalassemia
T02 MCHB
X
MCH Distrance Learning
T09 BHPr
X
Grow Your Own FQHC Nurse Traineeship
T12 BHPr
X
Grants to States to Support Oral Health Workforce Activities
T17 MCHB
X
Leadership Training in Pediatric Dentistry
T72 MCHB
X
Pediatric Pulmonary Centers
T73 MCHB
X
Leadership Education in Neurodevelopmental and Related Disorders Training Program
T76 MCHB
X
Maternal and Child Health Public Health Training Program
U01 MCHB
X
Partnership for State Title V MCH Leadership Community
U1E MCHB
X
Sickle Cell Treatment Demonstration Program
U1M BCRS
X
Native Hawaiian Health Scholarship Program
U22 MCHB
X
Heritable Disorders
U28 HAB
X
National Quality Improvement/Management Technical Assistance Center Cooperative Agreement
U32 MCHB
X
National Newborn Screening and Genetics Resource Center
U33 MCHB
X
Consumer Initiatives for Genetic Resources and Services
U37 MCHB
X
Newborn Screening Informatics Practice Network
U39 MCHB
X
Transition Services in Adolescent Health - Healthy and Ready to Work
U41 MCHB
X
Health Insurance and Financing/ CSHCN
U42 MCHB
X
Integrated Community Service Systems /CSHCN
U44 MCHB
X
Oral Health Resource Center & Partnership for Leadership in Oral Health
U45 MCHB
X
Adolescent Health/School-Based Health
U46 MCHB
X
3 Grants: Trainig Institute for Training Consultants with Child Care
U48 MCHB
X
Sudden Infant Death Syndrome
U49 MCHB
X
Child and Adolescent Injury and Violence Prevention Resource Centers Cooperative Agreement
U50 MCHB
X
Healthy Tomorrows Cooperative Agreement
U51 HAB
X
PEPFAR/Global AIDS Initiative: Rapid Expansion of Antiretroviral Therapy Program-Selected African & Caribbean Countries
U52 MCHB
X
Universal Newborn Hearing Screening - Cooperative Agreement
U53 MCHB
X
Cooperative Agreement - Evidence Base for Systems of Care
U56 ORHP
X
Rural Assistance Center for ORHP Cooperative Agreement
U58 BPHC
X
State and Regional Primary Care Associations
U59 MCHB
X
National Maternal and Child Health Data Resource Center
U68 BHPr
X
State Primary Care Offices (PCOs)
U69 HAB
X
National Training and Technical Assistance
U76 BHPr
X
Basic/Core Area Health Education Centers
U77 BHPr
X
Model State-Supported Area Health Education Centers
U91 HAB
X
Global AIDS Initiative: International AIDS Education and Training Center
U92 HAB
X
Global AIDS Initiative: Nursing Initiative
U97 HAB
X
Global AIDS Initiative: International Twinning Center
X82 MCHB
X
Traumatic Brain Injury Protection and Advocacy

 

Part I: Action Plan  
Action Item: Target Completion Date:
Expected Outcome:  

 

Key Steps
Timeframe
Responsible Party
Status
Bureau
         
         
         
Transition Date to Grant Application:

Additional Comments:


Technical Assistance Request

Part II: Technical Assistance Request

Section I: To be Completed by Grantee:

Summary of TA Request:

Expected Outcome:

Estimated Number of Days:


Section II: To be Completed by OPR:

Region:
Team Lead:
Grant #:
Bureau/Office Funded:
Project Officer:
OPR Review Team Concurrence with Request:
OPR Comments:
Project Officer Comments:
Recommendation Bureau/Office to Lead TA:

 

DEPARTMENT OF HEALTH & HUMAN SERVICES
HEALTH RESOURCES AND SERVICES ADMINISTRATION

Office of Performance Review (1)
<City> (2) Regional Division
<Street Address, Room/Suite Number>(3)
<City, State Zip>

Performance Report
for

<Grantee Name> (4)
<Grantee City, State>
EIN # - <###-###-###>

Date of Site Visit - <Month/Day/Year>(5)

<Name> (6), Review Team Leader
<Review Team Member Name>
<Review Team Member Name>
<Review Team Member Name>
<Name>, Review Team Consultant
<Name>, Observer

<Month/Day/Year> (7)

 

Executive Summary (8)

TABLE OF CONTENTS (9)

I. Grantee Information

II. HRSA Grant Program Information

III. Grantee Overview

IV. Grantee Performance

A. Performance Measure or Goal or Objective #1

B. Performance Measure or Goal or Objective #2

C. Additional Performance Measures or Goals or Objectives

V. Program Requirements

I. Grantee Information (10)

A. Name
B. Address: < >
C. Phone Number: <(###) ###-####>
D. Fax Number: <(###) ###-####>
E. E-mail Address: < >
F. Executive Director: <Name>
G. Grantee Summary: < >

II. HRSA Grant Program Information (11)

A. HRSA Grant Program #1: <Name of HRSA Grant Program>
1. Grant Number: <#>
2. Project Period: <Month/Day/Year - Month/Day/Year>
3. Budget Period: <Month/Day/Year - Month/Day/Year>
4. Amount of HRSA Funding as of <Site Visit Date>: <$ >
5. Program Director: <Name>
6. HRSA Project Officer: <Name>
Phone Number: <(###) ###-####>
E-mail Address: < >
7. HRSA Grants Management Specialist: <Name>
Phone Number: <(###) ###-####>
E-mail Address: < >


III. Grantee Overview (12)

A. Background on grantee organization
B. Description of HRSA grants received (number and value by Bureau/Office)
C. Population Served
D. Services/resources provided by the HRSA funded program

IV. Grantee Performance (13)

(for Category 1 Grant Programs)

A. Performance Review Measure (14)< >
1. HRSA Grant Program(s): (15)< >
2. Population Served: (16)< >
3. Services Provided: (17)< >

Performance Review Measures

Data Source:(19)< >

4. Performance Trend: (20) <graph and performance trend only if using a Performance Measure>
5. Performance Analysis (21)
a. Contributing Factors (CF)
CF1. < >
CF2. < >
b. Restricting Factors (RF)
RF1. < >
RF2. < >
6. Key Area Discussion/Promising Practices (22)
7. Performance Improvement Options (23)
a. (CF#, RF#)
b. (CF#, RF#)

B. (Repeat format used in “A” for each additional performance measure)

OR

(for Category 2 Grant Programs)

A. Program Goal or Objective (24) < >
1. HRSA Grant Program(s): (25)< >
2. Population Served: (26) < >
3. Services Provided: (27)< >
4. Performance Analysis (28)
a. Contributing Factors (CF)
CF1. < >
CF2. < >
b. Restricting Factors (RF)
RF1. < >
RF2. < >
5. Key Area Discussion/Promising Practices (29)
6. Performance Improvement Options (30)
a. (CF#, RF#)
b. (CF#, RF#)

B. (Repeat format used in “A” for each additional goal or objective)

V. Program Requirements (31)

A. Program Requirement #1:(32) < >
1. HRSA Grant Program(s): < >
2. Description of Issue: < >

Foot Notes

(1) Use the following template to format your report, including the following general report parameters: use 12 pt Times New Roman black font with standard margins; mark “DRAFT” on the draft Performance Report; spell out acronyms in the first instance when they appear in the report; and when using the automatic date function in MS Word, make sure it is a static date, rather than a date that changes every time the document is opened.

(2) Insert the name of the city in which your regional division is located.

(3) Insert your regional division’s address.

(4) Insert the grantee’s name, the city and State the grantee is located in, and the CRS-EIN. The CRS-EIN should contain twelve characters and can be found on the notice of grant award and in the grants database in eRoom. Also, please insert the grantee’s name in the footer of the Performance Report.

(5) Insert the site visit date.

(6) Insert the names of the review team lead, review team members, observers and/or consultants, as appropriate.

(7) Insert the date the Performance Report was completed. For the draft Performance Report, this should be the date is sent to the grantee for comments. For the final Performance Report, the date should be the date the final report is sent to the grantee.

(8) The Executive Summary should be a brief overview of the report designed to give readers a quick preview of its contents. It should describe the organization (Report Section I), contain the HRSA grants received and reviewed (Report Section II), provide key findings of the review (Report Section IV and V), and special accomplishments (Report Section IV). Summarize key information from Section IV. Summarize any special accomplishments or promising practices noted in Section IV.

(9) The Table of Contents should be completed after the report is completed and should list at a minimum each of the major (Roman Numeral) sections and Performance Measure or Program Goal or Objective

(10) Insert the grantee’s name, address, phone and fax numbers, e-mail address, and the name of the grantee’s Executive Director. .

(11) For each of the grantee’s HRSA funded programs, provide the requested grant/program information:

  • In the outline, please state the HRSA grant program name next to the heading “HRSA Grant Program #x:”. Each of the grantee’s HRSA grant programs should be listed as a separate letter in the outline (i.e., A. HRSA Grant Program #1: < >; B. HRSA Grant Program #2: < >; etc.).
  • For each HRSA grant program, list the grant number, the project period, the budget period and the amount of HRSA funding as of the site visit date. All of this information is located in the grants database in e-room; however the amount of HRSA funding needs to be verified a week before the site visit by entering the Electronic Handbook and looking at item 12A in the most recent Notice of Grant Award (NGA). If there are several awards for a given fiscal year for an individual program, please make sure to include all the award amounts when computing the amount of HRSA funding.
  • Insert the name of the program director as well as the name and contact information for the project officer and grants management specialist.

(12) The Grantee Overview should provide the reader with a sense of the history of the grantee, including the type of organization (i.e., university; hospital; not for profit organization; public entity, etc.) as well as a brief description of its HRSA funded program(s), including the services provided by these programs and the populations receiving these services. It may also include State and community health indicators/environmental information to give the reader a better sense of the needs of the community the grantee serves or the environment the grantee operates in. Additional background information may include a description of the area served and the percentage of the grantee’s budget that comes from HRSA; this information will provide the reader with a better understanding of the grantee and the HRSA contribution/investment in the organization.

(13) Grantee performance for Category 1 grants is reviewed through the use of Performance Measures. Grantee performance for Category 2 grants is reviewed through the use of Program Goals and Objectives. Follow the appropriate outline provided.

Up to 5 Performance Review Measures or Program Goals and Objectives may be presented for each HRSA funded program and/or major activity. As a general rule, Performance Review Measures or Program Goals and Objectives should focus on the core HRSA funded program activities of the grantee (i.e., program activities where the majority of a program’s resources, time, effort and/or staff are allocated.) In the case of very complex grantee programs, however, it may be necessary to examine several major activities within the core program. Generally, these choices will be dictated by the size and complexity of the HRSA funded program and the nature of the HRSA funding. For example, if the HRSA funded program is large and complex and the HRSA funding is applied broadly across the program, such as a community health center (CHC), it will usually be necessary to measure performance for the major activities or subunits within the CHC, as well as for the entire CHC. Typically, where activities or subunits are measured, the performance review should focus on those subunits or activities that are most significant (i.e., where the most resources and staff are allocated) within the scope of the larger program.

A set of national performance review measures have been identified for use with several HRSA programs. .

Overall, to ensure that Performance Reports are concise and useful, the following page limits are outlined below. Please use the chart below to determine the maximum number of Performance Review Measures or Program Goals and Objectives a report should include and the maximum report length (including graphs):

Number of HRSA Grant Programs/Major Activities Maximum Number of Performance Review Measures or Program Goals and/or Objectives* Performance Report – Max Number of Pages**
1
5
15
2
8
24
3
9
27
4
12
36


* For multiple funded grantees, please select cross-cutting measures or program goals or objectives whenever possible.
** Does not include Executive Summary, cover sheet, table of contents, or Grantee Contact/HRSA Grant Program Information.

(14) For each Performance Review Measure (Category 1), state the measure next to the appropriate heading. Each Performance Review Measure should be listed as a separate letter in the outline.

(15) For each Performance Review Measure, list the HRSA grant program(s) to which it relates.

(16) For each Performance Review Measure, describe the population served to which it relates. This description should include number of individuals served by the activity/service that relates to the Performance Review Measure being analyzed. In cases where the Performance Review Measure cuts across multiple HRSA grant programs, the populations served should reflect the populations served by all of the programs.

(17) For each Performance Review Measure, describe the services provided. This description should include the number and type of the services and/or resources provided relating to the performance measure. In cases where the Performance Review Measure cuts across multiple HRSA grant programs, the services/resources provided should reflect the services/resources provided by all of the programs.

(18) Use of graphs applies primarily to Category 1 grants and the use of Performance Measures. Use of Program Goals or Objectives usually do not lend themselves to graphing. Where graphs are used they must be legible. Please label the axes and include a title (i.e., Performance Review Measure and/or Program Goal or Objective) and legend. The axes should reflect the full scale, so the performance trend is correctly depicted. Graphs should be done in black and white and use different line formats to distinguish between multiple lines on a graph. Include a projected trend when possible (see endnote 18 below). When available and appropriate as comparisons, the graph may also include the grantee’s forecast for the current year, goal/target, HRSA standards and/or national performance data. The graph should include more than one actual data point. If there are not at least two data points for a measure, please consider for the Data Development Recommendations section below.

(19) Please state the source of the data used for the Performance Review Measure. Also, as appropriate, please provide a brief explanation of the reliability of the data. If a graph includes percentages, define the numerator and denominator in this section.

(20) To the extent possible, comment on the performance trend and provide the basis for comparison (e.g., national standards; grantee’s historical performance; another grantee’s performance; HRSA targets; etc.) Also, please include the review team’s professional judgment on whether the grantee’s current performance will continue at the same rate or improve or decline. These projections of future performance should be based on facts (i.e., past performance history) or informed opinions based on facts (i.e., similar grantee performance trends; etc.) – not just the grantee/review team’s specified goal. This item will be used only if a grantee’s performance has been graphed.

(21) Please list and describe the primary factors (2-4), in priority order that contribute to and restrict the grantee’s performance on the Performance Review Measure. Each Contributing/Restricting factor should align with the Performance Review Measure. For each Contributing Factor, state the factor next to the heading “CF#.”. Each Contributing Factor should be listed as a separate letter in the outline (i.e., CF1. < >; CF2. < >; etc.). Please follow the same format for each Restricting Factor (i.e., RF1. < >; RF2. < >; etc.)

(22) Include highlights of discussion with grantee regarding the grantee’s internal systems and processes, external issues (including HRSA policies and environmental, political, demographic, and economic trends) that are not captured in the contributing and restricting factors but impact on the grantee’s success in achieving its goals and objectives and sustaining successful partnerships. Also describe significant accomplishments and promising practices.

(23) List and describe, in priority order, 2-3 Performance Improvement Options for the Performance Review Measure. Each Performance Improvement Option should align with the performance factors and the Performance Review Measure. For each Performance Improvement Option, please identify at the end of the option in parentheses [e.g., (CF1; CF4; RF2; RF5)] all the Contributing and/or Restricting Factors that are being addressed. Please include evidence based, low/no cost options, potential partnerships (be specific) as well as technical assistance needs, as appropriate.

(24) For each Program Goal or Objective (Category 2), state the goal or objective next to the appropriate heading. Each Goal or Objective should be listed as a separate letter in the outline.

(25) For each Program Goal or Objective, list the HRSA grant program(s) to which it relates.

(26) For each Program Goal or Objective, describe the population served to which it relates. This description should include number of individuals served by the activity/service that relates to the Goal or Objective being analyzed. In cases where the Goal or Objective cuts across multiple HRSA grant programs, the populations served should reflect the populations served by all of the programs.

(27) For each Program Goal or Objective, describe the services provided. This description should include the number and type of the services and/or resources provided. In cases where the Program Goal or Objective cuts across multiple HRSA grant programs, the services/resources provided should reflect the services/resources provided by all of the programs.

(28)Please list and describe the primary factors (2-4), in priority order that contribute to and restrict the grantee’s performance on the Goal or Objective. Each Contributing/Restricting factor should align with the Goal and Objective. For each Contributing Factor, state the factor next to the heading “CF#.”. Each Contributing Factor should be listed as a separate letter in the outline (i.e., CF1. < >; CF2. < >; etc.). Please follow the same format for each Restricting Factor (i.e., RF1. < >; RF2. < >; etc.)

(29) Include highlights of discussion with grantee regarding the grantee’s internal systems and processes, external issues (including HRSA policies and environmental, political, demographic, and economic trends) that are not captured in the contributing and restricting factors but impact on the grantee’s success in achieving its goals and objectives and sustaining successful partnerships. Also describe significant accomplishments and promising practices.

(30) List and describe, in priority order, 2-3 Performance Improvement Options for the Program Goal or Objective. Each Performance Improvement Option should align with the performance factors and the goal or objective. For each Performance Improvement Option, please identify at the end of the option in parentheses [e.g., (CF1; CF4; RF2; RF5)] all the Contributing and/or Restricting Factors that are being addressed. Please include evidence based, low/no cost options, potential partnerships (be specific) as well as technical assistance needs, as appropriate.

(31) A Program Requirement is presented where a grantee is not in compliance with a requirement of applicable program statutes, regulations and/or grant award conditions.

(32) For each Program Requirement, please state the requirement next to the heading “Program Requirement #x:”. Each Program Requirement should be listed as a separate letter in the outline (i.e., A. Program Requirement #1: < >; B. Program Requirement #2: < >; etc.). For each Program Requirement, please specify the HRSA grant program(s) that the requirement relates to and provide a description of the issue(s) that led to the grantee’s non-compliance. [If there are no Program Requirement issues identified, please delete this section of the Performance Report (i.e., the Performance Report would include no reference to a section IV. Program Requirement).]