|
![](https://webarchive.library.unt.edu/eot2008/20080916005229im_/http://hrsa.gov/images/submain/1x1space.gif) ![](https://webarchive.library.unt.edu/eot2008/20080916005229im_/http://hrsa.gov/images/submain/1x1space.gif) |
![](https://webarchive.library.unt.edu/eot2008/20080916005229im_/http://hrsa.gov/images/submain/1x1space.gif) |
![](https://webarchive.library.unt.edu/eot2008/20080916005229im_/http://hrsa.gov/images/submain/1x1space.gif) ![](https://webarchive.library.unt.edu/eot2008/20080916005229im_/http://hrsa.gov/images/submain/1x1space.gif) |
Performance
Review
Protocol Guide: Principles and Implementation
Dear Colleagues:
Dear Colleagues:
We are pleased to share with you the attached
Performance Review Protocol Guide: Principles
and Implementation. We hope you find this
guide helpful and informative, as you proceed
with the performance review process.
As part of the Health Resources and Services
Administration’s overall efforts to
improve the nation’s health and access
to quality health services, the Office of
Performance Review (OPR) plays a central
role in achieving the HRSA mission by reviewing
and enhancing the performance of HRSA-supported
programs within communities and States.
The purpose of performance reviews is to
improve performance of HRSA funded programs
by working collaboratively with grantees
and HRSA Bureaus/Offices to measure program
performance, analyze the factors impacting
performance, and identify effective strategies
and partnerships to improve performance,
with a particular focus on outcomes.
Performance reviews also create the unique
opportunity to assess HRSA funded programs
within the context of the grantee’s
entire organization. Viewing programs from
such an integrative and comprehensive perspective
provides a deeper understanding of a grantee’s
performance and the impact or effectiveness
of HRSA funded programs on the public’s
health. The Performance Review Protocol
is a document that sets forth the basic
framework (four primary components) for
conducting the performance reviews. As designed,
the Protocol is meant to be dynamic and
flexible to ensure its applicability across
the wide range of HRSA funded programs.
The Performance Review Protocol Guide –
Principles and Implementation is a companion
to the Performance Review Protocol. The
Protocol Guide was crafted to facilitate
and direct the analytical and implementation
process, by providing an in-depth explanation
of the principles and methodology for conducting
performance reviews. The guide is meant
to build upon the performance review framework
presented in the Protocol and should serve
as a resource for OPR’s internal and
external stakeholders to illustrate our
approach to performance reviews.
OPR is committed to continuing our collaboration
with grantees and our fellow HRSA colleagues
through this performance review process.
By working in partnership, OPR will best
achieve its mission of improving and enhancing
the performance of HRSA-supported programs
within communities and States.
Thank you for your time and interest.
Rebecca Spitzgo
Associate Administrator
Office of Performance Review
Performance
Review Protocol – Principles and Implementation
Key Principles and Concepts
The Protocol is a framework which may
be applied to the wide variety of HRSA funded
programs and grantees. To apply the Protocol
effectively, one must understand its underlying
principles and concepts, described below.
-
Essential
Characteristics of a Good Performance
Review System
A good performance review system
should have the following characteristics:
- Useful
- Common
sense
- Simple
- Uses
plain language
- Uses
minimum paper
-
Data-driven
Decision Making
Effective performance management systematically
accesses and uses the best quantitative
and qualitative data and data analysis
available at every stage of decision making.
For example, the ability to manage performance
and, in particular, to achieve an impact
for the population served by a program
is inseparable from the ability to assess
whether such an impact has been achieved.
Progress on performance measures and/or
goals and objectives, provide such an
assessment – a “bottom line”
– in managing for performance.
- Language
Discipline
Too often terms are used interchangeably
and with a multiplicity of definitions in
talking about performance. The Protocol
recognizes the importance of the disciplined
(clear, consistent, and jargon-free) use
of language.
-
Population-level
Accountability vs. Program Performance
Accountability
In thinking about accountability, it is
important to distinguish between accountability
for achieving a condition of well being
for (1) a whole population (e.g., all
of the children in a county who are under
six years old) and (2) the population
or clients served by a program, agency
or service system (e.g., the clients of
a HRSA grantee). The significance of this
distinction lies in the recognition that
a single program, agency or even service
system cannot take sole responsibility
(or credit) for achieving a desired condition
of well being for a whole population.
-
Talk
to Action
Importantly, the Protocol gets from talk
to action quickly.
(1) Adapted from materials authored by
Mark Friedman
-
Ends to Means
The Protocol starts with
ends defined as either Performance Review
Measures or selected Program Goals and
Objectives and works backwards to means
(Performance Analysis, Key Area Discussion,
Performance Improvement Options, and Action
Items). Starting with ends – asking
what we want for whom at the outset –
saves countless hours and reams of paper
typically expended in most review processes.
It also provides focus: we know why we
are here. The progression of the Protocol
from ends to means involves four basic
steps (these steps mirror the four components
of the Protocol) which should first be
understood in general terms:
Step 1. First, define
what we want for whom (e.g.,
babies born healthy) in measurable
terms when there are adequate data
(e.g., % of prenatal clients who had babies
with low birth weights) or what goal or
objective is desired. In the cases where
a measure is chosen, the data for the
measure is graphed to show both historic
and projected trends. Following a review
of the performance trend, the question
becomes one of means: how to change (or
accelerate, if positive or the same) the
trend line (i.e., how to “turn (improve)
the curve”) of the trend line. In
the cases where there is a dearth of measurable
data or an inability to trend data, an
examination of the grantee’s application
should yield a set of three to five significant
goals and objectives on which to focus
the performance review.
Step
2. To turn the curve for a specific
performance measure or goal or objective,
start by analyzing the performance of
the grantee with respect to the particular
performance measure. Identify the most
important factors that must be addressed
to improve the performance trend. This
is also an opportunity to explore what
other issues that may not be directly
aligned with the performance measure but
are important to relevant to the grantee’s
overall performance and operations.
Step 3. Once factors
associated with performance have been
identified, determine what works
to address those factors, and what the
available options are for actions
that will turn the curve.
Step
4. Finally, deciding on a specific
action plan to turn the curve
and improve performance.
Following the implementation of the Action
Plan, the progression begins again, starting
with determining whether there has been
progress in turning the curve of the performance
trend line. The circle chart (2)
on the following
page illustrates the four steps
(2)
This circle chart is an adaptation of the
circle chart developed by Roger Fisher and
discussed in Getting to Yes, by Roger Fisher,
et. al.
Performance
Review Protocol:
Ends-to-Means Circle Chart
-
Broadening the Focus
The protocol encourages the focus of
the analysis be broadened beyond the
individual performance review measure
or goal or objective. This should include
a discussion of “Key Areas”
relating to grant implementation, best
or promising practices, or the challenges
and opportunities the grantee faces
in achieving its mission.
-
Collaboration
Finally, before moving ahead with a
detailed discussion of the four components
of the Protocol, it is important to
note that the Protocol – and the
performance review – is designed
to be conducted in collaboration. The
Protocol is, from this perspective,
a tool for collaboration: it provides
a framework from which the review team,
in collaboration with the HRSA project
officer(s) and the grantee work together
in selecting key Performance Review
Measures or Goals and Objectives, analyzing
the factors affecting performance, and
developing options and actions to improve
the grantee’s performance.
More specifically, the Protocol requires
systematically working through each
of the four components, making performance
data and analysis explicit, applying
certain decision making criteria, and
ultimately developing a set of actions
that will improve performance. The Protocol
both provides for efficient participation
and garners the best thinking of those
involved.
- Performance Review Approach
The Performance Review Protocol is a
performance improvement tool used with
each grantee to review its HRSA funded
programs. Due to the wide variability
in the scope and purpose of HRSA grant
programs, the Protocol provides for two
approaches to accommodate these variances
and best facilitate improvement for each
type of grant program.
All
HRSA grant programs (including cooperative
agreements) eligible for a review are
designated by OPR as either a Category
1 or Category 2 grant program for the
purpose of conducting a performance review.
(See Matrix of HRSA Grant Programs for
listing of grant programs by category).
- Category
1 includes: health services delivery
activities and professional training
programs with measurable data.
-
Category 2 includes: grant programs
funding professional training and
public health infrastructure development
that do not lend themselves to using
the same approach used for Category
1 grant programs typically due to
small numbers of recipients/clients,
limited term funding, or the scope
of the activity.
The review of Category 1 grant programs measures
program performance, analyzes the factors
impacting performance, and identifies effective
strategies and partnerships to improve program
performance. The review of Category 2 grant
programs analyzes the factors impacting the
ability of the grantee to meet selected goals
and objectives of the grant program and/or
achieving sustainability beyond the end of
HRSA support, and identifies effective strategies
and partnerships to improve performance/progress
on program goals and objectives.
For Category 1 grant programs, the four primary
components of the protocol are:
- Performance Review Measures
- Performance Analysis and Key Area Discussion
- Performance Report
- Action Plan/Technical Assistance/Follow-up
For Category 2 grant programs, the four primary
components of the protocol are:
- Goals and Objectives
- Discussion about Progress/Impact/Sustainability
- Performance Report
- Action Plan/Technical Assistance/Follow-up
The Four Components of the Protocol
The Protocol, incorporating the principles
and concepts discussed above, consists of
the following four components, which are
embedded in the previous Circle Chart.
![Four Components of the Protocol and the Ends-to-Means Circle Chart](images/endstomeans2.jpg)
The following is a detailed description
of the process for implementing each of
the four
components.
Performance Review Measures
Goals or Objectives
The Protocol begins with selecting Performance
Review Measures or Goals and Objectives
for each HRSA funded program based on the
grant program’s categorization –
Category 1 grant programs being those with
measurable data versus Category 2 grant
programs being those that do not lend themselves
to the development of performance measures
due to small numbers of recipients/clients,
limited term funding, or the scope of the
activity. Please see the Matrix of HRSA
Grant Programs on page 32 for a list of
grant programs by category. As will be shown,
the Performance Review Measures or Goals
and Objectives in many respects define the
scope and focus of the performance review
because the performance review and its two
primary documents, the Performance Report
and the Action Plan, all “begin at
the end” with the Performance Review
Measures or Goals and Objectives.
Performance Measures
In order to discuss Performance Review
Measures, we must first define performance
measures. For the purposes of the Protocol,
a performance measure is a set of data that
answers one of the three following questions
with respect to a program or major activity:
-
How much service
(or resources) is delivered?
- How well is it delivered? In
the Protocol, these kinds of measures are
called “effort measures.”
- What impact (3)
is produced? In the Protocol, these kinds
of measures are called “outcome
measures.
(3) The term “impact”
is not meant to imply an “impact analysis,”
which would involve isolating variables to
scientifically verify a causal relationship
between a specific activity and a corresponding
result. Such impact can, at best, be inferred
from the knowledge that certain activities
occurred and certain results followed.
The following table provides some common
examples of these three different kinds
of performance measures:
Common Examples of the Three Different
Kinds of Measures
1.
How much? |
Dollars
Spent
Number of target population/clients/customers
served
Number of activities (by type of service) |
2.
Effort Measures
(How well?) |
Administrative
efficiency: overhead as a percentage
of total cost; unit cost
Staff measures: client
staff ratio; workload ratio; staff turnover
rate; staff morale; % staff fully trained;
% clients seen in their own language
Access measures: waiting
time; waiting lists
Activity-specific measures:
% timely; % clients completing activity;
% correct and complete; % meeting standard |
3.
Outcome Measures
(Impact?) |
Client
results or client outcomes:
The % of clients achieving certain skills/knowledge
(e.g., nursing skills)
The % of clients achieving a desired
behavior (e.g., preventative behaviors,
adherence to prescribed medical regimens)
The % of clients achieving a desired
circumstance (e.g., access to health
care, a desired health status) |
The next three tables provide examples of
the three kinds of performance measures
for different programs:
Nursing
Education Program |
1.
How much? |
Number
of students |
2.
Effort Measure (How well?) |
Student-teacher
ratio |
3.
Outcome Measure (Impact?) |
%
of nursing students who enter nursing
upon graduation |
Health
Clinic |
1.
How much? |
Number
of patients treated |
2.
Effort Measure (How well?) |
%
of patients treated in less than one
hour |
3.
Outcome Measure. (Impact?) |
%
of babies born healthy (e.g., birth
weights) |
Drug/Alcohol
Treatment Program
|
1.
How much? |
Number
of persons treated |
2.
Effort Measure (How well?) |
%
of staff with training certification |
3.
Outcome Measure (Impact? |
%
of clients off alcohol or drugs: (a)
at exit;
(b) 12 months after exit |
For outcome and effort measures, a measure
that is a percentage is generally a more
useful than an absolute number. The percent
tells the impact with respect to the total
population served; a number will not provide
that information or context. For example,
it is more meaningful to know what percent,
rather than what number, of women who were
provided prenatal services had babies with
low birth weights. It is also extremely
powerful to take the data for a Performance
Review Measure collected over time and present
it in a graph to portray performance trends.
The following description of the steps
by which Performance Review Measures are
selected explains the role of each of the
three kinds of performance measures and
shows how Performance Review Measures are
graphed.
Goals or Objectives
In order to assess progress in attaining
a grant program’s goals or objectives
the reviewer must be familiar with the grantee’s
recent grant applications and progress reports.
The grant application, in response to the
HRSA Bureau or Office’s funding guidance,
will detail specific goals and objectives
that it will achieve to advance HRSA’s
mission and priorities. Periodic reports
submitted by the grantee will highlight
progress in attaining its stated goals and
objectives.
Selecting Performance Review Measures
and Goals and Objectives: HRSA Funded
Programs and Activities
It is essential to understand that a Performance
Review Measure or Goal or Objective may
apply to an entire HRSA funded program or
a major activity within the HRSA funded
program. Therefore, before selecting a Performance
Review Measure or Goal or Objective, the
performance review must, as a prerequisite,
be clear as to whether it is measuring performance
with respect to an entire HRSA funded program
or a major activity within the HRSA funded
program. As a general rule, Performance
Review Measures or Goals or Objectives should
focus on the core HRSA funded program activities
of the grantee (i.e., program activities
where the majority of a program’s
resource, time, effort and/or staff are
allocated.)
In the case of very complex grantee programs,
however, it may be necessary to examine
several major activities within the core
program. Generally, these choices will be
dictated by the size and complexity of the
HRSA funded program and the nature of the
HRSA funding. For example, if the HRSA funded
program is large and complex and the HRSA
funding is applied broadly across the program,
such as a community health center (CHC),
it will usually be necessary to measure
performance for the major activities or
subunits within the CHC, as well as for
the entire CHC. Typically, where activities
or subunits are measured, the performance
review should focus on those subunits or
activities that are most significant (i.e.,
where the most resources and staff are allocated)
within the scope of the larger program.
The selection of Performance Review Measures
or Goals or Objectives begins with answering
three primary questions.
-
What are the
grantee’s goals and objectives for
the activity?
When selecting Performance Review Measures
or a grantee’s Goals or Objectives
for review, it is important to consider
the goals and objectives and program priorities
of the awarding HRSA program.
-
Who is the population
served (e.g., patients, students, customers)?
For the purposes of selecting
a Performance Review Measure, the “population
served” will differ depending upon
the program or major activity being measured.
For example, if outreach activities are
being measured for a health center (the
term “outreach” is used here
to mean activities designed to increase
access to health care by underserved populations),
then the “population served”
will typically be the entire underserved
population within the geographic area
served by the program. If the outreach
activities are for a certain type of care,
such as a diabetes clinic, then the “population
served” might be all underserved
diabetics within the geographic area served
by the clinic. (4)
By contrast, if the
delivery of health services is being
measured, then the “population
served” will typically be the
clients actually receiving services
and/or enrolled in a program (e.g.,
the diabetics enrolled in a diabetes
clinic). In each case, if the program
does not collect exact data on the number
of people being served, then the number
should be estimated based upon the best
available data.
-
What are the
specific services or resources provided
to this population?
This question refers to the first type
of performance measures described above.
(“How much service or resources
is delivered?”). It should, therefore,
be answered in concrete, quantifiable
terms, indicating not only what are
the services or resources provided to
the “population served”,
but also how much of those services
are provided. For example, a financial
aid program for nursing students might
provide 100 students annually a total
of $200,000 in financial aid and 2 hours
per student of financial aid counseling.
Note that the description of services
or resources provided will often include
an indication of the number of clients
to whom the services or resources are
being provided.
(4)
If the population served is an estimated
number, then the basis for the estimate
should also be provided.
With the information provided by the answers
to these three questions, the selection
of Performance Review Measures or Goals
or Objectives for a HRSA funded program
or major activity can begin. In initial
discussions with the grantee and fellow
HRSA colleagues, the review team should
discuss these questions to assure that all
parties begin with the same basic assumptions.
Selecting Performance Review Measures:
Identifying Outcome and Effort Measures
Outcome Measures and
Effort Measures
Performance Review Measures
should, where such measures are available,
consist of a combination of outcome measures
and effort measures, for several reasons.
First and foremost, not all performance
measures are created equal. It is more important
to measure impact (outcome measures) than
it is to measure how well a service is delivered
(effort measures), and it is more important
to measure how well a service is delivered
(effort measures) than it is to measure
how much of a service is delivered. The
HRSA Performance Plan not only states that
outcomes are the most important performance
measures at HRSA, it also states that HRSA
will work to increase the use of such measures.
Second, even though these three kinds of
performance measures are not equally important,
each serves an essential role. As discussed
above, measuring “how much”
service or resources are provided is a necessary
first step in selecting effort measures
and outcome measures. Effort measures are
necessary to understand how well a program
is operating (for example, how efficiently
it is run), but these types of measures
do not provide data on the impact of the
program. Likewise, outcome measures will
provide data on impact, but not on how well
the program is run.
Finally, the HRSA Performance Plan notes
that, in implementing new programs, the
first data available often are only data
for measures of “how much” or
“how well” service or resources
are provided. The plan states that these
types of measures are “frequently
the most realistic and routinely measurable
indicators of performance … are often
the only indicators currently available
on an annual basis and reflect the level
of control an agency can bring to bear through
particular programs.” The data for
outcome measures remains essential; but
often such data is not available until after
the initial program start up is complete.
It should be noted, however, that data on
outcome measures are often most helpful
and easier to collect if they are established
at the outset of program design.
Identifying Performance Review Measures
for Category 1 Grant Programs
The answers to the first three questions
discussed above provide the essential parameters
for the identification of the selected Performance
Review Measures for a HRSA funded program
or major activity: goals and objectives,
the population served, and the services
or resources provided. As suggested earlier,
it is also important to discuss what the
grantee is trying to accomplish (especially
with respect to the impact on the population
served) and how the grantee knows if it
is making progress. In addition, when gathering
data and information during the pre-site
visit phase prior to the selection of performance
review measures, the review team must assess
the environment within which HRSA funded
programs operate as well as the financial
viability of the grantee.
With this information, the review team
in collaboration with the HRSA project officer
can, on a preliminary basis, identify Performance
Review Measures (both outcome measures and
effort measures) from the grantee’s
pool of available performance measures for
a program or major activity.
The review team in collaboration with the
HRSA project officer(s) and other appropriate
HRSA staff, contact the grantee to describe
the performance review process and select
the performance review measures. In selecting
the performance review measures, resources
that should be consulted include:
-
-
HRSA program performance
reporting requirements; and
-
The grantee’s
own measures and data, identified as
program (including business management)
goals and objectives in grant applications
and progress reports, and often connected
to a grantee’s strategic plans,
annual plans, and/or work plans.
-
Performance Review
Measures required by several HRSA Bureaus
and Offices
The next step in selecting Performance
Review Measures for a HRSA funded program
or major activity is to answer the following
question:
From the pool of measures, what
are the key outcome and effort measures?
It is essential in ensuring the usefulness
of the selected Performance Review Measures
to limit the total number of Performance
Review Measures used. This allows the performance
review to focus and develop a more in-depth
analysis and set of options to improve program
performance. Select no more than 5 Performance
Review Measures per HRSA funded
program or major activity (in total,
both effort measures and outcome measures)
from the pool of performance measures. Overall,
to ensure that Performance Reports are concise
and useful, the following limits on the
number of Performance Review Measures are
provided below:
#
of Grant Programs |
Max
# of Performance Measures* |
1 |
5 |
2 |
8 |
3 |
9 |
4 |
12 |
* For multiple funded grantees, please select
cross-cutting measures whenever possible.
Once the pool of Performance Review Measures
(outcome measures and effort measures) have
been identified for the HRSA funded program
or major activity in question, the next
step is to select the program or major activity’s
Performance Review Measures. To do so, the
pool of Performance Review Measures should
be rated using the following three criteria:
Communication Power
-
Are these measures of
outcomes and effort communicated easily?
Would those who pay attention to your
work (e.g., taxpayers, legislators, HRSA
project officers, clients) understand
what this measure means?
-
Do they measure the most
important outcomes and efforts? Is this
a measure of something of central importance
with respect to the program’s goals
and objectives (e.g., the health of a
baby at birth would be of central importance
to a prenatal clinic)? Is this measure
a good corollary for the many other measures
that might also be used (i.e., would the
trend represented by the data collected
for this measure be similar to the trend
represented by data collected for those
other measures)?
The process for applying these criteria
is outlined on the diagram below: “Criteria
for Selecting Performance Review Measures.”
The measures that have high ratings for
all three criteria are good choices for
Performance Review Measures for the HRSA
funded program or major activity. Of course,
sometimes the best available data in the
pool will not meet all three criteria. Following
the graphic is a worksheet for applying
the criteria, titled “Criteria for
Selecting Performance Review Measures Worksheet.”
Grantee
Performance Review Protocol:
Criteria for Selecting Performance Review
Measures
Directions:
List potential Performance Review Measures
for the HRSA funded program or major activity
in question and rate each as High, Medium,
or Low on each criterion: Communication
Power, Central Importance, and Data Power.
Criteria
for Selecting Performance Review Measures
Worksheet
-
List the Performance
Review Measures for the program or major
activity.
-
Rate each measure as
High, Medium
or Low for each of the
three criteria.
-
. Rank the measures to
select up to 5 Performance Review Measures:
the ones that rank high or highest on
all three criteria.
-
Note the measures that
rank high on communication and importance
but low on data. These may be candidates
for a “data development agenda”:
(i.e., measures we would use if the data
were available).
Suggested criteria for selecting
Performance Review Measures:
Communication: |
Would an average resident,
client or taxpayer understand what this
measure means? |
Central Importance: |
Does the measure say something of
central importance about the outcome/effort
the grantee is trying to measure? Is
it a good corollary for similar measures? |
Data: |
Does the grantee have quality data
available on a timely basis? |
Performance
Review Measures |
Criteria:
H, M, L |
Rank |
C |
CI |
D |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
In
summary, Performance Review Measures should
include a mix of outcome and effort measures,
whenever possible. They should each represent
a grantee’s efforts in a broader key area.
Up to 5 Performance Review Measures may be presented
for each HRSA funded program and/or major activity.
All of these measures should reflect the core
work of the grantee. Lastly, given the critical
importance of the financial viability of HRSA
funded programs, at least one financial measure
should be considered in the mix of Performance
Review Measures based on the pre-site visit
financial analysis of the program(s).
A set of national performance review
measures have been identified for use with
several HRSA programs. For a listing of these
programs as well as additional information
on the measures, please visit http://www.hrsa.gov/performancereview.
Identifying Goals or Objectives for
Category 2 Grant Programs
Similar to identifying Performance Measures
for Category 1 Grant Programs, the identification
of selected Goals or Objectives for the review
of Category 2 Grant Programs should consider
the communication power and the central importance
of the goal or objective, i.e., by reviewing
the grantee’s progress in attaining
the stated goal or objective the public, the
project officer, and stakeholders could generally
determine the grantee’s success in implementing
the approved grant program.
Graphing Performance Review Measures
Once the Performance Review Measures have
been selected, the next step is to assess
performance. The first step in assessing performance
is to graph the data for a Performance Review
Measure (5) .
In grant types where goals
and objectives will be used in lieu of performance
review measures, typically no data or very
limited data will be available. Therefore,
graphs will not be created.
Graphing Performance Review Measures
Once
the Performance Review Measures have been
selected, the next step is to assess performance.
The first step in assessing performance is
to graph the data for a Performance Review
Measure.
![Graphing Performance Review Measures](images/image042.gif)
Performance Review Measure
#1 Data Source: < >
The graph provides a concise,
visual representation of a grantee’s
performance on each selected Performance Review
Measure. Each graph should include units of
the measure on the vertical axis and units
of time on the horizontal axis. The graph
should also include a title and legend. When
available and appropriate as comparisons,
the graph may include the grantee’s
goal/target, HRSA standards and/or national
performance data. The graph should include
at least three actual data points. Each graph
should also provide the source of the Performance
Review Measure data.
Assessing Performance
Once the data are graphed for a Performance
Review Measure, performance may be assessed.
Assessing performance means answering the
following questions:
Performance Review Measures should be used
first and foremost to assess the improvement
of the grantee’s program against its
own history. To this end, the graph should
also include a trend line of the history of
the program’s performance and, to the
extent possible, a projection of future performance
(i.e., the projected trend assuming there
is no change in the grantee’s management
of the program). These projections of future
performance should be based on facts (i.e.,
past performance history) or informed opinions
based on facts (i.e., similar grantee performance
trends; etc.) – not just the grantee/review
team’s specified goal.
Trend lines may also be used to compare the
grantee’s performance with (1) a goal,
(2) a standard in the field, or (3) the performance
of other similar programs. These uses, however,
should be approached with caution. Goals are
often important for managing a program, but
are not inherently a reliable reflection of
a grantee’s performance potential; they
are sometimes set unrealistically high or
low based on reasons other than an objective
assessment of a grantee’s potential
performance. The reliability of standards
also varies. Standards for effort measures
are typically more well established and, therefore,
more reliable than standards for outcome measures.
Examples of standards for effort measures
include: staffing ratios; application processing
time; handicap accessibility; and waiting
time. Standards for outcome measures are almost
always more experimental, such as hospital
recovery rates and morbidity/mortality rates.
Similarly, comparisons to other programs
can lead to unfair and misleading conclusions
when such comparisons are cases of “apples
and oranges.” In the case of complex
service systems, for example, the variables
are usually so dynamic that it is hard to
imagine a scenario in which the comparison
of outcome measures between two such systems
would, alone, be a reliable indication of
program performance. Similar programs may
have significantly different clients or environmental
circumstances.
In the final stage of this component, the
performance review answers the question “Is
this acceptable?” The Performance Analysis
that follows will either focus on improving
performance if it is “not acceptable”
or sustaining performance if it is “acceptable.”
To the extent that a performance review can
reach such a judgment about the grantee’s
performance, the basis for that judgment,
such as a comparison to a goal or standard
should be stated clearly. By contrast, at
this point in the Protocol, it is not necessary
or even desirable to explain the underlying
reasons for a grantee’s performance,
which is the subject of the second component
of the Protocol: Performance Analysis.
Performance Analysis
In order to develop a plan to improve performance,
it is necessary to understand the factors
that impact performance (i.e., those factors
that contribute to and/or restrict performance
with respect to each Performance Review Measure),
answering the question:
In answering this question, the grantee,
in particular, has the opportunity to step
back from its day-to-day operations and consider
the larger issues it must deal with to improve
performance. This question also presents the
grantee with the opportunity to explain to
others (including funders) the nature of the
challenges it faces.
It should be noted that this analysis is
important whether or not a grantee’s
performance has met expectations. The analysis
will: (1) identify those contributing factors
that should be preserved to sustain performance;
(2) identify restricting factors that are
hindering performance; (3) identify factors
beyond the control of the grantee that might
have influenced performance (e.g., economic
trends); (4) identify those effective elements
of the grantee’s program that may be
useful to other grantees engaged in similar
work; and (5) determine whether performance
can still be improved (for example, if a goal
was, in retrospect, too low).
The worksheet below is a “force field
analysis,” which can be used to organize
the factors according to their contributing
and restricting influences on efforts to turn
or improve the curve of the performance trend.
Force Field Analysis Worksheet
In analyzing performance, the performance
review should consider the full range of factors
that might influence performance. The following
questions address different possible factors:
- How do the grantee’s internal
systems and processes contribute to or restrict
the grantee’s performance on each measure?
This question should include a review (when
applicable) of one or more of the four organizational
areas described in Appendix A of the Protocol:
a) Capacity and Systems;
b) Outreach and Consumer Feedback;
c) Business and Financial Management; and
d) Leadership and Staffing.
In addition, the review team should review
the relevant HRSA program expectations and
application guidance for additional insight
and inquiry points for the grantee, HRSA project
officer(s) and other appropriate HRSA staff
on the grantee’s internal systems and
processes.
- What external issues, including
HRSA policies and environmental, political,
demographic, and economic trends, are influencing
the grantee’s performance on each measure?
In this area, it is important to note that
external factors should not be immediately
dismissed because they are perceived to be
beyond the control of the grantee. A succinct
Performance Analysis that demonstrates the
significant impact of an external factor on
a grantee’s performance may well serve
as a catalyst for actions on the part of HRSA,
HRSA program policy, and/or the grantee.
- What are the grantee’s successes
and challenges in forming and sustaining partnerships
that support its performance on each measure?
The question of partnerships is critical
to achieving outcomes. It also leads to one
of the great temptations in working with outcome
measures, (i.e., to use lack of control as
an explanation for not accepting responsibility
for those client outcomes).
To some extent, this reaction is understandable.
A prenatal counseling program does not control
factors like drug dependence, domestic violence,
unemployment, or homelessness that might influence
whether or not a pregnant teenager who has
gone through the counseling program has a
healthy birth. Therefore, the question becomes:
is it fair, to have healthy births as a performance
measure for a prenatal counseling program?
![The Matter of Control](images/arrow.jpg)
Therein lies the paradox at
the heart of performance accountability: the
measures that matter the most (i.e., outcome
measures) are the ones over which a grantee
will have the least control. However, if complete
control was the litmus test for choosing performance
measures, there would be few, if any, outcome
measures. By contrast, it is because client
outcomes are so important that they are indispensable
to the performance review process. Furthermore,
most managers and front-line workers went
into their fields of work because they wanted
to make a difference in their clients lives,
which usually translates into outcomes.
Achieving such outcomes, however, often
inherently involves partnerships –
engaging and working with those who have
a potential role to play in achieving the
desired outcomes. When a program becomes
committed to client outcomes, it expands
its influence by partnering with those who
also have a stake in those client outcomes.
The identification of factors impacting
client outcomes points the way to the type
of partners who need to be engaged and accountable.
Therefore, a prenatal counseling program
might well partner with domestic violence
counselors, drug treatment programs, or
an employment training center to better
ensure that teenage moms have healthy births.
In addition to considering the
full range of factors influencing the trend
line for a performance measure, the performance
review should seek to prioritize those factors,
answering the question:
To identify those factors with the greatest
influence or “leverage” with
respect to performance, performance factors
should be analyzed from two perspectives.
First, it is important to not just identify
the most immediate and easily observed factors
impacting performance (i.e., the “proximate
causes”), but to engage in the kind
of rigorous analysis that will identify
the underlying or more systemic factors
(i.e., the “root causes”) that
impact the grantee’s performance.
One simple way to surface these root causes
is to use the question “Why?”
For example, if patient satisfaction is
poor, as measured by a patient satisfaction
survey, and the reason identified is that
staff members are not adequately trained,
ask “Why are staff members not adequately
trained?” If the answer is that there
is not enough time to train the staff, ask
“Why is there not enough time to train
staff.” Continue asking “Why?”
until you have identified the root causes.
Another successful method is to review on-site
the identified key factors and/or processes
(i.e., follow the patient tracking system;
billing procedures; etc.) to determine how
and whether, in fact, these factors/processes
lead to the selected Performance Review
Measure.
Second, once the root causes have been
identified for the trend line for each Performance
Review Measure, the performance review should
prioritize those root causes based on their
relative influence. The root causes that
have the biggest impact (or “leverage”)
on performance and, therefore, are the most
critical to turning or improving the curve
of the trend line, should be given the highest
priority.
One concern at this juncture in the Protocol
is that the Performance Analysis will almost
inevitably identify the highest leverage
performance factors as those that are also
truly beyond the control of the grantee
and its partners. For example, pervasive
poverty resulting in a lack of income may
be a root cause contributing to poor nutrition,
which in turn is a proximate cause for poor
birth outcomes. In this instance, even though
pervasive poverty has a very big impact
on birth outcomes, it may not be possible
for either the grantee or its partners to
have much impact on this economic condition.
The key at this juncture in the Protocol
is not to try to determine whether or not
solutions are available; rather, the performance
review should simply record in its entirety
their root cause analysis. In the example
given, the performance review would record
the links from pervasive poverty to income
to nutrition to birth outcomes. This information
will then, in turn, be used in the next
step in the Protocol: generating options
for improving program performance. It is
in this next step, which takes place in
the third component of the Protocol, discussed
below, that the performance review will
determine whether the grantee can only act
on a more proximate cause or, alternatively,
or in addition, whether there are any feasible
actions that can be taken with respect to
a root cause.
Using the example discussed above, if pervasive
poverty cannot be readily addressed, the
performance review might generate options
such as engaging partners to help improve
nutrition, such as the agency that administers
the Woman Infant and Children (WIC) feeding
program. On the other hand, or at the same
time, the performance review might identify
potential roles the grantee can play in
efforts to address the issue of pervasive
poverty systemically, such as participating
in a local economic development initiative.
While these examples of options to improve
program performance are provided here to
illustrate the importance of a rigorous
and thorough root cause analysis; they are
the subject of the next part of the Performance
Analysis: Performance Improvement
Options.
Key Area Discussion
During the course of the analysis of the
performance review measures or goals or objectives,
the review team, grantee, and project officer(s)
if available, may identify additional facts
regarding the implementation of the grant
program that are not directly aligned with
the measure but are worth discussing and recording.
The discussion may include broader internal
systems issues; external issues, including
HRSA policies; identification of cross-cutting
factors; as well as significant accomplishments
and best or promising practices. Including
the highlights of the discussion in the report
will broaden HRSA’s understanding of
the grantee and the grant program and will
enhance HRSA’s understanding of the
challenges and opportunities the grantee faces
in achieving its mission.
Program Requirements
The focus of performance reviews is to measure
program performance, analyze the factors behind
performance and develop options and actions
for performance improvement. In the course
of a review, however, a review team may identify
and review key compliance issues/program requirements.
If identified, these issues should be discussed
with the grantee and documented in the Performance
Report. When documenting such an issue, the
review team should refer to the current HRSA
program expectations and application guidance
for the applicable statutory and regulatory
Program Requirements, where appropriate. In
addition, during the pre-site conference calls
with the HRSA Project Officer(s) and other
appropriate HRSA staff, the review team should
ask if there are any concerns with respect
to Program Requirements that the review team
should review with the grantee during the
site visit.
Performance
Report
The third component of the Protocol concerns
the production of the Performance Report.
The Performance Report includes an Executive
Summary, a Table of Contents, Grantee Information,
HRSA Grant Program Information, a Grantee
Overview, Grantee Performance Information,
and Program Requirements. The report records
the contributing and restricting factors for
each of the Performance Measures or Goals
or Objectives and lists the Performance Improvement
Options identified collaboratively between
the review team and grantee. The development
of performance improvement options and a discussion
of actions the grantee can take to turn or
improve performance on the Performance Review
Measure (i.e., Performance Improvement Options)
is the final step of the onsite review. Once
the Performance Improvement Options have been
developed, the review team drafts a Performance
Report in which the Performance Improvement
Options are documented for the grantee. Based
on this on-site discussion or a subsequent
conference call between the grantee, the review
team and the project officer(s), if the project
officer(s) was not part of the on-site review
team, the grantee initiates the development
of the Action Plan, which is the subject of
the fourth and final component of the protocol.
Performance Improvement Options
While many options for actions to improve
performance may be developed, the selection
of Performance Improvement Options should
be limited to ensure that the most important
options are identified and addressed. On the
other hand, enough Performance Improvement
Options should be selected to present the
grantee with a variety of choices for action.
The following considerations are particularly
important in developing Performance Improvement
Options:
-
Does the Performance
Improvement Option correspond with one or
more of the root causes identified in the
Performance Analysis?
A key discipline in developing
Performance Improvement Options is the “alignment”
of those options with the root causes identified
in the Performance Analysis. This alignment
provides the rationale for selecting the
Performance Improvement Option. Without
such a rationale for the choice of a Performance
Improvement Option, there is no tie to improving
performance: the connection between the
“ends” (as measured by the Performance
Review Measure) and the “means”
is lost.
-
How strongly will the
Performance Improvement Option turn or improve
the performance on the Performance Review
Measure?
In addition to alignment
with the root cause(s) identified in the
Performance Analysis, the Performance Improvement
Options should also be assessed with respect
to the prioritization given to those root
causes. Given that resources are finite,
decisions with respect to the dedication
of resources (including staff time) to a
Performance Improvement Option must be based
on the expected impact of those resources
on performance. In other words, the Performance
Improvement Options presented should address
the most important root causes identified
in the Performance Analysis and, therefore,
be geared to having the greatest potential
impact on the performance trend for the
corresponding Performance Review Measure.
As noted in the discussion of the Performance
Analysis, this concept is sometimes referred
to as “leverage.”
-
Is the Performance
Improvement Option feasible?
Can it be done? This question
is the necessary counterpart to the question
of leverage. While the final determination
of feasibility must be made by the grantee
in selecting from among Performance Improvement
Options for its Action Plan, feasibility
should also be considered at least in broad
terms in developing the Performance Improvement
Options. Questions of feasibility, however,
should be handled so as not to limit creativity.
Sometimes the proposal of a potentially
infeasible option will be the catalyst in
the thinking process that leads to a highly
creative and feasible solution to a problem.
The importance of creativity is further
discussed in the context of “no-cost/low-cost
solutions,” below.
-
Are the Performance
Improvement Options evidence-based?
In weighing both leverage
and feasibility, research is indispensable.
Performance Improvement Options should be
based to the extent possible on research
or evidence-based practices (i.e., practices
that have been documented to work elsewhere).
There may, of course, be times that data
is limited and a program must move forward
with the best judgment of its experienced
professionals who are working to improve
performance. In most cases, however, the
review team should challenge the grantee
in its Performance Improvement Options to
implement only practices that are research
or evidence-based. Correspondingly, the
review team should document the relevant
source when noting the use of research-based
or evidence-based practices.
-
Have “no-cost/low-cost”
options been developed?
As noted above, creativity
is also a key to this work. Have all the
best options available been identified?
Often, however, such a discussion of what
can possibly be done will turn to and get
stuck on the need for additional funding.
Funding is almost always a critical need
and careful thought must be given to ways
to increase funding where needed. However,
the performance review should also ask the
grantee to explore “no-cost/low-cost”
options (i.e., options the grantee can pursue
with existing staff and/or resources) to
make sure the most potentially creative
options are identified and explored. What
assumptions does the grantee need to challenge
about the current way things are done? Has
the grantee considered ways it could work
differently with existing resources and
staff?
-
What partners may have
a role to play in improving program performance?
The critical role of partners
was explored in the discussion of Performance
Analysis, above. An important consideration
in developing Performance Improvement Options
is whether there are partners who may have
a role to play in improving program performance,
especially with respect to outcome measures.
If there are partners with a role to play,
then a Performance Improvement Option should
include identifying the partner and providing
a brief description of the contribution
they can make to performance improvement.
Action
Plan
Action Items
Finally, a grantee must decide specifically
what it will do to improve performance (i.e.,
the Action Items). This process which should
involve the grantee, the project officer(s),
and the review team, culminates in the Action
Plan, provides an opportunity for the grantee
to reflect upon the Performance Report, including
the Performance Improvement Options and Key
Area Discussions, and describe its Action
Items. The grantee is asked to describe the
specific action items to be completed for
each Performance Review Measure or Goal or
Objective and, when necessary, describe the
specific actions to be completed to address
any identified Program Requirement issues.
The grantee has the opportunity to request
technical assistance to support the implementation
of the Action Items.
The Action Plan builds on the work done in
developing the Performance Improvement Options.
The grantee should consider alignment with
program purpose and priorities, leverage,
and feasibility. With respect to leverage
and feasibility, the grantee will often need
to balance these two considerations in choosing
the actions it will implement. A Performance
Improvement Option that has high leverage
and high feasibility will, of course, be a
prime candidate for the Action Plan. The choice
among other Performance Improvement Options,
however, may involve some trade-offs between
leverage and feasibility and will need to
be weighed accordingly.
In addition, once the grantee has chosen
a Performance Improvement Option, the Action
Plan should provide greater specificity about
what the grantee is intending to do. In other
words, once the Performance Improvement Option
becomes an action item in the Action Plan,
the grantee should provide the details necessary
to indicate by whom, what, where, when, and
how the action will be implemented.
Action Plan Development
Action items should be a consolidation of
both short-term and long-term priorities that
will assist the grantee in achieving substantial
improvement in overall performance. The 3-5
action items address concerns discovered during
the performance review process that may relate
to the program goals, program objectives or
performance measures.
Action Item: The action
item is a specific activity of central importance
to the program that will improve the performance
of the grantee. State how the action item
is linked to a performance measure, performance
improvement option, grant goal or objective,
cross cutting key area or internal systems/process.
Action items should be implemented within
the lifecycle of the HRSA grant project period.
Focus on low-cost and no-cost action items
that can be accomplished within existing resources.
Expected Outcome: State
the quantitative or qualitative outcome anticipated
upon completion of the action item.
Key Steps with timeframes and parties
responsible:
-
Key steps: List the specific
steps the grantee plans to take to achieve
its goals and objectives. The steps should
be completed within the HRSA grant project
period.
-
Timeframes: State the timeframe
needed to complete the step.
- Parties responsible: State who will assure
that the action step is accomplished.
Action Plan Review: The
review team and project officer conduct a
joint review of the plan and provide feedback
to the grantee on the specificity, feasibility
and potential impact of the action items.
Status/progress/update:
Report on the progress of the action items;
including barriers encountered, needed revisions
and technical assistance updates.
Action Plan Follow-up
Upon submission of the Action Plan, the review
team and the project officer(s) review the
Action Items to ensure that they are aligned
with program purpose and priorities, that
areas of concern will be addressed, and that
specific outcomes are identified. The performance
review concludes with the adoption of an agreed
upon Action Plan to be implemented. The project
officer has the lead responsibility for monitoring
the action plan.
Please see the Action Plan/Technical
Assistance Request Template on page 35 for
further information on the recommended format.
Technical Assistance Guidance
Goal of Technical Assistance: HRSA
grantees participating in performance reviews
create an Action Plan to reflect and leverage
the discussions and findings of the review
and identify actionable items that will improve
the grantee’s performance. The grantee
may submit a technical assistance (TA) request
in support of their Action Plan.
Technical Assistance Proposal Development:
The TA request may address Program Goals,
Program Objectives or Performance Measures.
Types of TA requested include, but are not
limited to: billing and collections; educational
material development; medical records; financial
management; marketing; patient flow; and practice
management.
TA may be identified during the pre-site
and/or on-site. The TA request will be submitted
by the grantee to the OPR Review Team within
30 days of the on-site review along with the
Grantee Action Plan. TA requests will range
from 2 days to a maximum of 5 days.
Once the TA request has been submitted and
approved by the OPR Team and Project Officer
the OPR Team will forward it to the OPR Headquarters.
The Headquarters office will submit a consolidated
list of TA requests to the Bureaus/Offices
on a monthly basis for approval and a decision
regarding who will have the lead role of the
TA. If the Bureau has the lead role in the
TA process they will notify OPR when the TA
has been completed and provide a summary of
TA.
Other Periodic Reports
Review teams will submit items for the Environmental
Snapshot Report and the Policy and Trends
Report that document the key environmental
trends/issues (i.e., regional, State and/or
local) and/or policy issues (including HRSA
policy issues) impacting program implementation
and/or performance that were identified during
the performance review process, describing
their specific impact and providing recommendations
to HRSA for addressing these issues, as appropriate.
Summary
In summary, the review team will work with
grantees, HRSA project officer(s) and other
appropriate HRSA staff in the performance
review process to systematically and collaboratively
apply the Protocol. They will “begin
at the end” by selecting and assessing
progress on Performance Review Measures or
selected Goals and Objectives, analyzing performance
and discussing Key Areas regarding the implementation
of the grant, and generating Performance Improvement
Options. This is documented by the review
team in the Performance Report. Finally, the
grantee, in collaboration with the review
team and project officer(s) will develop Action
Items for an Action Plan to improve performance.
2008 OPR Matrix of HRSA Grant Programs
HRSA
Activity Code |
Bureau
or Office |
CATEGORY
1 Services and Training with Measures
(or required measurable goals) that have
data |
CATEGORY
2 Training, Infrastructure, other programs
not easily reviewed using Category 1 approach |
Program |
B04 |
MCHB |
X |
|
MCHB Title V Block Grant |
D04 |
ORHP |
|
X |
Rural Health Care Services Outreach
Grant Program |
D06 |
ORHP |
|
X |
Rural Health Network Development Program |
D11 |
BHPr |
|
X |
Nurse Education Practice and Retention |
D13 |
BHPr |
|
X |
Dental Public Health Residency Training
Grants |
D19 |
BHPr |
|
X |
Nursing Workforce Diversity |
D62 |
BHPr |
|
X |
Comprehensive Geriatric Education Program |
D66 |
BHPr |
|
X |
Nurse Education, Practice and Retention:
Enhancing Patient Care Delivery Systems |
D70 |
MCHB |
|
X |
State Implementation Grants for Integrated
Community Systems for CSHCN |
G22 |
OHIT |
|
X |
Telehealth Resource Center Grant Program |
G96 |
MCHB |
|
X |
Improving Understanding of Maternal
and Child Health and Health Care Issues |
G97 |
MCHB |
|
X |
Partnerships to Promote Maternal and
Child Health |
H12 |
HAB |
X |
|
Ryan White Part D: Women, Infants, Children,
Youth and Affected Family Members AIDS
Healthcare, service delivery |
H18 |
MCHB |
|
X |
State Systems Development Initiative |
H1C |
BPHC |
X |
|
Native Hawaiian Health
Care |
H1M |
HAB |
X |
|
Licensure Portability Grant
Program |
H21 |
MCHB |
X |
|
Traumatic Brain Injury |
H25 |
MCHB |
|
X |
Community Based Integrated
Service Systems |
H33 |
MCHB |
X |
|
EMSC Partnership Grants |
H3H |
ORHP |
|
X |
Small Rural Hospital Improvement
Program |
H49 |
MCHB |
X |
|
Healthy Start-Eliminating
Racial/Ethnic Disparities |
H4A |
HAB |
|
X |
National AIDS Education
and Training Centers |
H57 |
MCHB |
|
X |
Integrated Health and Behavioral
Health Care for Children And Adolescents |
H59 |
MCHB |
X |
|
Healthy Behaviors in Women |
H61 |
MCHB |
X |
|
Universal newborn hearing
screening |
H67 |
MCHB |
X |
|
Disparities in perinatal
health - border initiatives |
H76 |
HAB |
X |
|
Ryan White Part C Outpatient
EIS Program |
H80 |
BPHC |
X |
|
Health Center Cluster |
H87 |
MCHB |
X |
|
Thalassemia |
T02 |
MCHB |
|
X |
MCH Distrance Learning |
T09 |
BHPr |
|
X |
Grow Your Own FQHC Nurse
Traineeship |
T12 |
BHPr |
|
X |
Grants to States to Support
Oral Health Workforce Activities |
T17 |
MCHB |
X |
|
Leadership Training in
Pediatric Dentistry |
T72 |
MCHB |
X |
|
Pediatric Pulmonary Centers |
T73 |
MCHB |
X |
|
Leadership Education in
Neurodevelopmental and Related Disorders
Training Program |
T76 |
MCHB |
X |
|
Maternal and Child Health
Public Health Training Program |
U01 |
MCHB |
|
X |
Partnership for State Title
V MCH Leadership Community |
U1E |
MCHB |
|
X |
Sickle Cell Treatment Demonstration
Program |
U1M |
BCRS |
|
X |
Native Hawaiian Health
Scholarship Program |
U22 |
MCHB |
X |
|
Heritable Disorders |
U28 |
HAB |
|
X |
National Quality Improvement/Management
Technical Assistance Center Cooperative
Agreement |
U32 |
MCHB |
|
X |
National Newborn Screening
and Genetics Resource Center |
U33 |
MCHB |
|
X |
Consumer Initiatives for
Genetic Resources and Services |
U37 |
MCHB |
|
X |
Newborn Screening Informatics
Practice Network |
U39 |
MCHB |
|
X |
Transition Services in
Adolescent Health - Healthy and Ready
to Work |
U41 |
MCHB |
|
X |
Health Insurance and Financing/
CSHCN |
U42 |
MCHB |
|
X |
Integrated Community Service
Systems /CSHCN |
U44 |
MCHB |
|
X |
Oral Health Resource Center
& Partnership for Leadership in Oral
Health |
U45 |
MCHB |
|
X |
Adolescent Health/School-Based
Health |
U46 |
MCHB |
|
X |
3 Grants: Trainig Institute
for Training Consultants with Child Care |
U48 |
MCHB |
|
X |
Sudden Infant Death Syndrome |
U49 |
MCHB |
|
X |
Child and Adolescent Injury
and Violence Prevention Resource Centers
Cooperative Agreement |
U50 |
MCHB |
|
X |
Healthy Tomorrows Cooperative
Agreement |
U51 |
HAB |
|
X |
PEPFAR/Global AIDS Initiative:
Rapid Expansion of Antiretroviral Therapy
Program-Selected African & Caribbean
Countries |
U52 |
MCHB |
|
X |
Universal Newborn Hearing
Screening - Cooperative Agreement |
U53 |
MCHB |
|
X |
Cooperative Agreement -
Evidence Base for Systems of Care |
U56 |
ORHP |
|
X |
Rural Assistance Center
for ORHP Cooperative Agreement |
U58 |
BPHC |
|
X |
State and Regional Primary
Care Associations |
U59 |
MCHB |
|
X |
National Maternal and Child
Health Data Resource Center |
U68 |
BHPr |
|
X |
State Primary Care Offices
(PCOs) |
U69 |
HAB |
|
X |
National Training and Technical
Assistance |
U76 |
BHPr |
|
X |
Basic/Core Area Health
Education Centers |
U77 |
BHPr |
|
X |
Model State-Supported Area
Health Education Centers |
U91 |
HAB |
|
X |
Global AIDS Initiative:
International AIDS Education and Training
Center |
U92 |
HAB |
|
X |
Global AIDS Initiative:
Nursing Initiative |
U97 |
HAB |
|
X |
Global AIDS Initiative:
International Twinning Center |
X82 |
MCHB |
X |
|
Traumatic Brain Injury Protection and
Advocacy |
Part
I: Action Plan |
|
Action Item: |
Target Completion Date: |
Expected Outcome: |
|
Key
Steps |
Timeframe |
Responsible
Party |
Status |
Bureau |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Transition Date to Grant Application:
Additional Comments:
Technical
Assistance Request
Part II: Technical Assistance Request
Section I: To be Completed by Grantee:
Summary of TA Request:
Expected Outcome:
Estimated Number of Days:
Section II: To be Completed by OPR:
Region:
Team Lead:
Grant #:
Bureau/Office Funded:
Project Officer:
OPR Review Team Concurrence with Request:
OPR Comments:
Project Officer Comments:
Recommendation Bureau/Office to Lead TA:
DEPARTMENT
OF HEALTH & HUMAN SERVICES
HEALTH RESOURCES AND SERVICES ADMINISTRATION
Office
of Performance Review (1)
<City> (2) Regional Division
<Street Address, Room/Suite Number>(3)
<City, State Zip>
Performance
Report
for
<Grantee
Name> (4)
<Grantee City, State>
EIN # - <###-###-###>
Date
of Site Visit - <Month/Day/Year>(5)
<Name>
(6), Review Team Leader
<Review Team Member Name>
<Review Team Member Name>
<Review Team Member Name>
<Name>, Review Team Consultant
<Name>, Observer
<Month/Day/Year>
(7)
Executive
Summary
(8)
TABLE
OF CONTENTS (9)
I.
Grantee Information
II.
HRSA Grant Program Information
III.
Grantee Overview
IV.
Grantee Performance
A. Performance Measure or Goal or Objective
#1
B. Performance Measure or Goal or Objective
#2
C. Additional Performance Measures or Goals
or Objectives
V. Program Requirements
I.
Grantee Information (10)
A.
Name
B. Address: < >
C. Phone Number: <(###) ###-####>
D. Fax Number: <(###) ###-####>
E. E-mail Address: < >
F. Executive Director: <Name>
G. Grantee Summary: < >
II. HRSA Grant Program Information (11)
A.
HRSA Grant Program #1: <Name of HRSA Grant
Program>
1. Grant Number: <#>
2. Project Period: <Month/Day/Year - Month/Day/Year>
3. Budget Period: <Month/Day/Year - Month/Day/Year>
4. Amount of HRSA Funding as of <Site Visit
Date>: <$ >
5. Program Director: <Name>
6. HRSA Project Officer: <Name>
Phone Number: <(###) ###-####>
E-mail Address: < >
7. HRSA Grants Management Specialist: <Name>
Phone Number: <(###) ###-####>
E-mail Address: < >
III. Grantee Overview (12)
A. Background on grantee organization
B. Description of HRSA grants received (number
and value by Bureau/Office)
C. Population Served
D. Services/resources provided by the HRSA
funded program
IV.
Grantee Performance (13)
(for Category 1 Grant Programs)
A. Performance Review Measure (14)<
>
1. HRSA Grant Program(s): (15)<
>
2. Population Served: (16)<
>
3. Services Provided: (17)<
>
![Performance Review Measures](images/graph2.png)
Data
Source:(19)< >
4.
Performance Trend: (20)
<graph and performance trend only if using
a Performance Measure>
5. Performance Analysis (21)
a. Contributing Factors (CF)
CF1. < >
CF2. < >
b. Restricting Factors (RF)
RF1. < >
RF2. < >
6. Key Area Discussion/Promising Practices
(22)
7. Performance Improvement Options (23)
a. (CF#, RF#)
b. (CF#, RF#)
B. (Repeat format used in “A”
for each additional performance measure)
OR
(for Category 2 Grant Programs)
A.
Program Goal or Objective (24)
< >
1. HRSA Grant Program(s): (25)<
>
2. Population Served: (26)
< >
3. Services Provided: (27)<
>
4. Performance Analysis (28)
a. Contributing Factors (CF)
CF1. < >
CF2. < >
b. Restricting Factors (RF)
RF1. < >
RF2. < >
5. Key Area Discussion/Promising Practices
(29)
6. Performance Improvement Options (30)
a. (CF#, RF#)
b. (CF#, RF#)
B. (Repeat format used in “A”
for each additional goal or objective)
V.
Program Requirements (31)
A.
Program Requirement #1:(32)
< >
1. HRSA Grant Program(s): < >
2. Description of Issue: < >
Foot
Notes
(1) Use the following template to format your
report, including the following general report
parameters: use 12 pt Times New Roman black
font with standard margins; mark “DRAFT”
on the draft Performance Report; spell out acronyms
in the first instance when they appear in the
report; and when using the automatic date function
in MS Word, make sure it is a static date, rather
than a date that changes every time the document
is opened.
(2)
Insert the name of the city in which your regional
division is located.
(3)
Insert your regional division’s address.
(4)
Insert the grantee’s name, the city and
State the grantee is located in, and the CRS-EIN.
The CRS-EIN should contain twelve characters
and can be found on the notice of grant award
and in the grants database in eRoom. Also, please
insert the grantee’s name in the footer
of the Performance Report.
(5)
Insert the site visit date.
(6)
Insert the names of the review team lead, review
team members, observers and/or consultants,
as appropriate.
(7)
Insert the date the Performance Report was completed.
For the draft Performance Report, this should
be the date is sent to the grantee for comments.
For the final Performance Report, the date should
be the date the final report is sent to the
grantee.
(8)
The Executive Summary should be a brief overview
of the report designed to give readers a quick
preview of its contents. It should describe
the organization (Report Section I), contain
the HRSA grants received and reviewed (Report
Section II), provide key findings of the review
(Report Section IV and V), and special accomplishments
(Report Section IV). Summarize key information
from Section IV. Summarize any special accomplishments
or promising practices noted in Section IV.
(9)
The Table of Contents should be completed after
the report is completed and should list at a
minimum each of the major (Roman Numeral) sections
and Performance Measure or Program Goal or Objective
(10)
Insert the grantee’s name, address, phone
and fax numbers, e-mail address, and the name
of the grantee’s Executive Director. .
(11)
For each of the grantee’s HRSA funded
programs, provide the requested grant/program
information:
-
In the outline, please state the HRSA grant
program name next to the heading “HRSA
Grant Program #x:”. Each of the grantee’s
HRSA grant programs should be listed as
a separate letter in the outline (i.e.,
A. HRSA Grant Program #1: < >; B.
HRSA Grant Program #2: < >; etc.).
-
For each HRSA grant program, list the grant
number, the project period, the budget period
and the amount of HRSA funding as of the site
visit date. All of this information is located
in the grants database in e-room; however
the amount of HRSA funding needs to be verified
a week before the site visit by entering the
Electronic Handbook and looking at item 12A
in the most recent Notice of Grant Award (NGA).
If there are several awards for a given fiscal
year for an individual program, please make
sure to include all the award amounts when
computing the amount of HRSA funding.
- Insert
the name of the program director as well as
the name and contact information for the project
officer and grants management specialist.
(12) The Grantee Overview should provide the
reader with a sense of the history of the
grantee, including the type of organization
(i.e., university; hospital; not for profit
organization; public entity, etc.) as well
as a brief description of its HRSA funded
program(s), including the services provided
by these programs and the populations receiving
these services. It may also include State
and community health indicators/environmental
information to give the reader a better sense
of the needs of the community the grantee
serves or the environment the grantee operates
in. Additional background information may
include a description of the area served and
the percentage of the grantee’s budget
that comes from HRSA; this information will
provide the reader with a better understanding
of the grantee and the HRSA contribution/investment
in the organization.
(13) Grantee performance for Category 1 grants
is reviewed through the use of Performance
Measures. Grantee performance for Category
2 grants is reviewed through the use of Program
Goals and Objectives. Follow the appropriate
outline provided.
Up to 5 Performance Review Measures or Program
Goals and Objectives may be presented for
each HRSA funded program and/or major activity.
As a general rule, Performance Review Measures
or Program Goals and Objectives should focus
on the core HRSA funded program activities
of the grantee (i.e., program activities where
the majority of a program’s resources,
time, effort and/or staff are allocated.)
In the case of very complex grantee programs,
however, it may be necessary to examine several
major activities within the core program.
Generally, these choices will be dictated
by the size and complexity of the HRSA funded
program and the nature of the HRSA funding.
For example, if the HRSA funded program is
large and complex and the HRSA funding is
applied broadly across the program, such as
a community health center (CHC), it will usually
be necessary to measure performance for the
major activities or subunits within the CHC,
as well as for the entire CHC. Typically,
where activities or subunits are measured,
the performance review should focus on those
subunits or activities that are most significant
(i.e., where the most resources and staff
are allocated) within the scope of the larger
program.
A set of national
performance review measures have been
identified for use with several HRSA programs.
.
Overall, to ensure that Performance Reports
are concise and useful, the following page
limits are outlined below. Please use the
chart below to determine the maximum number
of Performance Review Measures or Program
Goals and Objectives a report should include
and the maximum report length (including graphs):
Number
of HRSA Grant Programs/Major Activities |
Maximum
Number of Performance Review Measures
or Program Goals and/or Objectives* |
Performance
Report – Max Number of Pages** |
1 |
5 |
15 |
2 |
8 |
24 |
3 |
9 |
27 |
4 |
12 |
36 |
* For multiple funded grantees, please select
cross-cutting measures or program goals or
objectives whenever possible.
** Does not include Executive Summary, cover
sheet, table of contents, or Grantee Contact/HRSA
Grant Program Information.
(14) For each Performance Review Measure (Category
1), state the measure next to the appropriate
heading. Each Performance Review Measure should
be listed as a separate letter in the outline.
(15) For each Performance Review Measure,
list the HRSA grant program(s) to which it
relates.
(16) For each Performance Review Measure,
describe the population served to which it
relates. This description should include number
of individuals served by the activity/service
that relates to the Performance Review Measure
being analyzed. In cases where the Performance
Review Measure cuts across multiple HRSA grant
programs, the populations served should reflect
the populations served by all of the programs.
(17) For each Performance Review Measure,
describe the services provided. This description
should include the number and type of the
services and/or resources provided relating
to the performance measure. In cases where
the Performance Review Measure cuts across
multiple HRSA grant programs, the services/resources
provided should reflect the services/resources
provided by all of the programs.
(18) Use of graphs applies primarily to Category
1 grants and the use of Performance Measures.
Use of Program Goals or Objectives usually
do not lend themselves to graphing. Where
graphs are used they must be legible. Please
label the axes and include a title (i.e.,
Performance Review Measure and/or Program
Goal or Objective) and legend. The axes should
reflect the full scale, so the performance
trend is correctly depicted. Graphs should
be done in black and white and use different
line formats to distinguish between multiple
lines on a graph. Include a projected trend
when possible (see endnote 18 below). When
available and appropriate as comparisons,
the graph may also include the grantee’s
forecast for the current year, goal/target,
HRSA standards and/or national performance
data. The graph should include more than one
actual data point. If there are not at least
two data points for a measure, please consider
for the Data Development Recommendations section
below.
(19) Please state the source of the data used
for the Performance Review Measure. Also,
as appropriate, please provide a brief explanation
of the reliability of the data. If a graph
includes percentages, define the numerator
and denominator in this section.
(20) To the extent possible, comment on the
performance trend and provide the basis for
comparison (e.g., national standards; grantee’s
historical performance; another grantee’s
performance; HRSA targets; etc.) Also, please
include the review team’s professional
judgment on whether the grantee’s current
performance will continue at the same rate
or improve or decline. These projections of
future performance should be based on facts
(i.e., past performance history) or informed
opinions based on facts (i.e., similar grantee
performance trends; etc.) – not just
the grantee/review team’s specified
goal. This item will be used only if a grantee’s
performance has been graphed.
(21) Please list and describe the primary
factors (2-4), in priority order that contribute
to and restrict the grantee’s performance
on the Performance Review Measure. Each Contributing/Restricting
factor should align with the Performance Review
Measure. For each Contributing Factor, state
the factor next to the heading “CF#.”.
Each Contributing Factor should be listed
as a separate letter in the outline (i.e.,
CF1. < >; CF2. < >; etc.). Please
follow the same format for each Restricting
Factor (i.e., RF1. < >; RF2. < >;
etc.)
(22) Include highlights of discussion with
grantee regarding the grantee’s internal
systems and processes, external issues (including
HRSA policies and environmental, political,
demographic, and economic trends) that are
not captured in the contributing and restricting
factors but impact on the grantee’s
success in achieving its goals and objectives
and sustaining successful partnerships. Also
describe significant accomplishments and promising
practices.
(23) List and describe, in priority order,
2-3 Performance Improvement Options for the
Performance Review Measure. Each Performance
Improvement Option should align with the performance
factors and the Performance Review Measure.
For each Performance Improvement Option, please
identify at the end of the option in parentheses
[e.g., (CF1; CF4; RF2; RF5)] all the Contributing
and/or Restricting Factors that are being
addressed. Please include evidence based,
low/no cost options, potential partnerships
(be specific) as well as technical assistance
needs, as appropriate.
(24) For each Program Goal or Objective (Category
2), state the goal or objective next to the
appropriate heading. Each Goal or Objective
should be listed as a separate letter in the
outline.
(25) For each Program Goal or Objective, list
the HRSA grant program(s) to which it relates.
(26) For each Program Goal or Objective, describe
the population served to which it relates.
This description should include number of
individuals served by the activity/service
that relates to the Goal or Objective being
analyzed. In cases where the Goal or Objective
cuts across multiple HRSA grant programs,
the populations served should reflect the
populations served by all of the programs.
(27) For each Program Goal or Objective, describe
the services provided. This description should
include the number and type of the services
and/or resources provided. In cases where
the Program Goal or Objective cuts across
multiple HRSA grant programs, the services/resources
provided should reflect the services/resources
provided by all of the programs.
(28)Please list and describe the primary factors
(2-4), in priority order that contribute to
and restrict the grantee’s performance
on the Goal or Objective. Each Contributing/Restricting
factor should align with the Goal and Objective.
For each Contributing Factor, state the factor
next to the heading “CF#.”. Each
Contributing Factor should be listed as a
separate letter in the outline (i.e., CF1.
< >; CF2. < >; etc.). Please follow
the same format for each Restricting Factor
(i.e., RF1. < >; RF2. < >; etc.)
(29) Include highlights of discussion with
grantee regarding the grantee’s internal
systems and processes, external issues (including
HRSA policies and environmental, political,
demographic, and economic trends) that are
not captured in the contributing and restricting
factors but impact on the grantee’s
success in achieving its goals and objectives
and sustaining successful partnerships. Also
describe significant accomplishments and promising
practices.
(30) List and describe, in priority order,
2-3 Performance Improvement Options for the
Program Goal or Objective. Each Performance
Improvement Option should align with the performance
factors and the goal or objective. For each
Performance Improvement Option, please identify
at the end of the option in parentheses [e.g.,
(CF1; CF4; RF2; RF5)] all the Contributing
and/or Restricting Factors that are being
addressed. Please include evidence based,
low/no cost options, potential partnerships
(be specific) as well as technical assistance
needs, as appropriate.
(31) A Program Requirement is presented where
a grantee is not in compliance with a requirement
of applicable program statutes, regulations
and/or grant award conditions.
(32) For each Program Requirement, please
state the requirement next to the heading
“Program Requirement #x:”. Each
Program Requirement should be listed as a
separate letter in the outline (i.e., A. Program
Requirement #1: < >; B. Program Requirement
#2: < >; etc.). For each Program Requirement,
please specify the HRSA grant program(s) that
the requirement relates to and provide a description
of the issue(s) that led to the grantee’s
non-compliance. [If there are no Program
Requirement issues identified, please delete
this section of the Performance Report (i.e.,
the Performance Report would include no reference
to a section IV. Program Requirement).]
![](https://webarchive.library.unt.edu/eot2008/20080916005229im_/http://hrsa.gov/images/submain/1x1space.gif) |