Chapter 1. Introduction
The Agency for
Healthcare Research and Quality (AHRQ) has, since its inception, funded
research and other activities concerning a variety of children's health issues.
At the same time, the Agency's strategic
priorities continue to evolve in the context of a changing policy environment. Currently, the Agency is crafting strategic
plans for its ten "portfolios of work." As
part of the planning process, the Agency is evaluating the various
cross-cutting, or cross-portfolio, priority populations, including
children. To aid in this assessment, the
RAND Corporation was contracted to evaluate AHRQ's child health
activities.
As requested by
AHRQ, this evaluation addresses four Primary Objectives:
- Measure and assess to what extent the Agency contributed and
disseminated and/or translated new knowledge.
- Measure and assess to what extent AHRQ's children's healthcare
activities improved clinical practice and health care outcomes and influenced
heath care policies.
- Measure and assess AHRQ's financial and staff support for
children's health research.
- Measure and assess to what extent the Agency succeeded in
involving children's health care stakeholders and/or creating partnerships to
fund and disseminate key child health activities.
To address the
RFTO's primary purposes, RAND developed and implemented a conceptual framework
based on state-of-the-art research evaluation theories (Roessner, 2002).1 This framework allowed an assessment of the
Agency's contribution to the development and dissemination of new knowledge and
the impact of that knowledge on policies, clinical practice, and health
outcomes. In addition, we used tools and
methods developed at
RAND to describe the
Agency's funding focus and financial investment in children's health over
time. The report will be helpful in
determining the extent to which Government Performance Results Act (GPRA) goals
were achieved and in fulfilling GPRA/OPART requirements.
Return to Contents
Policy Context
AHRQ's
mission, strategic goals, and organization
Established in
1989 as the Agency for Health Care Policy and Research and reauthorized in 1999
as the Agency for Healthcare Research and Quality, AHRQ has a mission of
improving the quality, safety, efficiency, and effectiveness of health care for
all Americans. AHRQ regards its customers
as clinical, healthcare system, and public policy decision makers. Its strategic plan, adopted in 1998, includes
the following goals to reflect the needs of its customers:
- Support improvements in health outcomes.
- Strengthen quality measurement and improvement.
- Identify strategies that improve access, foster
appropriate use, and reduce unnecessary expenditures.2
AHRQ's priorities
and organization have changed over time. Currently its research and other activities are organized into ten
"portfolios of work" such as health care management, data development, and
health information technology. Cutting
across these portfolios, AHRQ has also identified, as per the Agency's
authorizing language, priority populations, including minorities, women, and
children.
AHRQ's children's
health research program
There are
systematic differences between children and adults that strongly suggest the
need for child-focused models for health services. The differences have been summarized in an
analysis supported by AHRQ (Forrest, Simpson et al.
1997)
as the four D's: development, dependence, differential epidemiology, and
demographics. A fifth D, disparities,
has also been proposed (Gidwani, Sobo et al.
2003). Children develop at a rapid pace and their
development in one domain impacts development in others. Children are dependent on families, schools,
and other social and political institutions for their welfare in general and
for access to health care. Most children
are healthy, and in contrast to adult epidemiology in which a few
high-prevalence conditions account for the majority of morbidity and mortality,
pediatric epidemiology is characterized by a relatively large number of
low-prevalence conditions. Demographically, U.S.
children as a group are
disproportionately poor, of color, and uninsured; as a result, health
disparities are a major concern. Because
of the differences between children and adults, and the frequent focus of health
services and clinical research on adults, the knowledge base for children's
health is considerably less robust than that for adults.
Another important
point of consideration is that linking "downstream" health outcomes
to preventive and other child health services is much more difficult than it is
for adult health services—because children are inherently more resilient and
healthy and the lag times are longer and intervening confounders are more
influential, among other reasons. As a
result, a major challenge for children's health services research is the need
to establish unequivocally the value of health care for children as an
effective use of resources.
In this context,
AHRQ's vision for children's health is "to establish the Agency for Healthcare
Research and Quality as one of the Nation's key public organizations for
improving the quality and effectiveness of health care services to the Nation's
children and adolescents." This mission is consistent with the Agency's overall
mission: "to improve the outcomes and quality of health care, reduce its costs,
address patient safety, and broaden access to effective services, through the
establishment of a broad base of scientific research and through the promotion
of improvements in clinical and health system practices, including the
prevention of diseases and other health conditions."3 Policy advice and coordination of AHRQ child
health activities are provided by a Children's Health Advisory Group (CHAG),
which is composed of representatives of AHRQ offices and centers.
Supporting this
mission are six specific objectives for children's health research:
- Contribute to new knowledge about child health services.
- Create tools and nourish talent to strengthen the
knowledge base in child health services.
- Translate new knowledge into practice.
- Improve communication with stakeholders in child health.
- Include children and child health care in all
AHRQ-supported research, as scientifically and ethically appropriate.
- Balance the ARHQ research portfolio to represent a
broader range of children and child health care.
The first three
areas follow the Agency's general research mission. The second three are specific to child health
and cut across the research cycle and the Agency's three strategic goals4 The rest of the strategic plan outlines these
goals and their related objectives in greater detail. The strategic plan ends with an overview of
partnerships, a significant element for achieving AHRQ's goals in child health
(AHRQ Strategic Plan, 1999).
In addition to
funding extramural and intramural research, AHRQ children's health activities
include the development of child-specific databases (e.g. the HCUP Kids
Inpatient Database), tools for measuring quality, evidence reports, liaison
with and contributions to other entities both within and beyond DHHS, a
list-serve, and the child health scholar program.
Context
An evaluation of
AHRQ's children's health activities must be placed within the appropriate
context. In this vein, we note that AHRQ
was originally conceived by Congress as a mechanism to reduce the Medicare
budget—so child health has been an evolving adjunct to the Agency's basic
raison d'etre. Far fewer federal
resources are available for children's health care, and consequently there are fewer
efforts regarding quality, safety, efficiency, and effectiveness of child
health care. When it comes to quality
and safety, the focus in HHS has been on improving quality and increasing
efficiency for the Medicare population, and on HIT as a mechanism for quality
improvement. And Congress has
increasingly prescribed specific funding initiatives for the Agency, such as
HIT and patient safety—areas in which there are more adult- and
elderly-focused researchers working versus child-focused researchers.
Program evaluation
and performance review
The AHRQ approach
to funding research, development, and demonstration projects is based on the
concept of "the research pipeline" in which funded activities collectively
build the infrastructure, tools, and knowledge base necessary for practice
improvements. This pipeline brings
together a mix of projects covering important aspects of children's health
research and is fueled through a cycle of iterative research involving the
steps of needs assessment, knowledge generation, dissemination of knowledge to
practitioners, evaluation of field experience and outcomes from the new
techniques, and feedback to start a new cycle (AHRQ, 2002).
AHRQ includes
measures of its performance in improving child health care as part of its own
performance measurement strategy. In
2003, the Agency revised its Government Performance Results Act (GPRA)
framework for research on health care costs, quality, and outcomes to address
three strategic goals: 1) To have
measurable improvements in the quality, safety, and outcomes of healthcare for
Americans; 2) To develop the evidence base for policymakers and health systems
to use in making decisions about what services to pay for, how to structure
these services, and how these services are accessed; and 3) To build capacity
for improving the quality of health care delivery through research and
training. Indicators of AHRQ's
performance in meeting the first goal include a 5 percent reduction in the rate
of hospitalizations for pediatric asthma in persons under age 18 and a 5
percent reduction in the number of premature babies who develop Respiratory
Distress Syndrome.5
In 2002, the
Office of Management and Budget developed the Program Assessment Rating Tool
(PART) to implement the GPRA process. This 30-item questionnaire is to be used to assess the quality of every
federal program's design, goals, management and results to determine
effectiveness. Over a five-year period,
OMB expects to assess every federal program to ensure each is establishing an
aggressive action plan for improvement. The PART evaluation covers four critical areas of assessment—purpose and
design, strategic planning, management, and results and accountability. The first set of questions gauges whether the
program's design and purpose are clear and defensible. The second section involves strategic
planning, and weighs whether the agency sets valid annual and long-term goals
for programs. The third section rates
agency management of programs, including financial oversight and program
improvement efforts. The fourth set of
questions focuses on results that programs can report with accuracy and
consistency.6 Although not designed to answer these
questions directly, this evaluation should provide information useful to the
Agency in preparing future PART assessments.
Return to Contents
Evaluation
Framework
"What we really want to get at is not how many reports
have been done, but how many people's lives are being bettered by what has been
accomplished. In other words, is it being used, is it being followed, is it
actually being given to patients?... [W]hat effect is it having on people?"
Congressman John Porter, 1998, Chairman, House Appropriations
Subcommittee on Labor, HHS, and Education
The primary
objectives of the evaluation, as specified in the RFTO were to:
- Measure and assess to what extent the Agency
contributed new knowledge as a result of its funding of children's health
research (extramural and intramural) and disseminated and/or translated
effectively its findings to meet AHRQ's strategic objectives of improving the
safety, quality, effectiveness, and efficiency of health care as well as wider
DHHS strategic objectives.
- Measure and assess to what extent AHRQ's children's
health care activities, i.e., its research findings, meetings, conference
support, products, tools, etc., improved clinical practice and health care
outcomes and influenced heath care policies over the past fifteen years.
- Measure and assess AHRQ's financial and staff support
for children's health research as well as Agency internal handling of children's
health grants, contracts and intramural activities research with/among other
AHRQ programs, portfolios and activities and other DHHS and federal agency
efforts.
- Measure and assess to what extent the Agency succeeded
in involving children's health care stakeholders and/or creating partnerships
to fund and disseminate key child health activities.
Evaluating the impact of research programs is
challenging.7 Research programs are characterized by high
levels of uncertainty, long time horizons, and indirect and complex causal
paths between inputs and impact. Further, the research funder plays only one of the many roles (e.g.
knowledge developer, users, other stakeholders) in the collective that produces
knowledge and improvement. In order to
measure and assess the success of a research program, one must ask what program
performance means under these circumstances, and how it might be measured. As 'the Porter Question' above, indicates, a
key measure of the impact of a research program is the "effect [it is] having
on people."
Numerous conceptual models for evaluating research
have been developed. RAND
drew from these models, as well as from our experience with relevant research,
demonstrations, and evaluations, to develop a conceptual framework to structure
our design for the proposed evaluation.
One way to evaluate a research portfolio is by
describing the mix of investments over time, in terms of grants funded, and the
scientific output of this investment in the form of articles in the peer-reviewed
literature. This approach yields an
overview of the research portfolio in terms of inputs and outputs, and is
useful for evaluating the degree to which a research portfolio matches
strategic plans. Accordingly, we examined
research activities and publications over time and according to AHRQ strategic
plans.
A more detailed analysis, however, is also
possible. That is, research findings (in
the form of articles in the peer-reviewed literature) can be categorized
according to their potential impact in various spheres. An example of such a categorization scheme is
that developed by AHRQ staff to assess the impact of AHRQ's funding for
outcomes and effectiveness research (Stryer, Tunis et al.
2000). According to this framework, publications
derived from AHRQ-sponsored activities may have four possible types of impact:
those that are contributions to the health care knowledge base that result from
foundational studies to identify problems, generate hypotheses, establish the
effectiveness of interventions and develop new tools to explore these problems;
those in which a policy or program was created or modified as a result of the
finding; those in which patterns of care or clinician or patient behavior
changed as a result of a finding; or those that improve health care
outcomes. With this in mind, our
approach includes an analysis of the potential impacts of publications derived
from AHRQ-sponsored activities.
A second way of evaluating a research portfolio is
based on the idea that the distribution of the impact of research activities
(or in this case, ARHQ-sponsored child health activities) is highly skewed. That is, only a small fraction of child
health activities produce important research findings or result in demonstrable
changes in policies, clinical practice, or health care outcomes. On the assumption that an examination
of such key child health activities can generate rich detail about the
processes by which activities contribute to the realization of a variety of
valued outcomes and impacts, we have identified a small subset of high-impact activities (individual
activities or groups of related activities) and examined them in-depth. Thus, the second component of our conceptual
framework is to develop in-depth case studies of high-impact activities to
describe the factors most important in bringing about these impacts.
Finally, given that a wide range of actors—including funders, scientists, policymakers, clinicians, and families—are
involved in translating research findings into practice, it is imperative to
consider the points of view of a variety of relevant stakeholders when
assessing the impact of AHRQ's children's health care activities on policies,
clinical practice, or health care outcomes. Additionally, it is important to understand the ways that AHRQ partners
with other stakeholders. Thus, the third
main component of our conceptual framework is to use key stakeholder interviews
to assess the impact of AHRQ's research products and other children's health
activities from the perspective of the users of these products and other
interested parties.
Given this
conceptual framework and the primary purposes of the evaluation as specified by
the RFTO, we approached the evaluation in the following manner. To address evaluation objectives 1 and 3, we
used existing documentation to describe the full range of children's health
research and activities that the Agency funded, and bibliometric techniques to
describe the development of new knowledge and its dissemination through
publication in peer-reviewed journals. We supplemented this with key stakeholder interviews of AHRQ staff and
others. To address evaluation objective
2, we used an approach similar to Stryer and colleagues (2000) to categorize
the products of funded research projects (articles in peer-reviewed journals)
according to the four levels of impact. We also went beyond these data, identifying high-value products (through
final grant reports, review of documents from the Impact Case Studies Program,
and key informant interviews) and studied these cases in-depth by interviewing
the PI, Agency staff, and other stakeholders such as policy makers, clinicians,
or patients. High-value products identified
in conjunction with the Task Order Officer (TOO) and the Senior Advisor on
Child Health represent activities or groups of activities that were judged by a
variety of stakeholders to be 'successful.' We constructed these interviews to enable us
to determine how the research findings led to improvements and what
role(s) AHRQ played in this process. To
address evaluation objective 4, we used key stakeholder interviews to gather a
variety of perspectives on AHRQ's efforts in partnering with others around
children's health.
Return to Contents
Structure
of this Report
In Chapter 2, we
describe the methods used to develop the database of activities and
publications as well as the case studies and key informant interviews. In Chapter 3, we present the results of our
analyses, organized by primary objective. In Chapter 4, we conclude with a summary of findings and
discussion.
1. Although
we drew from theories about evaluating research, we recognize that the current
evaluation concerns other AHRQ activities as well as research.
2. http://www.ahrq.gov/about/stratpln.htm
3. http://www.ahrq.gov/about/cj2003/cjweb03.htm#mission
4. We note
that the Agency's strategic plan, both overall and for children, has not been
updated to reflect its new mission statement.
5. http://www.ahrq.gov/about/gpra2004/gpra2004.pdf
6. http://www.gpoaccess.gov/usbudget/fy04/pdf/budget/performance.pdf
7. We note
that while our charge was to examine child health activities broadly speaking,
AHRQ is primarily a research organization and, thus, the framework of
evaluating research programs applies here.
Return to Contents
Proceed to Next Section