Executive Summary
"What we really want to get at is not how many reports
have been done, but how many people's lives are being bettered by what has been
accomplished. In other words, is it being used, is it being followed, is it
actually being given to patients?... [W]hat effect is it having on people?"
Congressman John Porter, 1998, Chairman, House Appropriations
Subcommittee on Labor, HHS, and Education
Since its
inception, the U.S. Agency for Healthcare Research and Quality (AHRQ, or the
Agency) has funded research and other activities concerning a variety of
children's health issues. At the same
time, the Agency's strategic priorities continue to evolve in the context of a
changing policy environment. Currently,
the Agency is crafting strategic plans for its ten "portfolios of work." As
part of the planning process, the Agency is evaluating the various
cross-cutting, or cross-portfolio, priority populations, including children. To aid in this assessment, the RAND
Corporation was contracted to evaluate AHRQ's child health activities.
As requested by
AHRQ, this evaluation addresses four Primary Objectives:
- Measure and assess to what extent the Agency
contributed and disseminated and/or translated new knowledge.
- Measure and assess to what extent AHRQ's children's
healthcare activities improved clinical practice and health care outcomes and
influenced heath care policies.
- Measure and assess AHRQ's financial and staff support
for children's health activities.
- Measure and assess to what extent the Agency succeeded
in involving children's health care stakeholders and/or creating partnerships
to fund and disseminate key child health activities.
To address these objectives, RAND developed and
implemented a conceptual framework based on state-of-the-art research
evaluation theories (Roessner, 2002). This framework suggests that the effect of
any particular activity is likely to be highly indirect and that policy and
practice outcomes are multi-determined and depend on a loose network of actors
(funders, researchers, policy makers, the public, regulatory officials,
practitioners). It also suggests that
the impact of activities is typically characterized by a highly skewed
distribution, with a relatively small number of activities having a large
impact and the vast majority having a smaller impact. This framework allowed an assessment of the
Agency's contribution to the development and dissemination of new knowledge and
the impact of that knowledge on policies, clinical practice, and health
outcomes. In addition, we used databases
developed at
RAND to describe the Agency's
funding focus and financial investment in children's health over time.
Pursuant to
Primary Objective 1, we analyzed
funding for external activities and publications related to those activities,
overall and by category, and over time. In
terms of funding, we identified over $350 million that AHRQ used to fund external
research and activities related to children between 1990 and 2005. The
overall amount and average awards rose during the time period, despite a
decline in the last few years, as did the percent of the AHRQ overall budget
devoted to children's health activities. Examining external activities by AHRQ
strategic goals, children's health strategic goals, and AHRQ portfolios of work
shows that, over time, activities related to patient safety and health
information technology have increased from almost zero to about half of all
activities, reflecting trends in the Agency as a whole.
We also tracked publications derived from these
activities over time. In terms of AHRQ strategic goals, 54% of
the publications describe efforts to improve health outcomes and 31% relate to
the goal of improving access, appropriate use and efficiency. In terms of children's health strategic goals,
64% of the publications fit into the goal of contributing to new knowledge
about child health services. In terms of
AHRQ portfolios of work, 41% of the publications are categorized under the care
management portfolio and 31% describe results from projects on the cost,
organization, and socio-economics of health care. And because publications lag activities by
several years, in the near future, we expect an increasing number of
publications regarding health information technology and patient safety. We note that gathering the data required for
this analysis was difficult and time-consuming—there is no central repository
of information about all funding and all products that can be used for ongoing
monitoring of priority populations.
For Objective 1,
therefore, our evaluation shows that the Agency has contributed a substantial
body of new knowledge as a result of its funding for children's health research
(extramural and intramural) and has disseminated this new knowledge effectively
in the peer reviewed literature. This
analysis also shows that the child health portfolio has changed over time, reflecting
the overall Agency priorities.
It is difficult in
any field to trace a direct line between research or other activities and
improved health care activities or clinical practice, or between such activities
and an influence on health policy. Nevertheless,
pursuant to Primary Objective 2, we found that the 794 external and internal
publications arising from AHRQ's children's health activities were cited nearly
3,000 times in the scientific literature. Similar to previous analyses by AHRQ staff on
the results of outcomes research (Stryer, Tunis et al.
2000),
we found that 70% of the publications present research findings that may inform
policy and practice, but do not directly address particular decisions. Fourteen percent of the publications focus on
the impact of policy changes or have clear policy implications. Fifteen percent of the publications describe
research that evaluates clinical behavior, demonstrates changes in clinical
behavior, or demonstrates the use of tools in a clinical setting. One percent of the publications describe
research on the relationship between clinical or policy changes and health
outcomes.
Our case examples
and key stakeholder interviews, which focused on AHRQ's work in State
Children's Health Insurance Program/Child Health Insurance Initiative (SCHIP/CHIRI™),
attention deficit hyperactivity disorder (ADHD) and asthma illustrate several
lessons on maximizing the impact of AHRQ activities. First, the impact of AHRQ's activities (dissemination
of relevant information to policy-makers in a timely manner, changes in
practice and outcomes) is maximized when structures are in place to encourage
cooperation and communication among researchers and with a variety of
stakeholders. The SCHIP/CHIRI™ case illustrates
this point: the request for proposals (RFP) was structured to require
researchers to work with policy makers; AHRQ partnered with other funders; and
the individual projects were required to set aside funds to support initiative-wide
meetings and publications. In the case
of ADHD and asthma, one of the main impacts of AHRQ involvement was the ability
to draw on the resources of Centers for Education and Research on Therapeutics
(CERTs) to synthesize evidence, of practice-based research networks (PBRNs) to
change practice, and of tools to improve outcomes.
Second, in many of
the child health activities funded by AHRQ, impact all too often relied on the
individual efforts of Principal Investigators and AHRQ staff rather than being
programmed into the activities. With
some notable exceptions (for example, CHIRI™), activities are seen as focused on
generating products for academic journals. And, the AHRQ infrastructure was seen as being
oriented towards the 'front end' of research grants—soliciting, reviewing,
selecting, and funding—rather than the 'back end'—disseminating timely and
relevant information to policy makers, documenting impact on clinical practices
or outcomes, tracking and compiling the ways that AHRQ products are used by
various stakeholders.
Third, the
inherent tension in serving the needs of multiple stakeholders can be positive
or negative. In the case of CHIRI™,
tension lead to a creative solution that enhanced the impact of this set of
activities. In other cases, the tension
was not addressed as successfully. Many
interviewees alluded to the tension between the role of the Agency in funding
policy-relevant research versus directly informing the policy debate. Child health activities at AHRQ were seen as
being more focused on the former, rather than the latter. Another tension is between the needs of
academicians to generate peer-reviewed publications on the one hand, and the
needs of policy-makers, clinicians, and families for timely, actionable information
on the other. Clinicians and families
saw AHRQ's child health activities as being more focused on the needs of
academics rather than on those of the users. A further tension is between the broad spectrum of needs in children's
health and a very limited budget. Specific,
focused investments designed to generate spectacular answers to the "Porter
Question" (What effect is the research having on people?) were seen as key to
the Agency's continued viability.
In sum, for Objective
2, our bibliometric analysis, case studies, and key stakeholder interviews
suggest that children's health care activities at AHRQ, along with other child
health stakeholders, have played an important role in improving clinical
practice and health care outcomes and in influencing specific heath care
policies. They also suggest ways to
enhance the impact of future children's health activities.
AHRQ's objectives
and budget have changed over the years, both overall and for children's health
activities. A tightening budget and a
shift away from investigator-initiated external research grants have
highlighted questions regarding the place of children's health within AHRQ's
overall mission. Pursuant to Primary
Objective 3, two recurrent themes regarding AHRQ support for children's health
activities emerged from the key informant interviews. The first was the issue of whether children's
health should be considered (and funded) separately from other activities. Several interviewees made a cogent case for
theoretical, practical, and ethical reasons to consider children separately,
and made specific suggestions for how to accomplish this. The countervailing sentiment was that, in an
Agency with a relatively small budget, setting aside specific funding for
children's health activities was likely to result in a children's health budget
insufficient to accomplish any of the Agency's children's health strategic
goals. Nevertheless, there are specific
issues that are inherently child-only, such as research on SCHIP, for which it
is appropriate to argue for set aside funding.
The second theme
was the widespread perception that children's health activities at AHRQ were
dependent almost entirely on the efforts of individual Agency staff. Interviewees noted that there were virtually
no formal structures or policies at the Agency to ensure that children's health
was adequately addressed; the Child Health Advisory Group (CHAG) does not serve
this function. Interviewees praised
Denise Dougherty, as Senior Advisor on Child Health, and Lisa Simpson, the former
Deputy Director, for their unflagging efforts to raise the profile of children's
health, and noted that compared to other special populations, children's health
has a more organized and effective constituency within the Agency. However, they also noted that the Office of
Senior Advisor has neither authority nor funds to ensure inclusion of these
special populations and relied almost entirely on personal persuasion. They also noted that, with Lisa Simpson's
departure, high-level advocacy for children's health had diminished markedly.
Moreover, within Agency Offices and Centers, it was up to individual staff to
ensure that children's health was included in activities. Interviewees lamented that there did not seem
to be an institutional-level, coherent voice for child health at the Agency and
wondered aloud what would happen to children's health research at the national
level if AHRQ did not continue, in some important way, to lead.
Therefore, for
Objective 3, our interviews and case studies show that although individual
Agency staff have performed heroically in assuring continued support for child
health activities, there is a lack of authority or resources devoted to
children's health that has limited AHRQ's financial and staff support for children's
health research.
Pursuant to
Primary Objective 4, interviewees perceived AHRQ as having been successful in
nurturing a growing children's health services research community, through National
Research Service Award (NRSA) fellowships, conferences, and training grants. But they also noted that this same community
is in danger of dissolving or moving to other topics of inquiry as funding in
this area continues to be scarce. In
terms of partnering with other HHS entities, key stakeholders perceived limited
success. Interviewees at other HHS
entities professed little experience or interest in partnering with AHRQ,
despite substantial respect for the AHRQ personnel that they had interacted
with. As a result, AHRQ's partnerships with other HHS entities tend to involve
specific initiatives or committees and are initiated or maintained at the
program or project-officer level. Interviewees
could not cite examples of high-level inter-agency collaboration in children's
health. In terms of partnering with
policy entities, CHIRI™ was a good example of successful partnering, but
interviewees noted that there is little interest in children's health policy at
the federal level. Several interviewees
suggested that AHRQ ought to further collaborate with state-level policy makers
and with the Centers for Medicare and Medicaid Services (CMS) around SCHIP and
Medicaid. Interviewees reported mixed
success in partnering with the clinical and patient/family communities. They suggested that far more could be done to
partner with providers through professional societies such as the Association
of American Pediatricians (AAP) and other organizations involved in improving
children's health care.
Thus, for
Objective 4, we found that AHRQ staff has pursued numerous connections with
other agencies, but primarily through participation on committees and task
forces, both within and beyond HHS. The
Agency has had mixed success in involving children's health care stakeholders
and/or creating partnerships to fund and disseminate key child health
activities.
Building on this
analysis, we offer the following suggestions for the Agency's consideration:
Through their
children's health activities, AHRQ has successfully created and disseminated
knowledge and engaged the pediatric academic community, but has been less
successful in creating and documenting improvement and in engaging the
children's health quality, practitioner, and patient/family community. Answering the Porter Question for children
will require AHRQ's child health activities to focus more effectively on
identifying and pursuing opportunities to apply existing evidence, implement
change, and document improved processes and outcomes. It will also require the Agency to work more
closely with clinical organizations such as the American Academy
of Pediatrics (AAP) and with organizations organized around quality improvement
in order to better leverage the strengths of each. Experience with the CHIRI™ program as well as
with AHRQ's work on asthma and ADHD show that this is possible, at least in
well focused areas. The CHIRI™ program
required (and facilitated) collaboration between researchers and policymakers
and in that way could be a model for collaboration. However, it was organized
around an ongoing federal project requiring reauthorization. In the absence of a similar national quality
improvement project, it would be necessary to pursue a more piecemeal approach.
We note that, with the restriction of
funding for investigator-initiated grants and increasing prescriptiveness from HHS
regarding what funds can be used for, fewer opportunities exist for pediatric
quality improvement research.
The fate of children's health at AHRQ seems to rest
in the hands of a few extremely dedicated individuals who are forced to rely on
their powers of personal persuasion. In
order to institutionalize child health as an Agency priority, certain
structural and procedural changes are required. A more formal and rigorous monitoring of Agency funding commitment by
priority population should be instituted, with regular presentations to Agency leadership
about children's health activities and identification of under-funded areas. Endowing the Senior Advisor for Children's
Health with institutional authority (for example, participating in AHRQ staff
employee performance reviews) or with small amounts of discretionary funds to
pursue promising opportunities to answer the Porter Question would
substantially enhance effectiveness. AHRQ should also examine how to increase the effective use of the
contracts mechanism to pursue focused efforts to answer the Porter Question.
Given AHRQ's
limited budget, partnering/leveraging with other HHS entities is critical. The
initiative for partnering must come from the highest levels within the Agency,
rather than relying on program-officer level contacts. AHRQ should aggressively pursue opportunities
to collaborate with CMS around SCHIP and Medicaid. We note that AHRQ staff are currently engaged
in collaborative efforts with CMS around child health issues and that the
Agency's authorizing language must guide collaborations.
Finally, we note
that all of these analyses required a substantial effort on our part to
identify the AHRQ research and projects that were related to child health, as
well as to identify their outputs and effects. AHRQ should consider building a management
information system (or modifying existing systems) to reliably track the
inputs, outputs, and impacts of its child health activities so that this does
not have to be done on an ad hoc basis as for this study. Such an infrastructure would provide a
coherent and consistent picture of children's health activities at AHRQ, which
would be a useful tool for both management and evaluation purposes. The information system ought to be
focused on tracking dissemination and impact as well as getting through the
grant review and funding process.
Return to Contents
Proceed to Next Section
540 Gaither Road Rockville, MD 20850