Chapter 4. Summary and Conclusions
The four
objectives guiding this evaluation were:
- Measure and assess to what extent the Agency contributed new knowledge as a result of its funding for children's health research (extramural and intramural) and disseminated and/or translated effectively its findings to meet AHRQ's strategic objectives of improving the safety, quality, effectiveness and efficiency of health care as well as wider DHHS strategic objectives.
- Measure and assess to what extent AHRQ's children's health care activities, i.e., its research findings, meetings, conference support, products, tools, etc., improved clinical practice and health care outcomes and influenced heath care policies over the past fifteen years.
- Measure and assess AHRQ's financial and staff support for children's health research as well as Agency internal handling of children's health grants, contracts and intramural activities research with/among other AHRQ programs, portfolios and activities and other DHHS and federal agency efforts.
- Measure and assess to what extent the Agency succeeded in involving children's health care stakeholders and/or create partnerships to fund and disseminate key child health activities.
Summary
Primary Objectives 1 and 3
We addressed Objectives 1 and 3 together by examining the number of activities, funding for those activities, and number of publications over time and according to strategic goals and portfolios of work. In addition, we interviewed key stakeholders about their perspectives regarding support for children's health at AHRQ.
We found that over
the past 15 years, there have been significant shifts in the number of child
health activities funded, the amount of funding (both absolute and as a
percentage of the AHRQ budget), and the strategic goals or portfolios of work
funded.
Through most of
the 1990's, AHRQ maintained a steady level of external activities related to
children's health. Starting in 1999,
both the number of activities and the total funding directed towards children's
health projects rose dramatically before dropping off again in 2003. This corresponds to a relative increase in
the proportion of the total AHRQ budget that went toward external activities
related to children's health from 9 percent prior to 1999 to 28 percent during
the 1999 to 2002 time period. Since
2003, there has been a noticeable downshift in both the number of external
activities and the funding for them.
In examining the
types of external activities, we found a shift toward research on health
information technology and patient safety. The proportion of activities addressing the AHRQ Strategic Goals in
these areas increased from none to nearly one-half of the overall
portfolio. Similarly, activities that
corresponded to the AHRQ Children's Health Strategic Goal on creating tools and
nourishing talent for children's health services research went from 15 to 44
percent of the total. Looking at AHRQ's
external activities through the lens of the Portfolios of Research, we found
that the activities addressing the health information technology and patient
safety missions increased from 0 to 39 percent of the entire portfolio over
time. Each of these trends reflects
general shifts in priorities for AHRQ as a whole.
The number of
publications tracked the level of funding for strategic goals and portfolios of
work. Exceptions were that the percent
of publications in the health information technology and patient safety
portfolios were less than the percent of funding for those portfolios, but this
is probably due to the fact that these activities have been funded relatively
recently and so have likely not produced their full complement of publications.
Two recurrent
themes regarding AHRQ support for children's health activities emerged from the
key informant interviews. The first was
the issue of whether children's health should be considered (and funded)
separately from other activities. Several
interviewees made a cogent case for theoretical, practical, and ethical reasons
to consider children separately and made specific suggestions for how to
accomplish this. The countervailing
sentiment was that, in an Agency with a relatively small budget, setting aside
specific funding for children's health activities was likely to result in a
children's health budget insufficient to accomplish any of the Agency's
children's health strategic goals. Nevertheless, there are specific issues that
are inherently child-only, such as research on SCHIP, for which it is
appropriate to argue for set aside funding.
The second theme,
related to the first, was the perception that children's health activities at
AHRQ were dependent almost entirely on the efforts of individual Agency staff. Interviewees noted that there were virtually
no formal structures or policies at the Agency to ensure that children's health
was adequately addressed. Interviewees praised Denise Dougherty, as Senior
Advisor on Child Health, and Lisa Simpson, for their unflagging efforts to
raise the profile of child health, and noted that compared to other special
populations, child health has a more organized and effective constituency
within the Agency. However, they also
noted that the Office of Senior Advisor has neither authority nor funds to
ensure inclusion of this special population and relied almost entirely on
personal persuasion. They also noted
that, with Lisa Simpson's departure, advocacy for child health at the level of
the Office of the Director had diminished markedly. Moreover, within Agency
Offices and Centers, it was up to individual staff to ensure that children's
health was included in activities. Interviewees lamented that there did not seem to be an
institutional-level, coherent voice for child health at the Agency and wondered
aloud what would happen to children's health research at the national level if
AHRQ did not continue, in some important way, to lead.
Overall, the
Agency has supported a broad range of child health activities and produced a
substantial body of new knowledge in children's health. There was a marked increase in activities and
funding in 1999, followed by declines after 2002 and corresponding shifts, in
external research grants, away from investigator-initiated grants to targeted
solicitations increasingly focused on health information technology and patient
safety. These changes brought into stark
relief questions regarding the place of children's health within AHRQ's overall
mission and the corresponding lack of structural support (authority or
resources) devoted to children's health. Many key stakeholders worried aloud that
children's health research is critically endangered and wondered whether AHRQ
would continue to play a major leadership role in children's health on the
national level.
Primary Objective 2
We addressed Objective 2, measuring and assessing the impact of AHRQ child health activities, through bibliometric analyses and case studies. In the Stryer analysis, we found that the majority of publications resulting from AHRQ-funded child health activities were descriptive findings. This is consistent with AHRQ's first children's health strategic goal—to contribute new knowledge about child health services. A substantial minority of publications evaluated or informed policy or practice. Relatively few publications evaluated interventions to improve outcomes. This highlights the difficulty of publishing studies that answer the Porter Question. Categories of external activities that had relatively higher Stryer scores included 'Using data to make informed choices,' 'Translating new knowledge into practice,' 'Health information technology portfolio,' and 'Prevention portfolio.' The average child health publication was cited 6.6 times and appeared in journals of impact scores average about 3.6. Unfortunately, we do not have comparative data on other AHRQ or non-AHRQ research programs, which are necessary to put these analyses in context.
The qualitative case studies and key informant interviews were meant to illuminate the processes by which AHRQ has an impact on child health activities. Both case studies—SCHIP/CHIRI™ and moving evidence to practice in ADHD and asthma—illustrated several lessons on maximizing the impact of AHRQ activities.
First, impact is maximized when structures are in place to encourage cooperation and communication among researchers and a variety of stakeholders. The SCHIP/CHIRI™ case exemplified this lesson: The RFP was structured to require researchers to work with policy makers; AHRQ partnered with other funders and was able to benefit from the Packard Foundation's additional funding and also from its focus on dissemination and its ability to underwrite activities that foster collaboration (for example, dinner meetings); and the individual projects were required to set aside funds to support initiative-wide meetings and publications. In the case of ADHD and asthma, one of the main impacts of AHRQ involvement was the ability to draw on the resources of CERTs to synthesize evidence, of PBRNs to change practice, and of tools to improve outcomes.
A second lesson from the case studies was that, in many of the child health activities funded by AHRQ, impact (disseminating relevant information to policy-makers in a timely manner, changing and documenting change in practice and outcomes) all too often relied on the individual efforts of PIs and AHRQ staff rather than
being programmed into the activities. With
some notable exceptions (for example, CHIRI™), activities are seen as focused on
generating products for academic journals. Additionally, the AHRQ infrastructure was seen
as being oriented towards the 'front end' of research grants—soliciting,
reviewing, selecting, and funding—rather than the 'back end'—disseminating
timely and relevant information to policy makers, documenting impact on
clinical practices or outcomes, tracking and compiling the ways that AHRQ
products are used by various stakeholders.
A third lesson
from the case studies, and related to the second, was the tension inherent in
serving the needs of multiple stakeholders. In the case of CHIRI™, tension led to a
creative solution that enhanced the impact of this set of activities. In other
cases, the tension was not addressed as successfully. Many interviewees alluded to the tension
between the role of the Agency in funding policy-relevant research versus
directly informing the policy debate. Child
health activities at AHRQ were seen as being more focused on the former, rather
than the latter. Another tension is between the needs of academicians to
generate peer-reviewed publications on the one hand, and the needs of
policy-makers, clinicians, and families for timely, actionable information on
the other. Clinicians and family-oriented
organizations saw AHRQ's child health activities as being more focused on the
needs of academics rather than on their needs. Another tension is the one between a broad spectrum of needs in
children's health and a very limited budget. Specific, focused investments designed to
generate spectacular answers to the Porter Question were seen as key to the
Agency's continued viability.
Overall,
AHRQ-funded child health activities have had a substantial impact on the
research community, as the bibliometric analysis demonstrates. Impact on policy and clinical care was
maximized when structures were in place to encourage collaboration among
stakeholders and when dissemination and implementation were planned, not hoped
for.
Primary Objective 4
The key informant
interviews suggest themes regarding AHRQ's partnering with the children's
health services research community, with other HHS agencies, with the policy
community, and with the clinical/improvement community.
Interviewees
perceived AHRQ as having been successful in nurturing a growing children's
health services research community, through NRSA fellowships, conferences, and
training grants. However, they also
noted that this same community is in danger of dissolving or moving to other
areas of inquiry as funding in this area continues to be scarce.
In terms of
partnering with other HHS entities, key stakeholders perceived limited success.
Interviewees at other HHS entities
professed little experience or interest in partnering with AHRQ, despite
substantial respect for the AHRQ personnel that they had interacted with. The partnerships with other HHS entities tended to be around specific initiatives or committees and were seen as
initiated or maintained at the program or project-officer level. Interviewees could not cite examples of
high-level inter-agency collaboration in children's health.
In terms of
partnering with policy entities, CHIRI™ was a good example of successful
partnering, but interviewees noted that there is little interest in children's
health policy at the federal level. Several
interviewees suggested that AHRQ ought to further collaborate with state-level
policy makers and with CMS around SCHIP and Medicaid.
Interviewees
reported mixed success in partnering with the clinical and patient/family
communities. They suggested that far
more could be done to partner with providers through professional societies
such as the AAP and through organizations involved in improving children's
health care.
Overall, AHRQ's partnering with other entities has been mixed, but promising opportunities exist.
Return to Contents
Conclusions and Recommendations
Although the focus
of our analyses was on child health activities, many of the comments we
received related to AHRQ as a whole, and it seemed that at least some of the
child health specific issues that we identified may apply to other parts of
AHRQ as well. So while our conclusions
and recommendations address child health specifically, some might have more
general application.
Pursuant to
Primary Objective 1, our evaluation shows that the Agency has contributed a
substantial body of new knowledge as a result of its funding for children's
health research (extramural and intramural) and has disseminated this new
knowledge effectively in the peer reviewed literature.
It is difficult in
any field to trace a direct line between research activities and improved
health care activities or clinical practice, or between research and an
influence on health policy. Nevertheless, pursuant to Primary Objective 2, our
bibliometric analysis, case studies, and key stakeholder interviews suggest
that AHRQ's children's health care activities have played an important role,
along with activities of other child health stakeholders, in improving clinical
practice and health care outcomes and in influencing specific heath care
policies. We note, however, that all of
these analyses required a substantial effort on our part to identify the AHRQ
research and projects that were related to child health, as well as to identify
their outputs and effects.
AHRQ's objectives
and budget have changed over the years, both overall and for children's health
activities. A tightening budget and a
shift away from investigator-initiated external research grants have
highlighted questions regarding the place of children's health within AHRQ's
overall mission. Pursuant to Primary
Objective 3, our interviews and case studies show that individual Agency staff
have performed heroically in assuring continued support for child health
activities. But the interviews and case
studies also suggest that there is a lack of structural support (authority or
resources) devoted to children's health and that this has limited AHRQ's
financial and staff support for children's health research.
Pursuant to
Primary Objective 4, the Agency has had mixed success in involving children's
health care stakeholders and/or creating partnerships to fund and disseminate
key child health activities. AHRQ has
been successful in helping to create a community of children's health services
researchers and in disseminating policy information to state-level policy
makers. There has been some success in
engaging with the practice community through entities such as the AAP, but more
is needed. There has been limited
success in partnering with other HHS entities and with federal-level policy
makers.
Building on this
analysis, we offer the following suggestions for the Agency's consideration.
Through their
children's health activities, AHRQ has successfully created and disseminated
knowledge and engaged the pediatric academic community, but has been less
successful in creating and documenting improvement and in engaging the
children's health quality, practitioner, and patient/family community. Answering "the Porter Question" (What effect is the research having on people?) for children will require AHRQ's child health
activities to focus more effectively on identifying and pursuing opportunities
to apply existing evidence, to implement change, and to document improved
processes and outcomes. It will also
require the Agency to work more closely with clinical organizations such as the
AAP and with quality improvement organizations in order to better leverage the
strengths of each. Experience with the CHIRI™ program as well as with AHRQ's work on asthma and ADHD show that this is possible, at least in well focused areas. The CHIRI™ program required (and facilitated) collaboration between researchers and policymakers and in that way could be a model for
collaboration. However, it was organized
around an ongoing federal project slated for possible reauthorization. In the absence of a similar national quality improvement project, it would be necessary to pursue a more piecemeal approach.
We note that, with the restriction of
funding for investigator-initiated grants and increasing prescriptiveness from
DHHS regarding what funds can be used for, fewer opportunities exist for
pediatric quality improvement research.
The fate of children's health at AHRQ seems to rest in the hands of a few extremely dedicated individuals who are forced to rely on their powers of personal persuasion. In
order to institutionalize child health as an Agency priority, certain
structural and procedural changes are required. A more formal and rigorous monitoring of Agency funding commitment by
priority population should be instituted, with regular presentations to Agency
leadership and identification of under-funded areas. Endowing the Senior Advisor for Children's
Health with institutional authority (for example, participating in AHRQ staff
employee performance reviews) or with small amounts of discretionary funds to
pursue promising opportunities to answer the Porter Question would
substantially enhance effectiveness. AHRQ should also examine how to increase the effective use of the
contracts mechanism to pursue focused efforts to answer the Porter Question.
Given AHRQ's limited budget, partnering/leveraging other HHS entities is critical. The initiative for partnering must come from the highest levels within the Agency, rather than relying on program-officer
level contacts. AHRQ should aggressively
pursue collaboration with CMS around SCHIP and Medicaid. We note that AHRQ staff are currently engaged
in collaborative efforts with CMS around child health issues and that the
Agency's authorizing language must guide collaborations.
Finally, we note that all of these analyses required a substantial effort on our part to
identify the AHRQ research and projects that were related to child health, as
well as to identify their outputs and effects. AHRQ should consider building a system (or
modifying existing systems) to reliably track the inputs, outputs, and impacts
of its child health activities so that this does not have to be done on an ad
hoc basis as for this study. Such an
infrastructure would provide a coherent and consistent picture of children's
health activities at AHRQ, which would be a useful tool for both management and
evaluation purposes. In addition, given
shifts in the amount and direction of funding, it is increasingly important to
devote adequate resources to ensuring continuity in institutional memory and in
institutional message.
Return to Contents
Bibliography
Forrest, C. B., L. Simpson, et al. (1997). "Child health services research. Challenges and opportunities." JAMA 277(22):1787-93.
Gidwani, P., E. Sobo, et al. (2003). Laying the foundation: Identifying major issues in applied child health services research. Child health services research: applications, innovations, and insights. E. Sobo and P. Kurtin. San Francisco, Jossey-Bass: 25-66.
Roessner, D. (2002). Outcome measurement in the United States: State of the art. American Association for the Advancement of Science. Boston, MA.
Stryer, D., S. Tunis, et al. (2000). "The outcomes of outcomes and effectiveness research: impacts and lessons from the first decade." Health Serv Res 35(5 Pt 1):977-93.
Return to Contents
Proceed to Next Section