Chapter 3. Results
Primary Objectives 1 and 3
Primary Objective
1 was to measure and assess to what extent the Agency contributed new knowledge
as a result of its funding for children's health research (extramural and
intramural) and disseminated and/or translated effectively its findings to meet
AHRQ's strategic objectives of improving the safety, quality, effectiveness and
efficiency of health care as well as wider DHHS strategic objectives. Primary Objective 3 was to measure and assess
AHRQ's financial and staff support for children's health research as well as
Agency internal handling of children's health grants, contracts and intramural
activities research with/among other AHRQ programs, portfolios and activities
and other DHHS and federal agency efforts.
To address these objectives, we examined the number
of activities, level of funding, and articles produced. We first describe these over time, then by
AHRQ Strategic Goal, by AHRQ Children's Health Strategic Goal, and by AHRQ
Portfolios of Work. We also report the
views of key informants with respect to Objective 3. All of the tables cited in this section
appear in Appendix C.
Children's health
activities over time
Number of activities
Using the methods
described in the preceding section, we identified a total of 514 external
activities related to children's health. These included grants or contracts for specific research projects in
addition to grants for conferences, meetings, workshops and centers. Looking at the number of activities across a
16 year time period shows the variation across time that reflects changes in
priorities, staff and funding (Table 3). We then looked at the distribution of external activities across
different time periods: 1990-1995,
1996-1998, 1999-2002 and 2003-2005 (Table 4). We somewhat arbitrarily divided the time periods based on events of
importance for children's health at the Agency. For example, 1996 was the year that Lisa Simpson was appointed Deputy
Director, 1999 was the year the Agency was reauthorized as AHRQ, and 2002 was
the year Lisa Simpson left the Agency. AHRQ funded an average of about 14 external activities from 1990 through
1995. The average rose slightly to 18
external activities per year in the period of 1996-1998. Starting in 1999 (the year in which the
Agency for Health Care Research and Policy was reauthorized as AHRQ) and
lasting through 2002 the average rose substantially to 64. AHRQ funded an average of just over 39
external activities per year from 2003 to 2005.
Figure 1 shows the
total number of external activities by year with the average for the time
period noted by the dashed line. Even
within the time periods, there was considerable variation from year to year in
the number of external activities funded in a particular year. Nonetheless, and notwithstanding the relative
decrease in the last time period, the number of external activities rose
considerably over the time period studied.
Funding for activities
As described in
the preceding section, we obtained funding information for 426 of the 514
external activities. We were unable to
find funding information for most of the activities initially funded in 2005,
for those related to the evidence based practice centers, for contract work and
for a few others of the external research activities. Excluding these, we identified more than $350
million for external research and activities related to children between 1990
and 2005. The total funding for this subset of the external activities across
the 1990 to 2005 time period ranges from less than $5 million per year to more
than $55 million per year (Figure 2). Looking at the total funding across the time periods of interest, the
majority (59%) of the funding for occurred between 1999 and 2002 (Table 5).
Given the
variation in the number of activities per year, we also examined the average
funding per external activity over the life of the grant by the initial funding
year (Figure 3). Overall, the funding
varied from $10,000 for faculty development awards or capacity building grants
to more than $5 million for different quality, patient safety and HIT
activities like CAHPS® or CERTS. From
1990 to 1993 the average award rose from close to $400,000 to over $600,000
before dropping back to around the 1990 level by 1995. In 1996, the average funding per external
activity rose markedly to over $900,000 before a sharp decline in 1997 to just
over $200,000. The average award approached
$1 million in 1998 before going over in 1999. By 2005, the average award had stabilized somewhat at just under $1
million. The variation in the average
award reflects the start of different initiatives such as CHIRI™.
We also examined the external activities related to
children as a percentage of the total AHRQ budget (Figure 4). Using information
from AHRQ budget justification documents found on the AHRQ Web site, we found
the total funding for AHRQ during the time period of interest.17 Figure 4 shows the
percentage of the total AHRQ budget spent on external activities related to
children. Between 1990 and 1995 external activities related to children
averaged just 6 percent of the AHRQ budget. The percentage rose somewhat to an
average of 9 percent between 1996 and 1998. Again, the 1999 to 2002 time period
shows dramatic changes with the average percentage of external activities
related to children rising all the way to 23 percent of the AHRQ budget. From
2003 to 2005 the average fell again to 7 percent of the AHRQ budget. Note that
the child health activities are categorized by first year of funding, which the
overall AHRQ budget in any year includes projects that started at some time in
the past, so the comparison is only approximate.
Overall, the funding analyses shows that the
amount of funding and average awards for external activities related to
children rose during the time period, but that there was no actual increase in
the percent of the total AHRQ budget dedicated to these activities because the
AHRQ budget increased at a similar pace.
Publications
Recognizing that
the research projects take several years to complete and that publications
often lag behind, we focused our publications analysis on external research
projects that were funded from 1996 through 2002. Our bibliometric search (described in section
2) identified a total of 749 publications from the 258 AHRQ children's health
activities funded between 1996 and 2002.18 We first looked at the distribution of these
publications by the initial funding year of the grants (Table 6). Since the number of activities varied from
year to year, we also calculated the average number of publications per
activity for each year. The activities
funded in 1996 averaged 3.6 publications per project. The average fell in 1997 and 1998 before
rising substantially in 1999 to an average of 4.4 publications per
project. The pace of publications fell
again in 2000 to an average of 3.4 per activity. While projects funded in 2001 and 2002
averaged less than two publications per project, the lower numbers likely
reflect the time needed for research and publication rather than the
productivity of the activities per se. Over the time period examined, the external activities related to
children's health averaged almost three publications per activity.
AHRQ strategic
goals
We categorized the
external activities using three different typologies: AHRQ strategic goals, AHRQ children's health
strategic plan goals, and AHRQ portfolios of work. For the AHRQ strategic goals we looked at the
Strategic Plan released in 199819 and the FY 2006 Budget Justification.20 Both documents lay out strategic goals for
the agency (Table 7). AHRQ developed the
three Strategic Plan goals to carry out its mission. In developing the Budget Justification goals,
AHRQ looked across its entire research portfolio and created categories to
describe different groups of activities. The first three of these goals have been in place since the FY 2000
budget justification. The fourth goal was
added in the FY 2005 budget justification. We have not included the fourth goal in our categorization scheme since
it relates to AHRQ's internal processes.
We combined these
two sets of strategic goals into a single set of five AHRQ strategic goals. Table 7 shows how the two sets of goals map
into the consolidated set of AHRQ strategic goals. Since both sets had a goal related to access
we grouped those together. Otherwise we
included the other goals from each set separately. We then looked at how the external activities
distribute across these strategic goal categories (Table 8). Overall, over one-half of the activities
(52%) fall into the area of supporting improvements in health outcomes. This category includes a wide variety of
grants on the causes and effects of specific medical conditions, risk factors
and characteristics of those with specific medical conditions, and patient
behaviors. This category also includes
research on the effectiveness of interventions and establishing an evidence
base in a particular area. Research
activities that examined disparities in outcomes related to race/ethnicity,
income level, location or risk status were also included in this category. We classified activities related to
developing curricula, tools, decision aids or guidelines in this strategic plan
area. Training related activities such
as fellowships, faculty research programs and research collaborations also fell
into this category.
One-quarter of the
external activities related to health care utilization and access to care. These included research on utilization and
cost effectiveness of SCHIP, Medicaid, and HMOs. Ten percent of the external activities
related to technology and the development of information systems, including
studies on medical informatics, electronic interfaces, telemedicine, electronic
medical records and other health information technology projects. Another 11 percent of the projects focused on
patient safety, including research on medical errors and quality of care. Just two percent of the activities focused on
strengthening quality measurement and quality improvement.
Over time, the
distribution of activities across these strategic goals shifted in some cases
by substantial amounts. The proportion
of activities related to improving health outcomes declined from 63 percent
during 1990-1995 to 38 percent for the 2003-2005 time period. Over that same period, the percentage of
external activities focused on improving access, appropriate use and efficiency
declined from 37 percent to 13 percent. The relative emphasis on health information technology and data driven
projects increased from zero to 31 percent of external activities. A similar pattern emerges when looking at the
safety goal, where the percentage rose from 8 percent to 18 percent over
time.
We also looked at
the how the funding was distributed across the AHRQ Strategic Goals (Table 9). Among the subset of external
activities with available information on funding, 40 percent of the total
funding went to projects aimed at improving health outcomes. Nearly one-quarter (24%) of the total funded
projects went to promote patient safety. Another 21 percent of the total funding went toward external activities
to improve access, appropriate use, efficiency and costs. Overall, the funding for activities on data
and health information technology represented 13 percent of the total,
including studies on database development, medical informatics, telemedicine,
electronic medical records and other health information technology
projects. Relatively little (1%) of the
total funding went toward activities for quality measurement and
improvement.
When comparing the
distributions of funding and activities across the AHRQ Strategic Goals, we see
that relatively more funding went to promoting safety (24% of funding v. 11% of
activities) and less to improving health outcomes (40% of funding v. 52% of
activities). These differences
presumably reflect both the higher average costs of safety projects, and the
increase in the average cost of projects over time (Figure 2), since these
categories were more prominent later in the period.
We categorized
each publication using the different categorization schemes by assigning the
publication to the category of the external activity from which it derived
(Table 10). More than one-half of the
publications (54%) describe efforts to improve health outcomes. Nearly one-third of the publications (31%)
relate to the goal of improving access, appropriate use and efficiency. Ten percent of the publications from external
activities involve promoting patient safety. Relatively few of the publications fall into the strategic goal
categories on using data to make informed decisions (3%) and strengthening
quality measurement and improvement (2%). Overall, the distribution of publications across the AHRQ strategic goal
areas tracks closely the distribution of the activities across these
areas. There were somewhat more
publications than activities on improving access to care and efficiency (31% v.
25%) and fewer publications on health information technology and data use (3%
v. 10%).
Looking at the
analyses by AHRQ strategic goals more broadly, we see that the majority of
activities fit into the categories intended to "improve health outcomes" and
"improve access, appropriate use and efficiency and reduce costs." Over time, however, the proportion of
activities addressing these two strategic goals decreased from 100 percent of
the children's health activities to 51 percent. The number of activities addressing the strategic goals related to
safety and health information technology together make up 22 percent of the
entire portfolio, but have increased from 0 to 49 percent over time. The overall funding picture for the AHRQ
strategic goal areas largely reflects this trend with a relatively larger
proportion of the funding going towards activities in the safety area. As noted above, the publications more closely
tracked the distribution of activities with the majority of publications coming
from activities in the areas of improving health outcomes and improving access,
appropriate use and efficiency and reducing costs.
AHRQ children's
health strategic goals
For the second
categorization scheme, we looked at the 1999 strategic plan document that designated
children as a priority population.21 In
that document, AHRQ identified a set of six goals for their children's health
agenda:
- Contribute to new knowledge about child
health services.
- Create tools and nourish talent to
strengthen the knowledge base in child health services.
- Translate new knowledge into practice.
- Improve communication with stakeholders in
child health.
- Include children and child health care in
all AHRQ-supported research, as scientifically and ethically appropriate.
- Balance the AHRQ research portfolio to
represent a broader range of children and child health care.
Looking at the distribution of external activities
related to children's health across these six goals, nearly two-thirds (61%) of
the external activities related to contributing new knowledge about child
health services, including activities on outcomes, quality, safety, cost,
utilization and access to care (Table 11). One-quarter of the external activities focused on creating tools and
supporting researchers involved in children's health services research. This category includes activities designed to
develop instruments, tools or guidelines as well as those supporting research
networks and programs. The health
information technology, database development, electronic medical record, and
telemedicine activities also fall into this category.
The other four children's health agenda goals were
considerably less common. Eight percent
of the external activities were aimed at representing a broader range of
children and child health care. Here, we included activities that address
disproportionality, disparities and vulnerable populations. Six percent of the external activities
related to translating new knowledge into practice. Many of these were evidence reviews carried
out by the AHRQ-funded Evidence-Based Practice Centers. While many of the external activities could
be categorized as including children and child health care in all
AHRQ-supported research, we used this category for only a handful of activities,
preferring to categorize the activities according to the subject matter. The few activities that fell into this
category related to public health issues such as hospital disaster plans and
surge capacity that effect children as well as adults. None of the external activities were
categorized as primarily related to improving communication with stakeholders
in child health. While AHRQ funded a
number of conferences and workshops to bring stakeholders together, these
activities were categorized according to their topical area.
Over time, there
were notable shifts in how the external activities fit into the different
children's health agenda goal categories. From 1990 to 1995, nearly three quarters (72%) of the external
activities related to contributing new knowledge about child health
services. By the 2003-2005 time period,
the percentage had fallen to 45 percent. Over the same time period, more of the activities fell into the category
for creating tools and nourishing talent to strengthen the knowledge base with
the percentage rising from 15 percent in 1990-1995 to 44 percent in
2003-2005. The percentages for the other
children's health agenda goals remained fairly consistent over time.
We also looked at
the distribution of total funding across the Children's Health Strategic Plan
Goals (Table 12). Among those activities
with funding information available, 55 percent of the total funding went to
activities designed to contribute to new knowledge about child health
services. More than one-quarter (28%) of
the total funded external activities were aimed at creating tools and
nourishing talent. Nearly equal
percentages of the total funding went to projects to translate evidence into
practice (8%) and to include a broader range of children in AHRQ research
(9%). Very little of the total funding
(1%) went to projects that fit into the category on including children and
child health care in all AHRQ-supported research. As we mentioned earlier, while many
activities fit into this overarching goal area we categorized the activities by
subject matter so few activities fell into this category. Overall, the distribution of funding across
the Children's Health Strategic Plan Goals closely tracks the distribution of
activities.
We also
categorized the publications coming out of AHRQ's child-related external
activities according to the Children's Health Strategic Plan Goals (Table 13). Almost two-thirds (64%) of the
publications fit into the category on contributing to new knowledge about child
health services. Nearly one-fifth of the
publications (18%) related to the goal of creating tools and training
investigators. Nine percent of the
publications described translating knowledge into practice. Another nine percent related to broadening
the range of children included in research. The distribution of publications across the Children's Health Strategic
Plan Goals closely mirrors the distribution of the external activities
described earlier.
An overall picture
of the Children's Health Strategic Plan Goals shows that most of the activities
sought to "contribute to new knowledge about child health services" and to
"create tools and nourish talent to strengthen the knowledge base in child
health services". Over time, however,
the breakdown of activities in each of these categories changed dramatically
from a relative imbalance (72% v. 15%) to nearly equal (45% v. 44%). The distribution of funding and publications
across the Children's Health Strategic Plan Goals closely tracks the overall
distribution of activities with 61% related to contributing new knowledge and 25%
related to creating tools and nourishing talent.
AHRQ portfolios
of work
The third
categorization scheme used the ten AHRQ Portfolios of Research to classify the
external activities (Table 14).22
Overall, 37% of
the external activities related to children's health funded by AHRQ from 1990
to 2005 fall into the care management portfolio (Table 15). These included research projects on specific
medical conditions, health disparities, outcomes, quality improvement and the
development of instruments, tools, and guidelines to aid clinical
practice. Twenty-six percent of the
children's health research is related to the cost, organization, and
socio-economic portfolio of research. This research focuses on utilization, access and cost effectiveness and
includes the work on SCHIP and Medicaid.
Ten percent of
external activities related to the health information technology mission,
including those on medical informatics, electronic medical records, and bar
code technology. A few additional
activities specifically related to database development were categorized into
the data development portfolio of research. Eleven percent of the activities fall into the patient safety
portfolio. Overall, just six percent of
the activities addressed the prevention portfolio which focuses on evaluating
effectiveness and promoting evidence-based practice. Another seven percent of the activities are
categorized in the training portfolio. These include some fellowship and conference grants as well as grants to
support faculty research programs. Only
a few of the children's health related activities related to the pharmaceutical
outcomes portfolio (2%) and the system capacity and emergence preparedness
portfolio (1%).
Looking at the
distribution of activities over time, there are some notable shifts in the
relative emphasis on the different portfolios of research. Two of the portfolios decreased notably. By the 2003-2005 time period, activities
categorized in the care management portfolio had decreased from 50 percent to
26 percent of the total. Similarly,
research projects in the cost,
organization, and socio-economics portfolio declined from 38 percent to 13
percent. Two other portfolios gained
substantially. From 1990 to 1995 there
were no health information technology projects related to children's health at
AHRQ. By the 2003-2005 period, one-third
of the external activities involved health information technology
research. Similarly, the patent safety
portfolio grew from zero to 16 percent of all external activities. The percentages for the other portfolios of
work remained fairly consistent over time.
As with the other
categorization schemes, we also examined the distribution of total funding
across the AHRQ Portfolios of Research for those activities with available
funding information (Table 16). Nearly
equal percentage of the total funding went toward external activities in the
care management mission (23%) and cost, organization, socio-economics mission
(22%). Eighteen percent of the total
funded external activities on patient safety, while 14 percent funded health
information technology work. Less than
ten percent of the total funding was directed toward external activities in the
training mission (9%), prevention mission (7%), pharmaceutical outcomes mission
(6%), data development mission (<1%) and system capacity and emergency
preparedness mission (<1%). When
comparing the distribution of funding to the distribution of activities, a few
of the portfolios of research represent more of the total funding than the
activities. For example, external
activities related to health information technology represent 14 percent of the
funding and 10 percent of the activities. Likewise, patient safety activities are 18 percent of the total funding
and 11 percent of the activities.
The distribution
of publications according to the AHRQ Portfolios of Research shows that a
substantial minority (41%) of the publications were categorized under the care
management mission which includes research on specific medical conditions and
health outcomes (Table 17). Nearly
one-third of the publications (31%) describe results from projects on the cost,
organization, and socio-economics of health care. Nine percent of the publications relate to
patient safety and another 9 percent relate to prevention. Relatively few of the publications fell into
the portfolios for health information technology (3%), pharmaceutical outcomes
(4%) and training (3%). The distribution
of publications across the Portfolios of Research looks quite similar to the
distribution of the external activities described earlier.
Looking across all
of the analyses related to the Portfolios of Research, we see the same kind of
trend toward safety and health information technology that we saw with the AHRQ
strategic plan goals. Overall during the
study period, 37 percent of activities were related to the care management
mission and 26 percent were related to the cost, organization, and
socio-economics mission. Over time,
however, the proportion of activities addressing these two strategic goals
decreased by nearly half from 88 percent of the children's health activities to
49 percent. The number of activities
addressing the health information technology and patient safety missions each
make up about 10 percent of the entire portfolio, but have increased from 0 to
39 percent over time. For the funding,
we see that external activities related to patient safety and health
information technology represent more of the total funding than of the total
activities. As noted above, this pattern
is not seen with the publications where the distribution of publications
mirrors that of the activities.
Key informant
interviews regarding AHRQ support for children's health activities
Many
interviewees shared a similar view of AHRQ's support for children's health
activities. They were grateful for the support that had been given and praised
AHRQ staff for their tenacity in championing child health issues. However, they were frustrated that,
rather than being built into the structure of the organization, children's health activities were supported exclusively to the degree
that AHRQ staff devoted time and effort to them. As one interviewee reported,
"AHRQ deserves a
lot of credit for what they have done and Lisa (Simpson) and Denise
(Dougherty)'s work has been great. But
they need to institutionalize it rather than being dependent on one person. Children's health needs to become part of a
checklist that everyone goes through. Whatever activity or project someone is working on, they should be
looking from the beginning at whether there is a children's component."
This was the case for external research activities as well as for
intramural research and internal functions. Thus, RFP development, study section composition, decisions at the
funding committee level, reports generated by intramural researchers,
participation in the Children's Health Advisory Group (CHAG), and initiatives
at various Centers all depended on the effort of individual AHRQ staff members.
Respondents praised Denise Dougherty and
Lisa Simpson23 for
their efforts to bring children's health to the fore and to make sure that
children were represented in all AHRQ activities. Several noted that, compared to other priority
populations, children's health was successful in maintaining its profile at
AHRQ. However, respondents noted that
without specific authority or resources devoted to children's health or vested
in the position of Senior Advisor on Children's Health, virtually all
children's health activities relied on the personal influence of a small number
of AHRQ staff members.
Many respondents acknowledged the difficulties in integrating children's
health into the larger agenda and of raising its stature in the debate over
health care. As one interviewee noted,
"There is not a
coherent voice about children's health and health care issues at the policy
level at AHRQ. There is no clear
receptor site for those activities. Even
with the listserv and other kids' activities, there still is not a coherent
view. No one has figured out how to talk
about the children's quality agenda and AHRQ has not devoted a lot of resources
to area. It's starting to look at these
issues but only with outside prodding. "
Similarly,
respondents noted that AHRQ's budget for external research was extremely
limited and that much of it was earmarked for specific initiatives that were
more relevant to the Medicare population than to children. Reflecting this, several respondents worried
that children's health services research may be critically endangered, since
"if they (AHRQ) don't do it, then who will?" This is especially true given that no other
federal agency has the same kind of mandate as AHRQ.
Part
of the integration difficulty is reflected in the fact that creating a cohesive
children's health activities portfolio is challenging. As another respondent
reported:
"There is a strong
children's health portfolio but not necessarily something that naturally comes together
(like health IT) to show what you have learned. With children, AHRQ seems to focus on funding children's health research
generally without identifying what areas are most important. As a result, at the end of the day, they look
back through the research that has been done and try to create a cohesive whole
instead of creating a focus from the outset which would then result in a
cohesive story"
17. Appropriation History Table. February 1998. Rockville, MD: Agency for Health Care Policy and Research. http://www.ahrq.gov/about/cj1999/apphis99.htm; Justification for Budget Estimates for Appropriations Committees, Fiscal Year 2005. February 2004. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/about/cj2005/cjweb05.htm.
18. The
total number of activities in the publications analysis excludes those external
activities listed as conferences, meetings, workshops, trainings or centers
since these were not designed to produce publications in the peer-reviewed
literature.
19. AHCPR Strategic Plan. December 1998. Agency for
Health Care Policy and Research, Rockville, MD.
http://www.ahrq.gov/about/stratpln.htm
20. Fiscal
Year 2006 Agency for Healthcare Research and Quality Performance Budget
Submission for Congressional Justification
http://www.ahrq.gov/about/cj2006/cj2006.pdf
21. Strategic Plan: Children as a Priority
Population. December
1999. Agency for Healthcare Research and Quality, Rockville, MD.
http://www.ahrq.gov/child/CHStratPlan.htm
22. AHRQ Portfolios of Research. June 2005. Agency for Health Care Research and Quality, Rockville, MD.
http://www.ahrq.gov/fund/portfolio.htm
23. We note
that, while Lisa Simpson's support was certainly instrumental, it was not the
only factor in bringing child health activities to the fore: the efforts of
CHAG, as well as other Agency staff, and the emphasis on some aspects of child
health from the Secretary's level and the White House (e.g., SCHIP, childhood
asthma), and the presence of funding for investigator-initiated grants all
contributed as well.
Return to Contents
Proceed to Next Section
540 Gaither Road Rockville, MD 20850