Chapter 1. Introduction

The Agency for Healthcare Research and Quality (AHRQ) has, since its inception, funded research and other activities concerning a variety of children's health issues.  At the same time, the Agency's strategic priorities continue to evolve in the context of a changing policy environment.  Currently, the Agency is crafting strategic plans for its ten "portfolios of work."  As part of the planning process, the Agency is evaluating the various cross-cutting, or cross-portfolio, priority populations, including children.  To aid in this assessment, the RAND Corporation was contracted to evaluate AHRQ's child health activities. 

As requested by AHRQ, this evaluation addresses four Primary Objectives: 

1. Measure and assess to what extent the Agency contributed and disseminated and/or translated new knowledge.
2. Measure and assess to what extent AHRQ's children's healthcare activities improved clinical practice and health care outcomes and influenced heath care policies.
3. Measure and assess AHRQ's financial and staff support for children's health research.
4. Measure and assess to what extent the Agency succeeded in involving children's health care stakeholders and/or creating partnerships to fund and disseminate key child health activities.

To address the RFTO's primary purposes, RAND developed and implemented a conceptual framework based on state-of-the-art research evaluation theories (Roessner, 2002).¹  This framework allowed an assessment of the Agency's contribution to the development and dissemination of new knowledge and the impact of that knowledge on policies, clinical practice, and health outcomes.  In addition, we used tools and methods developed at RAND to describe the Agency's funding focus and financial investment in children's health over time.  The report will be helpful in determining the extent to which Government Performance Results Act (GPRA) goals were achieved and in fulfilling GPRA/OPART requirements.

Return to Contents

Policy Context

AHRQ's mission, strategic goals, and organization

Established in 1989 as the Agency for Health Care Policy and Research and reauthorized in 1999 as the Agency for Healthcare Research and Quality, AHRQ has a mission of improving the quality, safety, efficiency, and effectiveness of health care for all Americans.  AHRQ regards its customers as clinical, healthcare system, and public policy decision makers.  Its strategic plan, adopted in 1998, includes the following goals to reflect the needs of its customers:

• Support improvements in health outcomes.
• Strengthen quality measurement and improvement.
• Identify strategies that improve access, foster appropriate use, and reduce unnecessary expenditures.²

AHRQ's priorities and organization have changed over time.  Currently its research and other activities are organized into ten "portfolios of work" such as health care management, data development, and health information technology.  Cutting across these portfolios, AHRQ has also identified, as per the Agency's authorizing language, priority populations, including minorities, women, and children. 

AHRQ's children's health research program

There are systematic differences between children and adults that strongly suggest the need for child-focused models for health services.  The differences have been summarized in an analysis supported by AHRQ (Forrest, Simpson et al. 1997) as the four D's: development, dependence, differential epidemiology, and demographics.  A fifth D, disparities, has also been proposed (Gidwani, Sobo et al. 2003).  Children develop at a rapid pace and their development in one domain impacts development in others.  Children are dependent on families, schools, and other social and political institutions for their welfare in general and for access to health care.  Most children are healthy, and in contrast to adult epidemiology in which a few high-prevalence conditions account for the majority of morbidity and mortality, pediatric epidemiology is characterized by a relatively large number of low-prevalence conditions.  Demographically, U.S. children as a group are disproportionately poor, of color, and uninsured; as a result, health disparities are a major concern.  Because of the differences between children and adults, and the frequent focus of health services and clinical research on adults, the knowledge base for children's health is considerably less robust than that for adults.

Another important point of consideration is that linking "downstream" health outcomes to preventive and other child health services is much more difficult than it is for adult health services—because children are inherently more resilient and healthy and the lag times are longer and intervening confounders are more influential, among other reasons.  As a result, a major challenge for children's health services research is the need to establish unequivocally the value of health care for children as an effective use of resources.

In this context, AHRQ's vision for children's health is "to establish the Agency for Healthcare Research and Quality as one of the Nation's key public organizations for improving the quality and effectiveness of health care services to the Nation's children and adolescents." This mission is consistent with the Agency's overall mission: "to improve the outcomes and quality of health care, reduce its costs, address patient safety, and broaden access to effective services, through the establishment of a broad base of scientific research and through the promotion of improvements in clinical and health system practices, including the prevention of diseases and other health conditions."³  Policy advice and coordination of AHRQ child health activities are provided by a Children's Health Advisory Group (CHAG), which is composed of representatives of AHRQ offices and centers.

Supporting this mission are six specific objectives for children's health research:

1. Contribute to new knowledge about child health services.
2. Create tools and nourish talent to strengthen the knowledge base in child health services.
3. Translate new knowledge into practice.
4. Improve communication with stakeholders in child health.
5. Include children and child health care in all AHRQ-supported research, as scientifically and ethically appropriate.
6. Balance the ARHQ research portfolio to represent a broader range of children and child health care.

The first three areas follow the Agency's general research mission.  The second three are specific to child health and cut across the research cycle and the Agency's three strategic goals⁴  The rest of the strategic plan outlines these goals and their related objectives in greater detail.  The strategic plan ends with an overview of partnerships, a significant element for achieving AHRQ's goals in child health (AHRQ Strategic Plan, 1999). 

In addition to funding extramural and intramural research, AHRQ children's health activities include the development of child-specific databases (e.g. the HCUP Kids Inpatient Database), tools for measuring quality, evidence reports, liaison with and contributions to other entities both within and beyond DHHS, a list-serve, and the child health scholar program. 

Context

An evaluation of AHRQ's children's health activities must be placed within the appropriate context.  In this vein, we note that AHRQ was originally conceived by Congress as a mechanism to reduce the Medicare budget—so child health has been an evolving adjunct to the Agency's basic raison d'etre.  Far fewer federal resources are available for children's health care, and consequently there are fewer efforts regarding quality, safety, efficiency, and effectiveness of child health care.  When it comes to quality and safety, the focus in HHS has been on improving quality and increasing efficiency for the Medicare population, and on HIT as a mechanism for quality improvement.  And Congress has increasingly prescribed specific funding initiatives for the Agency, such as HIT and patient safety—areas in which there are more adult- and elderly-focused researchers working versus child-focused researchers.

Program evaluation and performance review

The AHRQ approach to funding research, development, and demonstration projects is based on the concept of "the research pipeline" in which funded activities collectively build the infrastructure, tools, and knowledge base necessary for practice improvements.  This pipeline brings together a mix of projects covering important aspects of children's health research and is fueled through a cycle of iterative research involving the steps of needs assessment, knowledge generation, dissemination of knowledge to practitioners, evaluation of field experience and outcomes from the new techniques, and feedback to start a new cycle (AHRQ, 2002).

AHRQ includes measures of its performance in improving child health care as part of its own performance measurement strategy.  In 2003, the Agency revised its Government Performance Results Act (GPRA) framework for research on health care costs, quality, and outcomes to address three strategic goals:  1) To have measurable improvements in the quality, safety, and outcomes of healthcare for Americans; 2) To develop the evidence base for policymakers and health systems to use in making decisions about what services to pay for, how to structure these services, and how these services are accessed; and 3) To build capacity for improving the quality of health care delivery through research and training.  Indicators of AHRQ's performance in meeting the first goal include a 5 percent reduction in the rate of hospitalizations for pediatric asthma in persons under age 18 and a 5 percent reduction in the number of premature babies who develop Respiratory Distress Syndrome.⁵

In 2002, the Office of Management and Budget developed the Program Assessment Rating Tool (PART) to implement the GPRA process.  This 30-item questionnaire is to be used to assess the quality of every federal program's design, goals, management and results to determine effectiveness.  Over a five-year period, OMB expects to assess every federal program to ensure each is establishing an aggressive action plan for improvement.  The PART evaluation covers four critical areas of assessment—purpose and design, strategic planning, management, and results and accountability.  The first set of questions gauges whether the program's design and purpose are clear and defensible.  The second section involves strategic planning, and weighs whether the agency sets valid annual and long-term goals for programs.  The third section rates agency management of programs, including financial oversight and program improvement efforts.  The fourth set of questions focuses on results that programs can report with accuracy and consistency.⁶  Although not designed to answer these questions directly, this evaluation should provide information useful to the Agency in preparing future PART assessments.

Return to Contents

Evaluation Framework

"What we really want to get at is not how many reports have been done, but how many people's lives are being bettered by what has been accomplished. In other words, is it being used, is it being followed, is it actually being given to patients?... [W]hat effect is it having on people?"

Congressman John Porter, 1998, Chairman, House Appropriations Subcommittee on Labor, HHS, and Education

The primary objectives of the evaluation, as specified in the RFTO were to:

1. Measure and assess to what extent the Agency contributed new knowledge as a result of its funding of children's health research (extramural and intramural) and disseminated and/or translated effectively its findings to meet AHRQ's strategic objectives of improving the safety, quality, effectiveness, and efficiency of health care as well as wider DHHS strategic objectives.
2. Measure and assess to what extent AHRQ's children's health care activities, i.e., its research findings, meetings, conference support, products, tools, etc., improved clinical practice and health care outcomes and influenced heath care policies over the past fifteen years.
3. Measure and assess AHRQ's financial and staff support for children's health research as well as Agency internal handling of children's health grants, contracts and intramural activities research with/among other AHRQ programs, portfolios and activities and other DHHS and federal agency efforts.
4. Measure and assess to what extent the Agency succeeded in involving children's health care stakeholders and/or creating partnerships to fund and disseminate key child health activities.

Evaluating the impact of research programs is challenging.⁷  Research programs are characterized by high levels of uncertainty, long time horizons, and indirect and complex causal paths between inputs and impact.  Further, the research funder plays only one of the many roles (e.g. knowledge developer, users, other stakeholders) in the collective that produces knowledge and improvement.  In order to measure and assess the success of a research program, one must ask what program performance means under these circumstances, and how it might be measured.  As 'the Porter Question' above, indicates, a key measure of the impact of a research program is the "effect [it is] having on people."

Numerous conceptual models for evaluating research have been developed. RAND drew from these models, as well as from our experience with relevant research, demonstrations, and evaluations, to develop a conceptual framework to structure our design for the proposed evaluation.

One way to evaluate a research portfolio is by describing the mix of investments over time, in terms of grants funded, and the scientific output of this investment in the form of articles in the peer-reviewed literature.  This approach yields an overview of the research portfolio in terms of inputs and outputs, and is useful for evaluating the degree to which a research portfolio matches strategic plans.  Accordingly, we examined research activities and publications over time and according to AHRQ strategic plans.

A more detailed analysis, however, is also possible.  That is, research findings (in the form of articles in the peer-reviewed literature) can be categorized according to their potential impact in various spheres.  An example of such a categorization scheme is that developed by AHRQ staff to assess the impact of AHRQ's funding for outcomes and effectiveness research (Stryer, Tunis et al. 2000).  According to this framework, publications derived from AHRQ-sponsored activities may have four possible types of impact: those that are contributions to the health care knowledge base that result from foundational studies to identify problems, generate hypotheses, establish the effectiveness of interventions and develop new tools to explore these problems; those in which a policy or program was created or modified as a result of the finding; those in which patterns of care or clinician or patient behavior changed as a result of a finding; or those that improve health care outcomes.  With this in mind, our approach includes an analysis of the potential impacts of publications derived from AHRQ-sponsored activities.

A second way of evaluating a research portfolio is based on the idea that the distribution of the impact of research activities (or in this case, ARHQ-sponsored child health activities) is highly skewed.  That is, only a small fraction of child health activities produce important research findings or result in demonstrable changes in policies, clinical practice, or health care outcomes.  On the assumption that an examination of such key child health activities can generate rich detail about the processes by which activities contribute to the realization of a variety of valued outcomes and impacts, we have identified a small subset of high-impact activities (individual activities or groups of related activities) and examined them in-depth.  Thus, the second component of our conceptual framework is to develop in-depth case studies of high-impact activities to describe the factors most important in bringing about these impacts.

Finally, given that a wide range of actors—including funders, scientists, policymakers, clinicians, and families—are involved in translating research findings into practice, it is imperative to consider the points of view of a variety of relevant stakeholders when assessing the impact of AHRQ's children's health care activities on policies, clinical practice, or health care outcomes.  Additionally, it is important to understand the ways that AHRQ partners with other stakeholders.  Thus, the third main component of our conceptual framework is to use key stakeholder interviews to assess the impact of AHRQ's research products and other children's health activities from the perspective of the users of these products and other interested parties.

Given this conceptual framework and the primary purposes of the evaluation as specified by the RFTO, we approached the evaluation in the following manner.  To address evaluation objectives 1 and 3, we used existing documentation to describe the full range of children's health research and activities that the Agency funded, and bibliometric techniques to describe the development of new knowledge and its dissemination through publication in peer-reviewed journals.  We supplemented this with key stakeholder interviews of AHRQ staff and others.  To address evaluation objective 2, we used an approach similar to Stryer and colleagues (2000) to categorize the products of funded research projects (articles in peer-reviewed journals) according to the four levels of impact.  We also went beyond these data, identifying high-value products (through final grant reports, review of documents from the Impact Case Studies Program, and key informant interviews) and studied these cases in-depth by interviewing the PI, Agency staff, and other stakeholders such as policy makers, clinicians, or patients.  High-value products identified in conjunction with the Task Order Officer (TOO) and the Senior Advisor on Child Health represent activities or groups of activities that were judged by a variety of stakeholders to be 'successful.'  We constructed these interviews to enable us to determine how the research findings led to improvements and what role(s) AHRQ played in this process.  To address evaluation objective 4, we used key stakeholder interviews to gather a variety of perspectives on AHRQ's efforts in partnering with others around children's health.

Return to Contents

Structure of this Report

In Chapter 2, we describe the methods used to develop the database of activities and publications as well as the case studies and key informant interviews.  In Chapter 3, we present the results of our analyses, organized by primary objective.  In Chapter 4, we conclude with a summary of findings and discussion. 

---

^1. Although we drew from theories about evaluating research, we recognize that the current evaluation concerns other AHRQ activities as well as research.

^2. http://www.ahrq.gov/about/stratpln.htm

^3. http://www.ahrq.gov/about/cj2003/cjweb03.htm#mission

^4. We note that the Agency's strategic plan, both overall and for children, has not been updated to reflect its new mission statement.

^5. http://www.ahrq.gov/about/gpra2004/gpra2004.pdf

^6. http://www.gpoaccess.gov/usbudget/fy04/pdf/budget/performance.pdf

^7. We note that while our charge was to examine child health activities broadly speaking, AHRQ is primarily a research organization and, thus, the framework of evaluating research programs applies here.

---

Return to Contents
Proceed to Next Section