Your browser doesn't support JavaScript. Please upgrade to a modern browser or enable JavaScript in your existing browser.
Skip Navigation U.S. Department of Health and Human Services www.hhs.gov
Agency for Healthcare Research Quality www.ahrq.gov
AHRQ Home | Questions? | Contact Us | Site Map | What's New | Browse | Información en español | E-mail UpdatesE-mail Updates
www.ahrq.gov
Evaluation of AHRQ's Children's Health Activities

Chapter 2. Methods

Our methods consisted of four parts: 1) We developed a database of child health activities (including extramural research, intramural research, conferences, meetings, workshops, training grants, and centers); 2) We identified publications derived from and the funding associated with these activities; 3) We sorted the activities into categories and coded the resulting publications according to potential impact, determined the number of times the publication had been cited, and recorded the impact factor of the journal in which the publication had appeared; and 4) We identified case studies and interviewed individuals associated with these case studies along with key stakeholders.  We describe each of these in turn.

Developing the Database of Child Health Activities

There was no single database that contained all or only children's health activities for the entire time period under study.  As a consequence, we developed a strategy that used all available sources to assemble the comprehensive list.  This list is included as Appendix A.8

External activities

To identify external AHRQ funded activities related to children's health for the period of 1990 to 2005, we searched the following sources:

  1. AHRQ Web site, specific populations, child and adolescent health, 9
    1. Funded projects, New Starts 1996-2005 pages.10  Total of 361 external activities identified. 
    2. Conference and Workshops links.  12 activities identified, 1 of which also identified in New Starts—11 additional external activities.
    3. Child Health Insurance Research Initiative (CHIRI™).11  9 activities identified, 8 of which also identified in New Starts—1 additional external activity.
    4. State Children's Health Insurance Program (SCHIP) page.12  4 activities identified, none of which identified through New Starts—4 additional external activities.
    5. Information technology and children's healthcare (IT) page.13  8 activities identified, 6 of which identified through New Starts—2 additional external activities.
  2. GOLD (which covers 1999-2006), child subcategory.  Total of 141 activities identified.  6 were duplicates and 117 were identical to activities identified in search 1, leaving 18 additional external activities.
  3. GOLD external research abstracts using "child" or "adolescent" or "infant" or "pediatric" as keywords.  237 activities identified.  178 had already been identified through search 1 or 2, leaving 59 additional external activities. 

As the public sources did not have information regarding external activities prior to 1996, we had AHRQ search an internal database, Agency Management Information System (AMIS) using the keywords "adolescent" or "child" or "infant" or "pediatric."  This search identified 362 external activities for the period of 1990-2004.  247 activities had already been identified through searches 1-3, leaving 114 additional external activities.

After identifying a total of 570 unique external activities related to children's health, we applied the following criteria: 

  1. Include if:  i) The activity has as one of its primary focuses child or adolescent health; ii) The activity includes both children and adults in its target population and involves a topic that directly affects child or adolescent health (i.e., asthma or diabetes); or iii) The activity is related to pregnancy, pre-natal care or obstetrics.
  2. Exclude if:  i) The activity is related to parents of children under 18 but includes no direct study of impact on children; ii) The activity is only tangentially related to children's health (children mentioned in the abstract but not really a focus of the study or a part of the target population); iii) The activity is related to fertility; iv) The activity had nothing to do with children.

Initially, one researcher reviewed all 570 external activities using the foregoing criteria, identifying those activities to be excluded.  All the excluded external activities were then independently reviewed by another researcher using the same criteria.  The discrepancies between the two reviewers were resolved through a discussion with the entire project team. 

Using these criteria and methods, we excluded 56 activities as set forth in Table 1.  After the exclusions, we were left with 514 unique, includable external activities.  We grouped these activities into 6 different categories: external research (407 activities), contracts (8 activities), EPC related activities (18 activities), APA support (16 activities), conferences, meetings, workshops, trainings (60 activities), and centers (5 activities).

Intramural activities

The only source of information we could locate for activities by AHRQ staff was in GOLD, which is a relatively new database that covers the time period of 1999-2006.  Searching the AHRQ staff research abstracts using the keywords "child" or "adolescent" or "infant" or "pediatric," we identified a total of 27 unique activities.  Using the same criteria as applied to the external activities, we included 24 and excluded 3 (activities not substantially related to children's health (2); activities only tangentially related to children's health (1)). 

Return to Contents

Identifying Associated Publications

External activities

After identifying external activities, we searched for publications or reports related to the 222 activities initiated from 1996 to 2002.  Prior to 1996, our list of activities was not comprehensive as our only source of information was the AHRQ internal AMIS database (as opposed to 1996 and after, where we cross-referenced activities from several sources, including the AHRQ Web site, GOLD and AMIS).  We also did not search for publications from projects initiated after 2002, since it would be unrealistic to expect these projects to have published all potential manuscripts by the time of our study. 

For the external research activities funded during 1996-2002, we had originally proposed to search PubMed® by grant number.  During the course of this evaluation, however, we discovered that AHRQ grant numbers are not consistently reported in PubMed®.  As a result, we used a somewhat less precise method for identifying publications.  We searched PubMed® and Web of Science using PI name (last name and first letter of first name—smith j* [au]) and keywords from the activity description, restricting searches to the dates after which the activity was funded.  After publications were identified, we reviewed the publications for grant numbers.  If the publication had a grant number listed, it was included if it matched the activity grant number and the publication appeared from the abstract to be related to children's health, and it was excluded if it did not match the activity grant number.  For those publications that did not have a grant number listed, we compared the publication abstract with the activity description and included the publication if it seemed substantially related to the activity and children's health.  After the keyword search, we conducted an additional PubMed® search using the PI name combined with a generic AHRQ grant number search (smith j* [au] AND hs* [gr]).  If there were any additional publications that had a matching grant number and were related to children's health, they were included.

From the foregoing searches, we were unable to identify any publications for 46 of the 222 External Research Activities funded during 1996-2002.  For these activities, we worked with an AHRQ librarian who also searched, but she ultimately did not find any additional publications.

For the activities listed as contracts, Evidence-based Practice Centers (EPC) related activities, Ambulatory Pediatrics Association (APA) support, conference, meetings, workshops, training, and centers, we used the search engine on the AHRQ Web site to find relevant publications.  We searched by grant number (if available) and principal investigator to identify publications from these activities. 

Intramural activities

Given the relatively limited information we had about internal children's health activities, we felt it would be more productive and comprehensive to elicit relevant publications from AHRQ researchers.  The Senior Advisor on Child Health, Denise Dougherty, sent an e-mail to all AHRQ staff asking that they identify their publications relating to children's health. 

Within AHRQ, intramural research regarding children's health is generally done in either the Center for Financing, Access and Cost Trends (CFACT) or the Center for Delivery, Organization & Markets (CDOM).  Jessica Banthin, who works in CFACT, provided a list of child health publications for the period of 1990-2004 for CFACT researchers, which was compiled from a survey of CFACT researchers.  Additionally, two other CFACT researchers, Chad Meyerhoefer and Samuel Zuevekas, provided a list of their relevant publications, none of which had been included in the CFACT list.  From these sources, a total of 54 publications were identified. 

For CDOM, we only received a response from two researchers.  Anne Elixhauser identified 8 publications, 3 of which were also identified in the CFACT list.  Cindy Brach, the CDOM researcher who led the CHIRI™ project, referred us to the AHRQ CHIRI™ Web page for her publications related to the CHIRI™ project.  But the publications listed on which Ms. Brach was a co-author had already been identified through the external publication search.  To avoid double counting, we did not include them in the list of intramural publications.  In an attempt to locate additional CDOM publications, we spoke with Carol Stocks, who maintains a list of publications related to the Healthcare Utilization Project, but she stated that most of those publications were from external research projects.  And she was unable to identify any additional CDOM intramural research publications or sources.  We also reviewed the AHRQ Web page for publications by CDOM which identified an additional 17 publications, although we were told by AHRQ employees that the Web page was not comprehensive or completely up to date.  Denise Dougherty identified 9 additional publications.  Finally, the AHRQ librarian assisted with the search and identified an additional 92 internal publications. 

In sum, we identified a total of 177 publications resulting from intramural research on children's health:  51 from CFACT, 22 from CDOM, 3 jointly from CFACT and CDOM, 9 from Denise Dougherty and 92 from the AHRQ librarian. 

Return to Contents

Determining Funding

Once all of the relevant grant awards were identified, the funding detail for each of the grant awards was extracted from the RaDiUS Database14 using the award numbers to match the records.  Since the RaDiUS Database was created in FY 1993, it does not contain any information grant activities that occurred prior to that year.  And, because information on the grant awards made by the Federal Government in the final quarter of FY 2005 had not yet been officially released by the Federal Government at the time of this analysis, FY 2004 is the most recent year for which complete information on such activities is available in RaDiUS.  Working within these parameters, the federal funds obligated to the relevant grant awards from 1993 through 2004 were extracted by fiscal year from RaDiUS and provided to the team for analysis. 

As a cross-reference, we compared these numbers to the funding listed in the AMIS database.  In many cases, the numbers matched.  If the two numbers differed, we used the higher number as the best estimate of the total grant funding.  For those activities for which funding was not identified through RaDiUS or AMIS, we used the search engine on the AHRQ Web site to find information on grant awards.  We searched by grant number (if available) and principal investigator for mentions of grant awards and amounts. 

Return to Contents

Categorizing Activities and Coding Publications

Categorizing activities

We categorized each activity according to three different schemes:  AHRQ strategic goals, AHRQ children's health plan strategic goals and AHRQ portfolios of work.  In Chapter 3, we describe each categorization scheme and the distribution, within each, of activities, funding, and publications.  We categorized publications based on the activity from which they were derived.  Intramural publications were not assigned to categories because the list of publications was developed separately from the list of internal activities.  We were not able to link the internal publications identified through our sources to specific activities in a systematic way. 

Coding publications

We sorted publications according to a scheme based on Stryer's four types of impact. Stryer's four categories for outcomes research were:

  1. Studies that identify problems, generate hypotheses, establish the effectiveness of interventions, and develop new tools to explore these problems.
  2. Studies in which a policy or program is created as a result of the research.
  3. Studies in which there is a documented change in what clinicians or patients do.
  4. Studies that result in changes in health outcomes.

We modified these categories by broadening the inclusion criteria to include studies that had potential for impact, in addition to studies showing actual impact, for two reasons.  First, AHRQ's external research on children's health includes types besides outcomes research, which was the focus of the Stryer analysis.  Second, as most policy researchers and evaluators of research programs understand, there is not a direct and linear relationship between a particular research finding and a particular policy or clinical change.  Our categories and their description are as follows:

  1. Research Findings:  Studies that identify or describe problems, generate hypotheses, present frameworks or conceptual models, or develop tools for methods or measurement.
  2. Policy Impact:  Research with clear policy implications or that examines the effects of policy or policy changes.  This research goes beyond describing a problem that could be amenable to policy; rather, it shows the implications of a specific policy, shows differences between policy choices, or evaluates the consequences of policy.  Research showing that uninsured children have less access to care would be category 1, whereas research showing that SCHIP enrollees got more access to care compared to the year prior to enrollment or research comparing access to care for children enrolled in SCHIP from different States would be category 2.
  3. Clinical Impact:  Research describing interventions that influence practice or that are aimed at influencing clinical practice (measuring or improving practice); or research developing or demonstrating tools for clinical use.  This research goes beyond describing problems in clinical practice or the development of tools for clinical practice.  Rather, it evaluates clinical behavior, demonstrates changes in clinical behavior, or demonstrates the use of tools in a clinical setting.  This research addresses how to get clinicians to adhere to evidence-based practice.  Research describing quality measures would be in category 1, whereas research comparing providers or health care systems with these quality measures, or showing how particular interventions affected scores on the measures would be category 3.
  4. Outcomes Impact:  Research determining which clinical or health behaviors affect health outcomes, describing the effect of an intervention on health outcomes, or aimed at influencing health outcomes.  Research describing measures of health status would be category 1.  Research evaluating an intervention designed to improve health status would be category 3 if there was no effect of the intervention and category 4 if the intervention showed an effect. Research establishing a link between a policy or practice and outcomes is category 4.

Publications were categorized based on their titles and abstracts.  We did not view the full text of the articles.  Publications for which an abstract was not available in PubMed® were not categorized.15

In judging publications, we privileged the results, rather than the implications of the research.  That is, a publication describing problematic attitudes of clinicians towards medication error reporting would be category 1 despite the clear implications that a) policies encouraging reporting are necessary and b) medical errors impact both clinical practice and health outcomes. In order to be category 2, the publication would have to examine, for example, the impact of a no-fault reporting policy on number of error reports.  To be category 3, the publication could examine the effect of a tool to flag possible drug interactions on the number of prescriptions corrected.  To be category 4, the publication would, for example, show the relationship between medical errors and mortality or morbidity or how an error-reduction intervention shortened length of stay or reduced complication rates.  Two researchers coded the publications independently, with discrepancies resolved via consensus.

We also recorded bibliometric characteristics.  We recorded the number of times each publication had been cited, as reported in ISI Web of Science's Science Citation Index-Expanded and Social Science Citation Index.  We did not correct for self-citation.  We recorded the impact factor of the journals in which the articles had been published.  The journal impact factor is the average number of times articles from the journal published in the past two years have been cited in the Journal Citation Reports year.  The impact factor is calculated by dividing the number of citations in the current year by the total number of articles published in the two previous years.  An impact factor of 1.0 means that, on average, the articles published one or two years ago have been cited one time.  An impact factor of 2.0 means that, on average, the articles published one or two year ago were cited two times. 

Because publications were assigned to categories within the categorization schemes based on the activities from which they were derived, we did not categorize internal publications according to these categorization schemes.  Therefore, descriptive analyses for the Stryer categories, citations, and impact factor included the intramural publications only for the overall analysis.  We excluded intramural publications when examining Stryer categories, citations, and impact factors by categorization scheme.

Return to Contents

Case Studies and Key Informant Interviews

We used qualitative methods to better understand the processes by which ARHQ and AHRQ personnel contribute to impact, the perspectives of other stakeholders, and to understand the impact of other children's health activities.  We developed case studies of high impact child health activities through reviewing existing documentation and interviewing individuals involved in these cases.  We supplemented these case studies with key stakeholder interviews of AHRQ staff and others to assess the impact of AHRQ's research products and other children's health activities from the perspective of AHRQ staff, the users of these products and other interested parties.

As mentioned above, these case studies were designed to gain in-depth understanding of high-impact activities (or groups of activities) and how this impact was achieved.

Key stakeholder interviews were designed to elicit the views of individuals who may or may not have been directly involved in the impact cases, but whose perspective is of particular interest, either because of their experiences with AHRQ, their current or former roles, or their ability to represent the perspective of an important group of stakeholders.

Potential cases were identified by the research team through interviews with AHRQ staff, the Impact Case Studies Program,16 and the data we collected on projects funded and manuscripts produced.  We originally identified three case studies: State Children's Health Insurance Program (SCHIP)/Children's Health Insurance Research Initiative (CHIRI™); Asthma and attention-deficit hyperactivity disorder (ADHD): Moving evidence to practice; and Quality Improvement.  However, on further consultation with the TOO and the Senior Advisor on Child Health, it was determined that the second two case studies could be thought of in combination.  Thus, our cases studies were SCHIP/CHIRI™ and Translating Research Into Practice (TRIP)/Quality Improvement in asthma and ADHD.  For the case studies, we considered a body of work (including externally funded research grants, intramural research, conferences, tools, evidence-based reviews, partnering with other organizations, and dissemination) rather than an individual research project to be the 'case.'

We developed a semi-structured interview protocol to serve both the case studies and the key stakeholder interviews.  The interview protocol was designed to assess a) whether and to what extent AHRQ's children's health activities (research, meetings, conference support, products, tools, etc.) had an impact on policy, clinical processes, or health care outcomes; b) Which processes influenced these activities' impact; c) The ways in which AHRQ staff contributed to the impact of these activities; and d) The ways in which structural or organizational characteristics of AHRQ contributed to the impact of these activities.  Additionally, interviews with AHRQ staff included the following topics: how AHRQ transmits/communicates its evidence-based, quality/safety philosophy and whether outside entities (including other federal agencies) respond or incorporate AHRQ's work; how much the Agency is asked for advice/input; how much that advice/input is valued; and how much support the AHRQ CHAG and others working on child health issues full or part-time get to focus on child health.  The interview protocol was finalized in consultation with the TOO and the Senior Advisor on Children's Health. Interview topics, organized by Primary Objective, are shown in Table 2.  The interview protocol is attached as Appendix B—the same protocol, suitably modified, was used for both external stakeholders and AHRQ staff.

Key stakeholders were selected in consultation with the TOO and the Senior Advisor on Children's Health, from the following groups: AHRQ staff, clinical stakeholders (such as members of the Children's Health Accountability Initiative (CHAI), the National Initiative for Children's Health Care Quality (NICHQ), and the American Academy of Pediatrics (AAP)), policy stakeholders such as the National Association of Children's Hospitals and Related Institutions (NACHRI) and others, patient/family stakeholders (such as Family Voices, Parent to Parent), and other funders traditionally interested in children's health (such as the David and Lucille Packard Foundation, the Commonwealth Fund, the Robert Wood Johnson Foundation). All those contacted consented to be interviewed.  Given that the interviews were semi-structured and adapted to the particular case or stakeholder, we did not ask the same questions of more than 9 individuals.

Data for both the key stakeholder interviews and the in-depth case study were in the form of interview notes.  These were summarized in text matrices across groups where comparable questions are asked.  From these raw data matrices, we identified emerging topics and themes.  Reports describing qualitative findings focus on content themes and utilize representative respondent quotes to illustrate key findings.

Return to Contents

Data Analysis

We performed descriptive analyses to address Primary Objectives 1 and 3.  For Primary Objective 2, we performed both descriptive and inferential analyses.  We used One-Way ANOVAS to assess the statistical significance of mean Stryer codes, numbers of citations, and impact scores across the various categorization schemes.

8. This list does not include the liaison and other activities of AHRQ staff and contract work reflected in products on the AHRQ Web site.

9. http://www.ahrq.gov/child/

10. http://www.ahrq.gov/child/fundprj.htm

11. http://www.ahrq.gov/chiri/

12. http://www.ahrq.gov/child/schip.htm

13. http://www.ahrq.gov/child/itchild.htm

14. https://radius.rand.org

15. We identified a total of 627 publications from external activities funded from 1996-2002. Of these, 601 had an abstract available from PubMed® or Web of Science.

16. http://www.ahrq.gov/research/jan02/0102RA22.htm

Return to Contents
Proceed to Next Section

 
AHRQ Advancing Excellence in Health Care