Chapter 3: Getting Involved in the Research Process
The Agency for Healthcare Research and Quality (AHRQ) relies on stakeholder involvement to ensure that research is relevant to and useful for decisionmakers. Stakeholder involvement also increases transparency in the research process, which is critical for maintaining the scientific integrity and credibility of AHRQ's work. Finally, once research is produced, it is hoped that involved stakeholders are more likely to actively use and disseminate the information that they helped produce.
This chapter describes the different types of stakeholders that AHRQ works with and why they are important, and then outlines the opportunities for involvement in different Effective Health Care Program (EHC) activities and what that involvement entails. Not all types of stakeholders are involved in all stages of research; instead they are targeted where they can have the most impact. If you have been asked to join a specific project activity, you can use this chapter to find a step by step description of the project and what is expected of you. If you are looking for opportunities to be involved, the shaded boxes in each section explain the different opportunities and which types of stakeholders are needed.
Who Is a Stakeholder and Why Are Their Views Important?
AHRQ has defined a "stakeholder" as persons or groups who have a vested interest in the clinical decision and the evidence that supports that decision.
Stakeholders may be patients, caregivers, clinicians, researchers, advocacy groups, professional societies, businesses, policymakers, or others. Each group has a unique and valuable perspective.
| Stakeholders |
Stakeholders' Perspective |
| Patients, caregivers, and patient advocacy organizations |
It is vital that research answer the questions of greatest importance to those experiencing the situation that the research addresses. Which aspects of an illness are of most concern? Which features of a treatment make the most difference? Which kind of presentation of research results is easiest to understand and act upon? |
| Clinicians and clinical professional societies |
Clinicians are at the heart of medical decisionmaking. Where is lack of good data about diagnostic or treatment choices causing the most harm to patients? What information is needed to make better recommendations to patients? What evidence is required to support guidelines or practice pathways that would improve the quality of care? |
| Institutional health care providers, such as hospital systems and medical clinics |
Many health care decisions are structured by the choices of institutional health care providers, and institutional health care providers often have a broader view of what is causing problems. What information would support better decisions at an institutional level to improve health outcomes? |
| Government agencies |
The responsibility for the nation's health care is shared across several agencies. What research is or could be funded? |
| Purchasers of health care, such as employers and public and private insurers |
Coverage by public or private purchasers of health care plays a large role in shaping individual decisions about diagnostic and treatment choices. Where does unclear or conflicting evidence make the decision of what to pay for difficult? Where is new technology or new uses of technology raising questions about what is standard of care? |
| Health care industry representatives |
The manufacturers of treatments and devices often have unique information about their products. |
| Health care policymakers at the Federal, State and local levels |
Policymakers at all levels want to make health care decisions based on the best available evidence on what works well and what doesn't. Comparative effectiveness research can help decisionmakers plan public health programs, design health insurance coverage, and initiate wellness or advocacy programs that provide people with the best possible information about different health care treatment options. |
| Health care researchers and research institutions |
Researchers gather and analyze the evidence from multiple sources on currently available treatment options. |
Opportunities for Involvement
A. Evidence Need Identification
Identifying a need for evidence is the beginning of any comparative effectiveness research process. This presents an opportunity for stakeholders to have a significant impact by nominating a topic for research. Once a topic has been nominated, it is further clarified though a process of Topic Development, which may allow further opportunities for input from the nominator and possibly other stakeholders.
Topic Nomination
The EHC Program accepts topic nominations from all individuals or organizations. Nominations can be submitted by anyone using the Web site (http://www.effectivehealthcare.ahrq.gov) and clicking the "Submit a Suggestion for Research" tab. For those who do not have access to a computer or the internet, nominations may be submitted by mail (go to Appendix B). The nomination form requests information about the importance of the topic being proposed, patients affected by the issue, and specific questions that research could help answer.
Get Involved as a Nominator
Any individual or organization can nominate a topic at any time using the EHC Program Web site: http://www.effectivehealthcare.ahrq.gov. Nominations may be submitted anonymously, although if the nominator includes contact information, an EHC Program representative may follow-up with the nominator if there is a need to clarify or further develop the nomination.
The nomination process typically takes 3-4 months. The nominator is asked to:
- Describe three to five specific, well-defined questions related to the topic of interest (go to Appendix D for tips on developing strong research questions).
- Provide as much information as possible in order to guide the process for selecting which nominations will go forward for research.
- Be as specific as possible, particularly regarding the health care intervention or service of interest, the population of interest, and how you expect this research to affect healthcare. Providing more information initially will help guide the process for selecting research topics.
-
Include any supporting documentation with the nomination by mail or E-mail, or by uploading electronically through the Web site nomination form.
- Be available to answer questions or further clarify the nomination, if needed. This is known as "topic development." Clinical or other experts may also be consulted during this process.
- Consider serving in other roles if the topic goes forward for research (described below).
|
Topic Development
When nominations are under review the nominator may be asked to provide further information for clarification. Clinical or other experts may also be consulted to ensure the context of the nomination is accurately considered. This process of clarification is known as "topic development," and is conducted by one of the Evidence-based Practice Centers (EPCs). Once the topic nomination is fully "developed," it is discussed by the AHRQ Topic Selection Committee which decides whether the nominated topic is appropriate for an evidence synthesis, should be considered for evidence generation, or whether it will not be pursued at this time based on of the following questions:
- How widespread and serious is the disease or problem proposed for study?
- What are the costs associated with the disease and available treatments?
- How much controversy exists about treatment?
- What are the potential impacts for improving care and/or reducing costs?
- Would research results be relevant to Federal health care programs such as Medicare, Medicaid or the Children's Health Insurance Program (CHIP)?
An explanation of the Topic Selection Committee's decision is provided to the nominator and posted on the EHC Program Web site (http://www.effectivehealthcare.ahrq.gov). Anyone can check the status of a nomination at any time on the EHC Program Web site.
B. Evidence Synthesis
The Effective Health Care Program produces two types of evidence synthesis reports: Comparative Effectiveness Reviews and Technical Briefs. Both have opportunities for Stakeholder involvement, although each report follows a slightly different process as described below.
Comparative Effectiveness Reviews
Topic Refinement
If a nominated topic has been selected to move forward as a Comparative Effectiveness Review, AHRQ will assign the topic to one of the Evidence-based Practice Centers (EPCs) for topic refinement. Topic refinement is the process of clarifying the scope of a topic and defining the questions so that it is ready to undergo research. Refinement requires several steps:
Kick-Off Call
Once an EPC is assigned a topic for refinement, a kick-off call is scheduled for key staff from the EPC, AHRQ, Eisenberg Center, and the Scientific Resource Center (SRC) to organize and discuss the research plan. The kick-off call is facilitated by the EPC with guidance from a staff person assigned by AHRQ. This staff person is known as a "Task Order Officer" (TOO). The kick-off call is intended to help develop a common understanding of the task at hand, as well as establish agreement on the methods, plans, and timeline for completing the research. The nominator may be asked to participate in this call to help clarify the intent of the topic and to communicate important contextual information.
Developing the Key Questions
Good research requires a good set of research questions. The key research questions for each Comparative Effectiveness Review are formulated and refined with the help of key informants to ensure research addresses the questions important to decisionmakers, represents an accurate scope of issues, and produces the most valuable product. Key informants often include the nominator as well as other decisionmakers who can contribute to defining the scope and key questions of a research report. Key informants may include patients and caregivers, clinicians (including both generalists and experts in relevant specialties), representatives of relevant professional and consumer organizations insurers and health plan representatives that make coverage and benefit decisions, public policymakers, and others with experience in making health care decisions relevant to the topic. During the topic refinement process, the focus of the original nomination may be narrowed, expanded, or shifted depending on the input received from the key informants.
This input is gathered through key informant calls, which are scheduled and coordinated by the EPC assigned to do the research. There may be one or several calls held. The Guide for Key Informants, including roles and responsibilities, and what to expect is included in Appendix E.
Get Involved as a Key Informant
Key informants are stakeholders with direct experience with the topic being researched, as patients or caregivers, clinicians, policymakers, insurers, or other health care decisionmakers. Key informants offer unique perspectives that help to refine key questions before the research begins. They also provide context, as well as help direct questions for specific considerations such as side-effects, benefits, harms, and quality-of-life issues.
The EPC assigned to a research topic will invite approximately six to eight individuals to participate in the topic refinement process as key informants. The nominator of the topic will often be invited to participate to address the original intent of the nomination and to increase the likelihood that the end products will meet the originating need. All key informants must complete Disclosure of Interest forms, and may be asked to submit a short description of their experience with the topic. The requirement of disclosure bolsters transparency, assists in mitigating bias, and helps create a balance of perspectives among the key informant group.
Once key informants are approved by AHRQ to participate, the EPC is responsible for scheduling and coordinating conference calls and/or other opportunities for input. The number of calls or methods used to collect input will vary depending on the complexity of the topic. It is recommended that calls include as many of the identified key informants as possible to foster more robust discussions. In the event that this is not possible, it may be necessary to schedule individual calls, calls with subsets of the identified key informants, or use other methods for soliciting input from key informants.
The Topic Refinement process is expected to take 4 months. Key informants should expect to:
- Submit a completed Disclosure of Interest form. Conflict of Interest policy is available at Appendix F.
- Submit a brief description of their experience with the topic.
- Participate in at least one, and possibly several, phone calls with the EPC and other key informants. Typically, calls last 1 hour, and use a toll-free number. Often it is necessary to schedule multiple calls.
- Be available to answer questions if the research team requires additional information.
- Help guide the formation of key questions, which may involve the use of a PICO(TS) format. For more information on formulating questions using a PICO(TS) please see Appendix D.
- Have their names and contact information shared with the Effective Health Care Program for potential consultation on future work, unless otherwise requested.
Key informants who are also nominators should anticipate that the original nomination may be altered or changed during the topic refinement process, in order to ensure the greatest possible application and relevancy.
The Effective Health Care Program assists EPCs in identifying and supporting key informants in the topic refinement process. To indicate interest in participating as a key informant, contact the Effective Health Care Program at (301) 427-1502 or EffectiveHealthCare@ahrq.hhs.gov. |
Once key informants have provided input into the development of research questions, the EPC develops a draft set of key questions. The draft questions are then posted on the EHC Program Web site for public comment. The final key research questions, along with an analytic framework and research abstract, will guide the research process.
Get Involved by Providing Public Comments on the Key Questions
Anyone may comment on draft key questions, and the assigned EPC will consider incorporating feedback in the final key questions. It is critical that the questions posed by these documents reflect the concerns and dilemmas of consumers, clinicians, policymakers, and other health care decisionmakers. Public posting is another opportunity for involvement of the whole range of stakeholders and a way to ensure the broadest possible relevancy of the research report.
The Public Comment period lasts for 4 weeks. Anyone who wishes to comment on the Key Questions should expect to:
You can sign up at http://effectivehealthcare.ahrq.gov/index.cfm/join-the-email-list1/ to receive alerts when Key Questions are posted for clinical areas that interest you. |
Research Review
Developing a Comparative Effectiveness Review involves systematically reviewing the literature and analyzing the quality of existing studies and data. This process can take up to 12 months to complete, during which time scientific investigators thoroughly and methodically examine information. During the development of a report, there will not be opportunity for communication with the institution conducting the research. All communication regarding the topic at this phase must go through AHRQ in order to ensure that the investigators remain as objective as possible. The process of developing a research report, however, does include an opportunity for stakeholder involvement through Technical Expert Panels and during Peer and Public Review.
Technical Expert Panels
Technical Expert Panels provide expert advice about the clinical specialty being studied as well as about research methods. Therefore Technical Expert Panels are primarily comprised of clinical, research, and methodological experts who can provide information and guidance on technical aspects of the review as it is completed. Technical experts are assembled by the EPC assigned to the report. The AHRQ task order officer is available to help identify participants for Technical Expert Panels if needed.
Get Involved as Part of a Technical Expert Panel
Participation on Technical Expert Panels is usually limited to researchers, clinical experts, statisticians, and specialists who can help ensure the methodological rigor of the research report. Generally comprised of five to eight members, Technical Expert Panels help focus the literature search, identify inclusion/exclusion criteria, and assist in the evaluation of available evidence. The size and composition of the technical expert panel are intended to create a balance between content and methodological expertise. The assigned EPC is responsible for convening the Technical Expert Panel, with approval from the AHRQ task order officer.
The Research process is expected to take up to 12 months. Technical Experts should expect to:
- Submit a completed Disclosure of Interest form. Conflict of Interest policy available at Appendix F.
- Submit a brief description of their experience with the topic.
- Participate in at least one, and possibly several, phone calls with the EPC and other Technical Experts. Typically, calls last 1 hour, and use a toll-free number. Often it is necessary to schedule multiple calls.
- Be available to answer questions if the research team requires additional information.
- Have their names and contact information shared with the Effective Health Care Program for potential consultation on future work, unless otherwise requested.
|
Peer and Public Review
Once a draft of the research review has been completed by the EPC, a peer review panel is assembled to provide additional and technical review of the report. The peer review process is coordinated through the Scientific Resource Center.
Get Involved in the Peer Review Process
Research products undergo a peer review process to ensure scientific integrity and quality of research reports. AHRQ, the Scientific Resource Center, and the assigned EPC identify peer reviewers for specific topics. Decisionmaker organizations (such as professional societies) are encouraged to suggest experts to participate in peer review. Suggestions for peer reviewers can be made during the topic nomination, topic development, and topic refinement processes by contacting EffectiveHealthCare@ahrq.hhs.gov. The Scientific Resource Center coordinates the peer review process, which typically lasts 3 months. The EPC considers all peer review comments and modifies the final report as appropriate.
Peer reviewers should expect to:
- Complete their review of the draft research review within 4 weeks.
- Have their names and contact information shared with the SRC for potential consultation on future work unless otherwise requested.
|
The draft report undergoing peer review is posted simultaneously on the EHC Program Web site for public comment. An announcement is sent through the EHC Program listserv that the draft is available for comment. Reports are typically available online go to http://www.effectivehealthcare.ahrq.gov) and select "Join the E-mail List" in the lower left corner.
Get Involved by Providing Public Comments on the Draft Report
Anyone may comment on draft report, and the assigned EPC will consider incorporating feedback in the final key questions. It is critical that the questions posed by these documents reflect the concerns and dilemmas of consumers, clinicians, policymakers, and other health care decisionmakers. Public posting is another opportunity for involvement of the whole range of stakeholders and a way to ensure the broadest possible relevancy of the research report. Anyone who wishes to comment on the Key Questions should expect to:
You can sign up at http://effectivehealthcare.ahrq.gov/index.cfm/join-the-email-list1/ to receive alerts when draft reports are posted for clinical areas that interest you. |
After all public and peer review comments have been received, the final report is prepared. The process of responding to and addressing public comments can take up to three months. The final report is then posted on the EHC Program Web site at http://www.effectivehealthcare.ahrq.gov. A notice of availability is also sent to individuals and organizations who have signed up through the AHRQ LISTSERV® to receive announcements.
The Agency for Healthcare Research and Quality supports and is committed to the transparency of its review processes. Therefore, the following are posted on the EHC Program Web site within 3 months after a final report is posted: (1) all comments received from the public, and (2) all the responses made by the authors of a draft report to the public comments (i.e., the "disposition of comments"). Each comment will be listed with the name and affiliation of the commentator, if such information is provided. Public commentators are not required to provide their names or affiliations to submit suggestions or comments. Contact information will be used to communicate with commentators if there are questions about submitted comments.
Research Needs Development
The development of a research needs document is a relatively new phase for the Effective Health Care Program. The research needs document will be produced by the EPC preparing the main research report. After completing a research review, including identification of evidence gaps, the EPC will convene a group of stakeholders, including investigators, funders, and others to prioritize future research needs as they relate to the research topic. The results of these discussions and prioritization will be summarized in a separate research needs document.
Get Involved Identifying Needed Research
As this process is new, methods of involving stakeholders in the development of the research needs document are being tested. Research institutions will consult with decisionmakers regarding how and what type of research should be prioritized to meet the identified evidence gaps.
The role of a stakeholder at this point is to participate in discussions to describe and prioritize research needs.
Stakeholders involved in identifying research needs should expect to:
- Read and review portions of the research report.
- Review suggestions and draft language regarding the prioritization of research gaps and needs for additional evidence.
- Provide comments in individual conversations or in group settings, such as dedicated meetings or conference calls.
- Have the process take up to 2 months.
In a transparent and systematic formal process, all stakeholders, including clinicians, funding agencies, and researchers, consider the gaps identified in the systemic research reviews between available medical knowledge and the needs of clinical practice. Participants in the discussion include the researchers who worked on the individual review where the gap was first identified, stakeholders with interest in the topic, clinicians with particular expertise in the topic area, and agencies with funds for potential future research. Also involved are researchers with expertise in the clinical area and in study design, who can help identify evidence needs and develop new research projects based on the findings of the comparative effectiveness review. It is hoped that this process will help shape future research plans and set priorities for a national investment in new research.
Inputs to the evidence gap identification process include nominations and recommendations of stakeholders by groups like the Federal Coordinating Council for Comparative Effectiveness Research and the Institute of Medicine's project on Priority Setting for Comparative Effectiveness Research, as well as AHRQ's systematic review process. |
Select for Figure 2: Points of Stakeholder Engagement for Systematic Reviews.
Return to Contents
Proceed to Next Section
540 Gaither Road Rockville, MD 20850 