Current Validity of AHRQ Clinical Practice Guidelines

Prepared for:
Agency for Healthcare Research and Quality
U.S. Department of Health and Human Services
2101 East Jefferson Street
Rockville, MD 20852

Prepared by:
Southern California Evidence-based Practice Center based at RAND, Santa Monica, CA
Paul Shekelle, M.D., Ph.D.
Center Director / Project Director
Sally Morton, Ph.D.
Center Co-Director

Eduardo Ortiz, M.D., M.P.H.
Martin Eccles, M.D.
Jeremy Grimshaw, M.D.
Steven Woolf, M.D., M.P.H.
Investigators

This document is in the public domain and may used and reprinted without permission except those copyrighted materials noted, for which further reproduction is prohibited without specific permission of the copyright holders.

Ortiz O, Eccles, M, Grimshaw J, et al. Current validity of clinical practice guidelines. Technical Review 6 (Contract No. 290-97-0001 to the Southern California Evidence-based Practice Center at RAND). AHRQ Publication No. 02-0035. Rockville (MD): Agency for Healthcare Research and Quality. September 2002.

Practice guidelines can play an important role in improving the quality of health care by ensuring that practitioners and patients base their decisions about appropriate health care for specific clinical circumstances on systematically developed evidence. During the first half of the 1990s, the Agency for Health Care Policy and Research (AHCPR), the predecessor of the Agency for Healthcare Research and Quality (AHRQ), prepared a series of clinical practice guidelines for use by the health care community and patients. Subsequently, AHRQ decided to focus instead on the preparation of evidence-based practice reports, evaluating the science base on topics that could serve in the creation of clinical practice guidelines by professional societies and other organizations.

AHRQ, through its Evidence-based Practice Centers (EPCs), sponsors the development of evidence reports and technology assessments to assist public- and private-sector organizations in their efforts to improve the quality of health care in the United States. The reports and assessments provide organizations with comprehensive, science-based information on common, costly medical conditions and new health care technologies. The EPCs systematically review the relevant scientific literature on topics assigned to them by AHRQ and conduct additional analyses when appropriate prior to developing their reports and assessments.

At present, clinical practice guidelines -- which often draw on AHRQ's EPC reports -- are made available through the National Guideline Clearinghouse, sponsored by AHRQ in partnership with the American Medical Association and the American Association of Health Plans. The guidelines in the Clearinghouse are considered current if less than 5 years old and representative of the scientific evidence in the field. Once AHRQ stopped developing guidelines, the existing agency guidelines were in danger of becoming obsolete as new evidence accumulated. The review published herein was commissioned in early 1999 to determine to what degree the earlier AHRQ clinical practice guidelines remained scientifically current. As a result of the EPC's findings, AHRQ made the decision to take all but three guidelines out of active circulation, and to encourage other organizations to develop guidelines addressing the same issues. Because of the overwhelming popularity of these guidelines in the ! clinical community, and the presence within each guideline of substantial content that was independent of clinical advances, AHRQ has archived the guidelines for historical reference. All of these guidelines are accessible through the AHRQ Web site (http://www.ahrq.gov).

The Agency expects that the technical reviews such as this one will inform the health care community by providing important information concerning the processes used to improve the quality of evidence-based research and its applications.

We welcome written comments on this technical review. They may be sent to: Director, Center for Practice and Technology Assessment, Agency for Healthcare Research and Quality, 6010 Executive Blvd., Suite 300, Rockville, MD 20852.

To assess the current validity of 17 clinical practice guidelines developed under the auspices of the Agency for Health Care Policy and Research (AHCPR), now renamed the Agency for Healthcare Research and Quality (AHRQ)

For each of the guideline topics, we conducted literature searches of MEDLINE and Embase, limited to articles designated as guidelines, editorials, or reviews, and further restricted for some topics to particular key general medical and specialty journals. Additionally, we conducted a survey of members of the original AHCPR guideline expert panel to identify new evidence relevant to the validity of the existing guideline statements.

Identified evidence was used to assess the current validity of individual practice guideline statements. These results were used to classify each guideline into one of the following categories:

One hundred twenty-one of 170 (71%) surveys sent to experts were returned. Limited literature searches identified 6,994 titles, of which 181 potentially relevant articles were retrieved and reviewed. With the addition of evidence suggested by the experts, we reviewed 208 articles. Based on these data, seven guidelines were judged to be sufficiently obsolete as to warrant withdrawal (Depression in Primary Care vols. 1 & 2, Cataract in Adults, Sickle Cell Disease, Benign Prostatic Hyperplasia, Unstable Angina, Heart Failure, and Otitis Media with Effusion), three guidelines were judged to be still valid and retained (Pressure Ulcers in Adults, Treatment of Pressure Ulcers, and Cardiac Rehabilitation), and six guidelines were judged to be worth retaining but relevant new information for clinicians needed to be appended (Acute Pain Management, Urinary Incontinence in Adults, Management of Cancer Pain, Acute Low Back Problems in Adults, Post-Stroke Rehabilitation, and Reco! gnition and Initial Assessment of Alzheimer's Disease and Related Dementias).

Almost half of the existing AHRQ practice guidelines should be withdrawn, and an additional six guidelines should have additional information appended. Three guidelines are judged sufficiently valid to remain in circulation.

One of the original missions of the Agency for Health Care Policy and Research (AHCPR) was to facilitate the development of clinical practice guidelines. The Agency's Clinical Practice Guideline Program ran from 1990 to1996 and was perceived to have advanced significantly the science of practice guideline development and to have created guidelines that represented "state of the art" management for the selected conditions. It has now been five to ten years since the various clinical practice guidelines were developed, and some of these guidelines may no longer represent the best in clinical practice. Thus, the Agency (now renamed the Agency for Healthcare Research and Quality, AHRQ) recently asked the Southern California Evidence-based Practice Center to conduct a systematic assessment of the current validity of the Agency's existing clinical practice guidelines and make recommendations about which of those guidelines should be withdrawn from circulation.

We first developed a conceptual model for evaluating when guidelines may need to be updated or withdrawn. Based on consultation with several experts, we propose that there are six situations, listed below, that may require a guideline to be updated (or withdrawn) relating to changes in evidence, changes in the values placed on evidence, changes in resources available for health care, and changes in performance.

New preventive, diagnostic, or treatment interventions may have emerged to complement or supercede other interventions.
New evidence may require updating of the estimates of benefits and harms for existing interventions.
New evidence may identify as important outcomes that were previously unappreciated or wholly unrecognized.
Evidence that current practice is optimal may change.
The values that individuals or society place on different outcomes may change over time.
The resources available for health care may change significantly.

The process of assessing whether there have been sufficient changes in any of these factors to warrant updating a guideline includes two stages: (1) identifying significant new evidence, and (2) assessing whether the new evidence warrants updating or withdrawal. Ideally, the most thorough way to identify significant new evidence would be to conduct a new systematic review (including a search for evidence about current performance), but such a process would be costly and time consuming. Therefore, our model uses the combination of limited literature searching and the guidance of experts from relevant disciplines as a more pragmatic way to help identify potentially significant new evidence (including current performance). We reason that evidence sufficient to invalidate an existing practice guideline would, in general, be of a magnitude that it is known to experts in the field or has been published as significant articles in major general interest or specialty medical journ! als. Questioning experts from relevant disciplines about possible new evidence, accompanied by a search of the major relevant medical journals, should then suffice to identify the presence or absence of important new evidence.

We implemented this process by identifying individual statements within each guideline under review. We then asked the original guideline panel chair(s) to identify those panel members who are knowledgeable about particular guideline statements and other relevant clinical experts. The panelists so identified were sent a survey with the specific guideline statements, and asked to answer two major questions. First, were they aware of new evidence or developments in the field relevant to the guideline statement? If the answer was yes, then the expert was asked whether the evidence was sufficient to invalidate the guideline statement. This judgment of "sufficiency" was based on the criteria presented above (new interventions, new data on benefits and harms, new outcomes, or evidence that the guideline is no longer needed). The expert was asked to provide supporting references for any new evidence. The second question was whether a need has arisen to address new issues in the ! guideline that were not covered previously, assuming that the issues fall within the original scope of the guideline. If the answer was yes, then the clinical experts were asked to specify the issues.

Supplementing this process were limited literature searches which were focused and expedient starting from the point in time when the literature search for the original guideline panel ended. For most guidelines the searches were restricted to the major general interest and important specialty medical journals. One or two physicians trained in evidence-based medicine reviewed the literature searches. Titles, abstracts and articles were sequentially reviewed, seeking new evidence regarding the guideline statements. A second search was performed for newly published guidelines on the same topics by searching the U.S. National Guidelines Clearinghouse (www.guidelines.gov) and other sources, including ten government agencies. Finally, we identified guidelines from professional societies, disease societies, medical organizations, medical associations, and patient advocacy groups.

The process described above identified evidence on the current validity of each individual statement within each guideline. In order to make a judgment about retaining or withdrawing the entire guideline, we reviewed all of the evidence for each entire guideline. We assigned guidelines into one of the following categories:

We received replies from the original panel chair(s) for 15 of the 17 clinical practice guidelines. For one of the two practice guidelines for which we did not have a response, we contacted the Evidence-based Practice Center that had participated in the review of the literature for a recent "guideline update" process, and asked the lead member of the task order team complete the survey concerning validity. For the remaining clinical practice guideline, we used information from one of the members of the guideline development team to determine of which statements to send to which experts.

We sent out the individual guideline statements to a total of 175 experts. Five surveys were returned with the notation that the panelist was no longer at the given address and no forwarding address was known. Of the remaining 170 outstanding surveys, 121 (71%) were returned. The number of responses varied by guideline; however, for all but three guidelines more than 60% of the surveys were returned.

We retrieved 5,860 titles during our initial search, which focused on review articles and editorials. We identified 1,232 titles when our search was focused on key specialty journals and journals of major general interest. Overall, we reviewed a total of 6,994 titles. From these titles, 613 were selected for further review based on their potential for providing new information relating to the respective guideline areas. Abstracts were obtained and reviewed for all 613 of these titles. From these, 181 full-text articles were retrieved and reviewed by our physician reviewers. In addition, 159 guidelines were also retrieved and reviewed. With the addition of evidence suggested by the experts, we reviewed 766 abstracts and 208 articles.

After considering the results of our limited literature searches and survey, along with our assessment of the relative importance each guideline statement contributes in producing good outcomes, we classified the AHRQ guidelines into the three categories described above. The following table presents our recommendations for each of the guidelines. In all, we recommended that (1) seven guidelines be completely withdrawn, because diagnostic or therapeutic recommendations that we judged contributed greatly to improving patient outcomes were obsolete or missing; (2) three guidelines be retained in circulation, as they were judged to still represent good clinical care; and (3) six guidelines be retained with new evidence appended, as the new evidence was judged to be insufficient to warrant total withdrawal of the guideline but of sufficient importance that clinicians interested in the guideline should be made aware of its existence. This new evidence could be appended to the g! uideline either electronically or via a paper insert. For one guideline pertaining to quality mammography, we had an insufficient response to our survey and inadequate technical knowledge in radiologic technique to adequately assess the new evidence we identified regarding developments in mammographic imaging.

New information has made seven, or almost half, of the AHRQ clinical practice guidelines sufficiently obsolete that we recommend they be withdrawn. For an additional six guidelines (35%), we believe that clinicians should be made aware of important new evidence that has developed, however, the information is not sufficient to warrant withdrawal of the entire guideline. For these guidelines, we recommend retaining the original guideline with the new evidence appended to it, either electronically or as a paper insert. We judged three of the guidelines as valid and continuing to represent good care, and for one guideline we could not reach a conclusion.

We used a combination of limited literature searches and a survey of expert opinion to identify new evidence. These were complementary techniques in this evaluation, as some evidence identified using one method was not identified using the other method. For most of the guidelines, we received an acceptable response rate to our survey, which is a testimony to the dedication of those members of the original guideline panel as they took the time to complete the survey without compensation. The limited literature searches were easier to implement than a full literature search, but more difficult to implement than the survey of experts, particularly for those clinical areas where we were less familiar with the literature. It is likely that an updated limited literature search by the original guideline development team could be implemented more easily. Both methods, survey and limited literature search, could be improved, and our initial experience with these techniques have be! en a valuable learning lesson.

Another lesson from this project is empiric evidence of the rate at which guidelines go out of date. Over a four-to-eight year period, three quarters of the AHRQ guidelines were judged to be sufficiently out of date as to recommend withdrawal, or to require that new evidence be appended. In almost all of the cases, the significant new information concerned improvements in therapy. Guideline programs need to have systems in place to regularly assess their guidelines for current validity. This process could be made more efficient if, during development of guidelines, it was specified what type of new information (size of new clinical trial, magnitude of result, amount of new resources, changes in the costs of interventions, etc.) could trigger a change in the recommendations

The paper formats in which most guidelines are currently published make it difficult to withdraw or update specific statements, a further barrier to an efficient method for ensuring guidelines remain valid. However the increasingly widespread publication of guidelines in electronic format could facilitate a modular approach in which specific statements could be easily removed or replaced as new evidence emerges. Even if guideline statements are not replaced with new ones, an electronic format makes it easy to append comments about important content areas that the guideline does not address.