Evidence Report/Technology Assessment

Number 26

Prepared for:
Agency for Healthcare Research and Quality
U.S. Department of Health and Human Services
2101 East Jefferson Street
Rockville, MD 20852

http://www.ahrq.gov/

Contract No. 290-97-0019

Prepared by:
New England Medical Center Evidence-based Practice Center
Joseph Lau, MD
Principal Investigator

John P.A. Ioannidis, MD
Ethan Balk, MD, MPH
Catherine Milch, MD
Priscilla Chew, MPH
Norma Terrin, Ph.D.
Thomas A. Lang, MA
Deeb Salem, MD
John B. Wong, MD
Investigators

AHRQ Publication No. 01-E006

May 2001

On December 6, 1999, under Public Law 106-129, the Agency for Health Care Policy and Research (AHCPR) was reauthorized and renamed the Agency for Healthcare Research and Quality (AHRQ). The law authorizes AHRQ to continue its research on the cost, quality, and outcomes of health care and expands its role to improve patient safety and address medical errors.

This report may be used, in whole or in part, as the basis for development of clinical practice guidelines and other quality enhancement tools, or a basis for reimbursement and coverage policies. AHRQ or U.S. Department of Health and Human Services endorsement of such derivative products may not be stated or implied.

The Agency for Healthcare Research and Quality (AHRQ), through its Evidence-based Practice Centers (EPCs), sponsors the development of evidence reports and technology assessments to assist public- and private-sector organizations in their efforts to improve the quality of health care in the United States. The reports and assessments provide organizations with comprehensive, science-based information on common, costly medical conditions and new health care technologies. The EPCs systematically review the relevant scientific literature on topics assigned to them by AHRQ and conduct additional analyses when appropriate prior to developing their reports and assessments.

To bring the broadest range of experts into the development of evidence reports and health technology assessments, AHRQ encourages the EPCs to form partnerships and enter into collaborations with other medical and research organizations. The EPCs work with these partner organizations to ensure that the evidence reports and technology assessments they produce will become building blocks for health care quality improvement projects throughout the Nation. The reports undergo peer review prior to their release.

AHRQ expects that the EPC evidence reports and technology assessments will inform individual health plans, providers, and purchasers as well as the health care system as a whole by providing important information to help improve health care quality.

We welcome written comments on this evidence report. They may be sent to: Director, Center for Practice and Technology Assessment, Agency for Healthcare Research and Quality, 6010 Executive Blvd., Suite 300, Rockville, MD 20852.

Douglas B. Kamerow, M.D.	John M. Eisenberg, M.D.
Director, Center for Practice	Director
and Technology Assessment	Agency for Healthcare Research
Agency for Healthcare Research and Quality	and Quality
and Quality

The authors of this report are responsible for its content. Statements in the report should not be construed as endorsement by the Agency for Healthcare Research and Quality or the U.S. Department of Health and Human Services of a particular drug, device, test, treatment, or other clinical service.

Acute cardiac ischemia (ACI) includes unstable angina pectoris (UAP) and acute myocardial infarction (AMI) and is the leading cause of death in the United States. The rapid and accurate diagnosis of ACI has substantial clinical and economic benefits. We have updated a 1997 report that evaluated the diagnostic technologies for identifying patients with ACI in the emergency department (ED), where most patients with ACI enter the health care system.

We conducted a comprehensive MEDLINE search of the English-language literature published between 1966 and December 1998. Additional articles were retrieved from references cited in the 1997 report and bibliographies of retrieved articles. Search terms included those related to the diagnosis of ACI, AMI, and UAP in the ED and to the following technologies: prehospital electrocardiography (ECG), continuous/serial ECG, nonstandard leads ECG, exercise stress ECG, the ACI Time-Insensitive Predictive Instrument (ACI-TIPI), the Goldman chest pain protocol, biomarkers, sestamibi myocardial perfusion imaging, echocardiography, and computer-based decision aids.

We reviewed studies that assessed the diagnostic performance of these technologies and their impact on health care delivery. Study populations were adult patients presenting to EDs with signs and symptoms suggestive of ACI.

Study test performance or clinical impact data were abstracted. Each study was also assessed for its applicability and methodological quality. Evidence tables were constructed for the technologies. Meta-analyses were conducted to summarize the test performance and the clinical impact of the technologies. A decision and cost-effectiveness analysis was performed to evaluate the tradeoff between costs associated with a technology and its triage accuracy.

We screened 6,667 MEDLINE titles, retrieved 407 articles, and included 105 studies in the analysis. Most studies evaluated the diagnostic performance of the technologies; a few evaluated the clinical impact of routine use. In the general ED setting, only ACI-TIPI was shown to be able to identify most of the patients with ACI and to reduce unnecessary hospitalizations. Single measurement of biomarkers at presentation to the ED has poor sensitivity for AMI although most biomarkers have high specificity. Serial measurements can greatly increase the sensitivity for AMI while maintaining excellent specificity. However, biomarkers cannot identify most UAP.

Seventeen technologies and four combinations of technologies were evaluated in the cost-effectiveness analysis. Through the use of diagnostic performance data, the combination of troponin T and echocardiography was shown to have the best triage accuracy for patients with ACI and to be the most cost-effective. Through the use of clinical impact data, ACI-TIPI was shown to have the best triage accuracy for patients with ACI and to be the most cost-effective technology.

• A diverse array of technologies with varying degrees of diagnostic accuracy is available for use in general or selected patient populations to diagnose ACI in the ED.
• No single technology is able to identify all ACI patients and at the same time avoid hospitalizing many patients without ACI.
• Most studies evaluated patients with AMI rather than ACI.
• Research on the diagnosis of ACI in the ED is characterized by great heterogeneity among studies because of the large number of variables that can be studied. This heterogeneity has resulted in fragmented evidence that is not easily synthesized into a coherent whole.
• Some technologies (e.g., echocardiography, sestamibi imaging, exercise ECG) remain underevaluated, and little research has been done on the value of sequential testing or on test combinations.
• The number of studies on the clinical impact of these technologies is inadequate.
• The quality of reporting by studies in this area needs to be improved.

This document is in the public domain and may be used and reprinted without permission except those copyrighted materials noted for which further reproduction is prohibited without the specific permission of copyright holders.

Lau J, Ioannidis J, Balk E, et al. Evaluation of Technologies for Identifying Acute Cardiac Ischemia in Emergency Departments. Evidence Report/Technology Assessment Number 26. (Prepared by The New England Medical Center Evidence-based Practice Center under Contract No. 290-97-0019) AHRQ Publication No. 01-E006, Rockville, MD: Agency for Healthcare Research and Quality. May 2001.

Acute myocardial infarction (AMI) is the leading cause of death in the United States. Investigating the causes, progression, and treatment of AMI continues to be a national research priority. In clinical medicine, much research has focused on the early diagnosis and treatment of acute cardiac ischemia (ACI), which includes both unstable angina pectoris (UAP) and AMI. In 1991, the National Heart, Lung, and Blood Institute (NHLBI) of the National Institutes of Health initiated the National Heart Attack Alert Program (NHAAP) to study the issues related to rapid recognition and response to patients with signs and symptoms of ACI in emergency department (ED) settings, the point at which most of these patients enter the health care system. This ongoing effort brings together scientists, clinicians, and NHLBI staff with a Coordinating Committee that includes representatives of 40 professional organizations.

In 1994, the NHAAP Working Group on Evaluation of Technologies for Identifying Acute Cardiac Ischemia in the Emergency Department was formed to assess the technologies for diagnosing ACI and AMI in the ED. Members of the Working Group had expertise in the areas of cardiology, emergency medicine, general internal medicine, family practice, and nursing, as well as in the specific disciplines of meta-analysis and health services research. The Working Group reviewed all technologies for diagnosing ACI in the ED. The assessments of these technologies in actual use in EDs, and the nature, extent, and quality of the evidence on which the assessments were based, are presented in the Working Groups final 1997 report, An Evaluation of Technologies for Identifying Acute Cardiac Ischemia in the Emergency Department.

In 1998, the Agency for Healthcare Research and Quality (AHRQ, formerly the Agency for Health Care Policy and Research [AHCPR]), working as a partner for the NHLBI's NHAAP, contracted with the New England Medical Center's Evidence-based Practice Center (EPC) to update the 1997 NHAAP report. The EPC was charged with evaluating the evidence on these diagnostic technologies published since October 1994. As before, the purpose of the review was to assess the accuracy of technologies for diagnosing ACI in the ED and their clinical impact when used in this setting. However, the original 1997 report did not provide quantitative estimates of the test performance or clinical impact of the diagnostic technologies. To address this, we conducted meta-analyses where possible in which we reexamined all the studies reviewed in the original report, abstracted the necessary data, and combined these data with more recently published studies. We also conducted decision and cost-effectiveness (CE) analyses to investigate the interactions between technologies' diagnostic performances and costs, populations, and outcomes, and to provide an evidence-based framework on which to base recommendations. NHAAP Working Group members helped frame some of the study issues, but they were not involved in the evaluation of evidence or in the writing of this report.

We conducted a systematic and comprehensive search of the English-language literature published between 1966 and December 1998. Literature was retrieved from a computer MEDLINE search, references cited in the 1997 Working Group report, review of references of retrieved articles, and assistance from domain experts. Search terms included those related to the diagnosis of ACI, AMI, and UAP in the ED, and to the following technologies: prehospital electrocardiography (ECG), continuous/serial ECG, nonstandard leads ECG, exercise stress ECG, the ACI Time-Insensitive Predictive Instrument (ACI-TIPI), the Goldman chest pain protocol, biochemical tests and biomarkers (e.g., creatine kinase [CK] or its subunit [CK-MB], troponin T, etc.), sestamibi myocardial perfusion imaging, echocardiography, and computer-based decision aids.

We followed the general approach for selecting studies taken by the Working Group in its report. We considered reports if they came from work done in the ED setting; results from other settings (e.g., the cardiac care unit) were used only if little or no ED-based data were available. Data from non-ED settings were used with the understanding that they suggest potential utility but do not directly apply to the emergency setting.

We accepted prospective and retrospective studies that evaluated one or more of the technologies considered in this evidence report and included patients 18 years and older who presented to the ED with symptoms suggestive of ACI. We placed no restrictions on patients' gender or ethnicity. In general, ED testing consists of either a single test performed within the initial 4-hour period after presentation to the ED or repeated testing up to14 hours after the patient's presentation to the ED. We accepted studies with minor deviations from this standard.

Data were abstracted according to a written protocol and were summarized in evidence tables.

The evidence-grading scheme we used assesses four dimensions that are important for the proper interpretation of the evidence:

• Size of the study (weight of the evidence).
• Applicability (population category and prevalence of disease).
• Diagnostic performance or magnitude of clinical impact.
• Methodologic quality (internal validity).

We grouped the populations and settings of the studies using a four-category scale to help interpret the results. We also collected data about the prevalence of ACI or AMI to assist the interpretation. The four defined population categories are:

• Category I -- Studies that included all patients with signs and symptoms suggestive of ACI, such as chest pain, shortness of breath, jaw pain, acute pulmonary edema, and so forth. This is the most inclusive category.
• Category II -- Studies that used chest pain as the inclusion criteria. Most studies belong to this group. Category II is a subset of category I.
• Category III -- Studies that included patients with chest pain but excluded those with clinical or ECG findings of AMI. Many studies, especially studies of stress cardiac imaging or testing, belong to this group. Category III is a subset of category II.
• Category IV -- Studies in which all patients were hospitalized or which used additional criteria that enrolled highly selected subpopulations. We also placed retrospective studies in this category.

When there were sufficient data for a technology, we used three complementary methods of synthesizing data across several studies to report on its test performance: summary receiver operating characteristics (SROC) analysis, separately combined sensitivity and specificity values using a random effects model, and the summary diagnostic odds ratios using a random effects model.

We defined a three-level methodologic quality scale for test performance studies graded as follows:

• A (least bias) -- such as a study that adheres to the traditionally held concepts of high-quality diagnostic evaluation, including clear descriptions of the population and setting; clear descriptions of the reference standard, the test under investigation, and the diagnostic criteria; masked interpretation of the reference and the test under investigation; verification of the diagnoses in all or most of the patients with negative results; and no significant reporting errors that are likely to result in substantial bias.
• B (susceptible to some bias) -- a study that does not meet all the criteria in category A but whose deficiencies are unlikely to cause major bias.
• C (likely to have significant bias) -- a study with significant design or reporting flaws that cannot preclude major bias. This category includes studies in which verification bias could be a major issue and studies that have significant amounts of missing information or discrepancies in their reporting.

In the few instances where there are sufficient data reported by clinical impact studies, dichotomous outcomes expressed as risk ratio or continuous outcomes were combined using a random effects model.

We defined a three-level methodologic quality scale for clinical impact studies graded as follows:

• A (least biased) -- such as prospective controlled trials.
• B (susceptible to some bias) -- such as prospective cohort studies.
• C (likely to have significant bias) -- other designs or studies with significant conduct or reporting problems that could lead to large bias.

The MEDLINE literature search identified 6,667 titles, a third of which were published from 1994 onward, indicating increased research activities on this topic over the past 5 years compared with the previous 27 years. From these abstracts, 407 full articles were retrieved for review, 105 of which are included in the analysis.

A diverse array of technologies with varying degrees of diagnostic accuracy is available for use in general or selected populations to diagnose ACI in the ED. About half the studies analyzed were in population category II and about 30 percent in category III. Prevalence of AMI across studies, even within population categories and in similar settings, varied widely with little indication that similarly reported inclusion criteria among studies resulted in similar levels of AMI prevalence.

Despite this, there is some indication that overall, studies that included all patients with chest pain (population category II) have higher prevalence of AMI than either studies that included all patients with symptoms suggestive of ACI (population category I) or studies that excluded patients with diagnostic ECGs (population category III). In addition, though differences in AMI prevalence among different settings are not statistically significant, there is evidence that studies that analyzed only admitted ED patients have higher prevalence of AMI than those that included all ED patients. Thus, these two populations may truly be different.

Most studies evaluated the accuracy of the technologies; only a few evaluated the clinical impact of routine use. To summarize:

• Prehospital 12-lead ECG has moderate sensitivity (76 percent) and specificity (88 percent) for diagnosis of ACI. It has demonstrated a reduction of the mean time to thrombolysis by 33 minutes and short-term overall mortality in randomized trials.
• In the general ED setting, only ACI-TIPI has demonstrated, in a large multicenter clinical trial, a reduction in unnecessary hospitalizations without decreasing the rate of appropriate admission for patients with ACI.
• The Goldman chest pain protocol has good sensitivity (about 90 percent) for AMI but has not been shown to result in any differences in hospitalization rate, length of stay, or estimated costs in the single clinical impact study performed. Its applicability to patients with UAP has not been evaluated.
• Single measurement of biomarkers at presentation to the ED has poor sensitivity for AMI, although most biomarkers have high specificity (over 90 percent). Serial measurements can greatly increase the sensitivity for AMI while maintaining their excellent specificity. Biomarkers cannot identify most patients with UAP.
• Diagnostic technologies to evaluate ACI in selected populations, such as echocardiography, sestamibi perfusion imaging, and stress ECG, may have very good to excellent sensitivity; however, they have not been sufficiently studied.

Decision and cost-effectiveness analyses were performed for 17 technologies and 4 combinations of technologies that have been evaluated in the literature and this report. The cost analysis is from the payers' perspective (e.g., health insurance companies); patient outcomes are either appropriate triage or 30-day survival of patients with ACI.

As not all technologies can be applied to all patients in the ED (such as stress ECG), two different ED populations were used for the analysis: a general population model, which includes all patients in the ED, and a subgroup model, in which high-risk patients are excluded. Stress tests, sestamibi imaging, and serial and continuous ECG were evaluated only in the subgroup population.

As expected, technologies with the best diagnostic accuracy for AMI and UAP have the highest values for appropriate triage for patients with ACI. Technologies that are more effective (greater number of patients with ACI appropriately triaged) tend to have higher total costs, with the exception of ACI-TIPI. The biomarkers are least costly and have the lowest values for appropriate triage. Algorithms, combination technologies, and echocardiography are the next most effective technologies, in that order. Sestamibi imaging and exercise ECG are more expensive than other technologies but have excellent diagnostic performance for ACI.

Based on data using only the diagnostic performance data of technologies, the combination technology of troponin T and echocardiography has the best CE among all technologies applicable to the general population model. If results from clinical impact studies are incorporated, ACI-TIPI has the best CE because of its very high triage accuracy and low cost.

The incremental CE of troponin T and echocardiography is about $7,670 per additional appropriate triage for a patient with ACI compared with serial or combination biomarkers. The incremental CE of the next most effective technology, the artificial neural network, is approximately $10,560. Given the economic ramifications and the effects on the patient of a missed ACI diagnosis, this incremental CE for troponin T and echocardiography is minimal.

Because the estimates for detection of UAP are based on sparse data, we also evaluated the triage accuracy and cost-effectiveness of technologies for appropriate triage for patients with AMI only. The relative CE rankings do not change compared with the rankings for patients with ACI. There are few but important differences, however, in triage accuracy: (1) the Goldman protocol improves significantly, (2) serial CK-MB improves slightly, and (3) the combination of troponin T and echocardiography is slightly better than ACI-TIPI (a difference of one patient with AMI appropriately triaged).

The combination of troponin T and echocardiography is the most cost effective, followed by the artificial neural network. The incremental CE between these two technologies is much larger than in the general ACI model: approximately $137,000 per additional appropriately triaged patient with AMI.

In the low-risk patient subgroup model, ACI-TIPI is again the most cost-effective technology if data from clinical impact studies are incorporated. Sestamibi stress imaging has the best diagnostic performance (detects 82 percent of patients with ACI), followed by sestamibi rest scanning and exercise ECG. The costs of exercise ECG and stress sestamibi are nearly the same. The incremental CE between the two technologies is a mere $364 per appropriately triaged patient, reflecting the higher effectiveness of stress sestamibi for its cost relative to exercise ECG.

The incremental CE between stress sestamibi imaging and the next cost-effective technology, the combination of troponin T and echocardiography, is much greater: $12,757. However, given that stress sestamibi imaging results in the appropriate triage of 37 additional patients with ACI (per 1,000 ED patients) compared with troponin T and echocardiography, it appears to be a very cost-effective technology.

If data from the ACI-TIPI trial are used, the incremental CE of using ACI-TIPI compared with troponin T and echocardiography is only $1,502 per additional appropriate triage for a patient with ACI, a truly negligible increase for improved triage accuracy.

Considering only triage accuracy for patients with AMI, the combination of troponin T and echocardiography is the most cost effective. Exercise ECG and stress sestamibi imaging also have excellent triage accuracy; however, the per ED patient costs of these two technologies is about $500 more than that of troponin T and echocardiography.

• Most studies evaluated the performance of a technology in diagnosing AMI; future studies should also evaluate a technology's performance in diagnosing UAP.
• Some technologies (e.g., echocardiography, sestamibi imaging, exercise ECG, serial biomarkers, and new biomarkers such as P-selectin and fatty acid binding proteins) remain underevaluated.
• To date, most studies have evaluated the application of a single technology on patients. Research is needed to determine whether combinations of tests, such as a panel of biomarkers, or of multiple modalities, such as ECG with serial CK-MB measurements, perform better than the component tests alone.
• Because good test performance, in isolation, does not automatically translate to appropriate utilization or desired outcomes, clinical impact studies are needed to evaluate the clinical outcomes of the actual use of the test.
• The prevalence of ACI among the studies varies widely and may be explained only partially by differences in patient populations. The wide variation of prevalence has an unknown effect on test performance and interpretation of the results and may indicate incomplete reporting of study biases. We need to understand the reason for the heterogeneity of the prevalence among studies with seemingly similar patient populations.
• The methodological quality and the reporting of the diagnostic performance studies on this topic varies widely and could be improved substantially.