John M. Eisenberg *, Agency for Healthcare Research and Quality
Notice of Copyright
This article was originally published in the Journal of Health Politics, Policy and Law. All rights reserved. This material may be saved for personal use only, but may not be otherwise reproduced, stored, or transmitted by any medium, print or electronic, without the explicit permission of the copyright holder. Any alteration to or republication of this material is expressly prohibited.
It is a violation of copyright law to reproduce any copyrighted information from this publication without first obtaining separate permission directly from the copyright holder who may charge fees for the use of such materials. It is the responsibility of the user to contact and obtain the needed copyright permissions prior to reproducing materials in any form.
Permission requests should be directed to:
Journals Division
Duke University Press
Box 90660
Durham, NC 27708
Fax: (919) 688-3524
Introduction
Can We Bridge the Gap?
Are the Differences Irreconcilable? An Agenda on Evidence in Law and Health Care
Evidence: Population Probabilities or Individual Causation?
Pre Hoc and Post Hoc Evidence
Clinical Progress and Legal Reform
Adjudication of Differences
What Are the Rules?
Who Makes the Decision?
Agenda on Evidence in Law and Health Care
References and Notes
Popular attention has focused of late on the role of evidence in health care. Physicians have been encouraged to practice "evidence-based medicine," so that their clinical decisions would be based upon a foundation of solid science, especially using research that has applied rigorous epidemiologic methods and has been published in peer-reviewed journals. Evidence-based medicine involves increased reliance on formal, systematic analysis and synthesis of the research literature to determine clinical effectiveness. It challenges consensus-based judgments and applies critical assessment of the available research to decide if there is methodologically sound evidence that the outcomes of a clinical option are favorable, and it identifies types of patients for whom the service is most effective.
The response of some clinicians has been gratitude for the recognition, implicit in evidence-based medicine, that the everyday practice of clinical care can be an intellectually rigorous undertaking. Others have responded less gently, asking, in essence, "So what have I been practicing, magic?"
Indeed, there is sufficient evidence to suggest that most clinicians' practices do not reflect the principles of evidence-based medicine but rather are based upon tradition, their most recent experience, what they learned years ago in medical school, or what they have heard from their friends. The average physician is said to read scientific journals approximately two hours per week, and most are likely overwhelmed by the volume of material confronting them.
Practicing evidence-based medicine is not easy. No clinician alone can absorb and synthesize the vast amount of literature available, make judgments on its quality, and translate it into practice. Evidence-based medicine relies less on the integrative capability of the individual clinician and more on systematically organized synthesis, analysis, and integration.
The archetype of the physician who practices without a foundation of evidence-based medicine is depicted by the old saw about the doctor who describes the rationale for his decisions. When he says "in my experience," he means he has taken care of one patient like this. If he says "in my series of cases," he means he has taken care of two. And if he says "in case after case after case," he means he has seen three. Despite jokes like this one about doctors practicing opinion-based rather than evidence-based medicine, there is actually a rich tradition in clinical education of discussing the scientific literature and relying on research to frame clinical questions. It is routine on rounds in teaching hospitals to ask what the evidence is for decisions and to expect medical students, residents, and attending physicians to know the relevant literature.
When Abraham Flexner called for reform of the apprenticeship style of medical education in Medical Education in the United States and Canada, his famous Carnegie Foundation report published ninety years ago, he was calling for an adoption of what we call evidence-based medicine, and abandonment of what could be called eminence-based medicine. The Flexnerian revolution in medical education, credited with converting medicine to a scientific framework and reforming medical training from apprenticeship to education, was based on the foundation of factual knowledge. More recent and more sophisticated approaches to education recognize that no physician will know all of the literature, but that evidence-based practice should rely upon knowledgeable access to the literature, knowing how to find the appropriate evidence, where to look for it, and how to judge its strength and appropriateness for the patient at hand. Thus the culture of medical education is already moving toward building an evidence base for practice built on science. The availability of on-line search capacity has enhanced the ability of clinicians to practice evidence-based medicine with just-in-time information at the point of care.
This recent emphasis on evidence-based medicine and information systems has the promise to enhance the use of evidence in clinical practice as well as in medical education. Transferring this reliance on evidence from the culture of medical education to that of medical practice is central to improving the quality of care. In general, however, the practice of medicine has lagged behind conceptual and educational advances in relying on scientific evidence to drive clinical decisions.
Even more challenging than changing medical practice will be to introduce evidence-based approaches to the decisions made by leaders of systems of health care, including managed care organizations, hospitals, group practices, and integrated systems. The tradition in board rooms is much less one of calling on the literature than calling on a consultant (who, we must hope, knows the literature). Similarly, those who make decisions at the level of broad national or regional policy, such as legislators and government officials, often make their decisions based upon evidence but rely more upon witnesses at hearings and advocates who lobby for their causes than on systematic literature reviews. Just as clinicians need to move from opinion-based practice to evidence-based practice, health care managers and policy makers need to move from eminence-based decisions to evidence-based decisions.
The concept of evidence is a familiar one to lawyers, who study evidence in law school in courses on evidence taught by professors of evidence and who make the best case for the evidence that they can discover on behalf of their client. But in the law, the search for truth is an adversarial one, with the view that if both sides make the best case for their positions, the truth will emerge from the evidence that is presented. Research by legal, epidemiologic, and health services scholars of evidence, including those who participated in this workshop, can elucidate the differences between the ways in which healthcare experts and lawyers approach the concept of evidence.
Oliver Wendell Holmes Jr. (1997: 14-15), who disappointed his physician-father by studying law instead of medicine, wrote that in law "the man of the future is the man of statistics and the master of economics." Nonetheless, ninety-two years later, in 1989, the National Research Council's Panel on Statistical Assessments as Evidence in the Courts wrote that courts continued to rely primarily on adversarial statistical witnesses, and it called on judges to increase their use of neutral, court-appointed experts. It also recommended courses on statistics for law students and that an impartial body, such as a federal research agency, conduct relevant studies in advance of litigation to evaluate the issues (Fienberg 1989).
These different definitions of evidence in the healthcare and legal communities converge and conflict in several ways. The book A Civil Action and its popular movie rendition dramatized the difficulties in translating epidemiologic evidence into courtroom evidence. In addition to deciding what evidence should be admitted, there is the challenge of determining how the evidence should be weighed in driving a decision. Scholars seek to reconcile evidence that is probabilistic in health care with evidence that is "without a reasonable doubt" in criminal law or the "preponderance" of evidence in civil cases. Whether in malpractice or other cases, this difference in the way in which evidence is approached creates a cultural divide between medicine and the law, a conflict with its roots in different epistemologies of evidence and their impact on judgments about what care should be provided to those who are ill or are trying to stay well.
In 1993, when the Supreme Court ruled in Daubert v. Merrell Dow Pharmaceuticals (509 U.S. 579, 113 S. Ct. 2786 [1993]) that trial judges should ensure the scientific merit of evidence that is entered into court, it moved in the direction of reconciling science and the courts. Justice Harry Blackmun, writing the Court's opinion, emphasized that the Court interpreted Rule 702 of the Federal Rules of Evidence (which had been adopted in 1975) to require that "scientific" evidence have "grounding in the methods and procedures of science," and explained that the Court was "confident that federal judges possess the capacity to undertake this review." The Court refrained from defining criteria by which this determination could be made but suggested publication and peer review, as well as evaluation of the known or potential rate of error. It explicitly abandoned "general acceptance" as the only criterion, which had been the test of scientific evidence for seventy years, although the Court allowed that "widespread acceptance can be an important factor in ruling particular evidence admissible, and a 'known technique that has been able to attract only minimal support within the community' may be properly viewed with skepticism."
The current debate over medical necessity and the definition of appropriate medical services highlights the conflict between the concept of evidence as it is used in the law and as it is used by healthcare experts. Epidemiologists warn that the absence of evidence of effectiveness is not tantamount to evidence of the absence of effectiveness, and they try to explain "type one" and "type two" errors to an oft-befuddled jury.1 In the Patients' Bill of Rights debate, public policy makers have sought a definition that will assure that patients have access to those services that will improve their health but not those services that incur unnecessary expenses in costly care, especially if they are likely to do more harm than good.
This fuzzy boundary-between care that is useful and care that is not likely to benefit and may even harm the patient-has been a thorny one for policy makers for some time. In May 2000, the Health Care Financing Administration (HCFA) released its proposed process by which evidence will be used to make coverage decisions soon, a dilemma that has beset federal administrators since Medicare was established. Debate over the Patients' Bill of Rights has also focused on what the rules of evidence should be to override or affirm the decision of a health plan not to cover a particular clinical service for a particular patient (65 Federal Register 31,124-31,129 [2000]).
Every participant in the healthcare system should care about how evidence is defined. Patients will receive services based upon how evidence is weighed, and clinicians will provide services based upon their conclusions about the evidence of effectiveness and risk. Healthcare managers, purchasers, and system leaders will make decisions based upon the evidence that certain services should be provided to the clientele that they serve, and policy makers, including judicial policy makers such as judges and juries, will weigh evidence to decide whether harm has been done because a service was or was not provided.
When the Agency for Health Care Policy and Research was renamed the Agency for Healthcare Research and Quality in December 1999, many of us found the acronym AHRQ quite fitting in that it can be pronounced "arc." An arc bridges gaps, and we believe that the Agency for Healthcare Research and Quality should do just that. The agency should bridge the gap between what healthcare decision makers need to know and what is known, and between what is known and what is done in health care. It should bridge the gap of disparate health care that is provided to different members of our society by providing evidence that can drive more informed decision making. The agency's motto, "Quality Research for Quality Health Care," speaks to this responsibility of improving health care through research.
It is fitting then that AHRQ should try to bridge the gap between the way in which evidence is used in health care and in the law. We conceived of and supported this meeting because we felt that it would be important to begin and to sustain a dialogue among experts on evidence in health care and the law. Much as we often quip about the United Kingdom and the United States being divided by a common language, so it is with medicine and the law. We may use the same word—"evidence"—but clinicians and lawyers understand it quite differently.
Sometimes, at the end of a day of serious discussion and scholarly debate such as we enjoyed at this workshop on evidence, confusing and controversial topics become clearer. A vision appears for how reconciliation can be achieved, and for how groups that are in disagreement at the beginning can reach consensus at the end. We were not able to achieve that reconciliation in our one-day meeting. Instead, our differences of methodology and professional cultures have been illuminated in ways that most of us could not have appreciated before the meeting.
However, I suspect that there is one item upon which we agree—that there is a discrepancy in the understanding and application of evidence between health care and the law, and that patients are caught in the middle. Clinicians, purchasers, healthcare organizations, and other patient advocates will continue to be frustrated by the lack of clarity in the way in which the word "evidence" is used in our fields. We see the gap between health care and the law, and the need to communicate about our concepts of evidence in health care, as well as the need to reconcile these two views of evidence as we seek to understand which services should and should not be provided.
During the course of the workshop, I noted six different ways in which these tensions between legal and health care approaches to the concept of evidence were revealed.
It is difficult for statistics about populations to be interpreted at the level of the individual in any field. The discipline of clinical epidemiology was created in order to help translate population-based findings into improved clinical decisions. Yet the focus of evidence when presented by epidemiologists, and even when presented by most healthcare researchers, is at the level of a population or a large group of patients.
As Sir William Osler wrote, "Medicine is the art of probability." Most clinical decisions are based upon the knowledge that health is a stochastic process, that outcomes are probabilistic, and that it is difficult to predict where a patient will fall in a bell-shaped curve. Although much work is being done to refine these predictive measures, health care remains a probabilistic activity that tries to make decisions for individuals in the context of population-based information.
The medical educators and researchers who coined the term "evidence-based medicine" wanted to emphasize this probabilistic aspect of medical decision making, which they considered to be a paradigm shift in applying science to practice (Sackett and Guyatt 1992). Gordon H. Guyatt used the term "evidence-based medicine" in the title of his 1991 editorial for the ACP Journal Club. He and other leaders in this new field advocated a quantitative approach to the practice of medicine, including estimating probabilities of disease in the process of investigation, and estimating the likelihood and magnitude of response to treatment in deciding whether to administer a drug, or recommend a surgical procedure.
In contrast, the approach taken by the law seems to be one of asking whether an individual was harmed by a particular action or might have been denied the opportunity for benefit from an action that was not taken. It asks if there is evidence that A caused B in a particular individual. Therefore, law relies on evidence of the instance; health care relies on evidence of the generalizable. Health care decision makers often think about justice as the fairest way of distributing services among a population of individuals and temper the utilitarian frameworks of decision analysis and cost-effectiveness with concern for equity, that is, the fairness of how opportunities to benefit from health care are distributed among members of a group or society. Although the law of evidence is a standard set of rules that overlooks particular individualized situations, the law is largely based upon tenets of individual rights, wrongs, and harms, and the use of evidence is in evaluating causation in a particular instance.
Decisions made by clinicians, using evidence, are almost always made before the outcome is known. Thus these decisions rely upon probabilistic estimates, hoping to improve the likelihood for the desired outcome. While clinicians do look back at the decisions that they have made in order to learn from their errors through quality improvement and educational exercises, such as morbidity and mortality conferences and clinical-pathological conferences, medicine remains a discipline in which evidence is generally used to inform a decision to take an action for a patient. In law, by contrast, evidence is often used to determine whether an action that has already been taken was appropriate, harmful, or inappropriate. In health care, evidence is used pre hoc to support decision making to render improved health; in the law, evidence is used post hoc to judge responsibility and to render justice.
Therefore, in health care, evidence helps to determine the likelihood that an event in the future will be beneficial to the patient. In the law, evidence is used to determine the causation of an event in the past to determine who was accountable for it and who was harmed by it. It is unclear how to reconcile this fundamental difference in approach.
A third difference is the way in which change occurs in law and health care. In health care, changes in clinical practice may follow a major scientific finding or the release of a new product, but usually only after a few leading clinicians have adopted the new service with others following after. Research has shown that physicians rely upon their expert colleagues to lead clinical practice change, and that diffusion is often slow until a substantial number of these opinion leaders have adopted the new practice. This way of changing medical practice—scientific breakthrough followed by early adoption by experts and subsequent diffusion to other clinicians—is well known to pharmaceutical companies, who have, for some time, relied upon experts in a particular area to adopt their new products and then to spread the new practice to other physicians.
In contrast, in the law fundamental change generally occurs on the basis of a major decision by a judge (or a panel of judges) or by a legislative action. Because change in the law is based upon precedent, it depends upon a major case having been decided. Rather than following a long policy debate or decision by a group of leading practitioners, a key judge must make a decision, and his or her decision must be upheld by higher courts, to change the way in which legal practice is conducted. Thus the Daubert case has created an opportunity to change the way science-based evidence is used in the courts, and even so, this applies only in federal courts and does not necessarily affect the use of evidence in state courts. Rule 702 of the Federal Rules of Evidence has been adopted in at least thirty-eight states, however, and legal experts predict that eventually most state courts will be required to reexamine their standards for admissibility of scientific evidence (Walsh 1997).
A fourth area in which health care and the law approach evidence differently is the way in which experts are used to adjudicate differences. In health care, evidence is generally accepted only after it has been published in a peer-reviewed journal. Thus the decision about whether research findings are worthy of publication is a key step in the creation of the evidentiary base for clinical practice. Submitted articles are sent out for review, sometimes in a blind fashion, and the reviewers have an opportunity to critique the article before it is published. However, once the article is published, only an occasional letter to the editor or other articles with different findings are likely to refute the evidence. In addition to reviewing manuscripts that have been submitted, journal editors have the opportunity to invite commentary by experts and to evaluate or invite syntheses and analyses of large fields of literature. Although these review articles, like original science articles, are reviewed by other experts, the final decision is still in the hands of the editors, who decide whether or not to publish the article.
In contrast, lawyers choose which evidence to submit, and the judge decides what the jury will hear. Few judges have adopted the approach of identifying court-appointed experts, as a journal editor might invite reviewers or invite a review article. Gerard Anderson, Mark A. Hall, and Teresa Smith (1998) described the small number of cases in the years before Daubert in which the decision was based on evidence about the appropriateness of care, and commented on the need for reform in order that scientific rigor have more influence on judicial decisions. In many ways, the role of reviewers for medical publications is the health care equivalent of court-appointed experts, who are impartial and offer commentary based on their expertise rather than advocacy.
The adversarial approach to presenting evidence in the law is another fundamental difference in the approach to evidence. As the National Research Council Report stated, "Of the many points of tension between science and the law, the aspect of law that is by far the most frustrating to scientists is the adversarial nature of court proceedings" (Fienberg 1989: 15-16).
In health care, despite the tradition of relying upon journals, there has not been an agreed-upon set of rules of evidence that clinicians could use to accept or reject assertions of scientific articles. Because of the need for a way to classify evidence, the U.S. Preventive Services Task Force (1996) offered a methodology for the strength of evidence and the levels of recommendation to be made on the basis of that evidence. The quality of evidence can be scored as follows:
I. Evidence obtained from at least one properly designed randomized controlled trial.
II-1. Evidence obtained from well designed controlled trials without randomization.
II-2. Evidence obtained from well-designed cohort or case-controlled analytic studies, preferably from more than one center or research group.
II-3. Evidence obtained from multiple time series with or without the intervention.
III. Opinions of respected authorities, based on clinical experience, descriptive studies, or reports of expert committees.
Alfred Berg (1998), who chairs the task force, has described how these rules of evidence can be translated into decision-support tools such as clinical practice guidelines. When the Congress reauthorized the Agency for Healthcare Research and Quality, it required that AHRQ evaluate the rules of evidence that are used in evidence-based medicine. In staffing the U.S. Preventive Services Task Force as it develops its third report, AHRQ has initiated a review of these criteria.
In the law, rules of evidence have been much more clearly articulated and codified in scholarly articles but principally via key cases that have established precedent for accepting or rejecting evidence, such as Daubert. Even so, subsequent cases, particularly Kumho Tire Co. v Carmichael, made it clear that the test of reliability is flexible and that the role of judges as the gatekeepers of evidence allows them substantial discretion in what they allow to be admitted as evidence (Kumho Tire Co. v. Carmichael (131 F.3d 1433 [11th Cir. 1997], reversed, 526 U.S. 137 [1999]). Thus, in both law and medicine, the rules of evidence are still in flux and may make it difficult for a clinician or health plan trying to decide whether to provide a service to know just how scientific experts or the court would rule on the strength of the evidence. To help deal with this uncertainty, AHRQ's evidence-based practice centers serve as a national resource for evaluating evidence. Still, it is left to the decision maker to determine what to do with the evidence about the outcomes and effectiveness of health care services.
In both the law and in health care, it is important to determine who will separate the wheat from the chaff. Who will determine what evidence should be followed? In both instances, evidence is taken more seriously if it is presented by someone with credentials that make him or her seem like an expert, whether it be an expert witness or the author of a scientific article. However, in health care, the merits of the evidence must be able to stand on their own, even if the evidence is presented by an unknown researcher.
In the law, there are fewer tests for the validity of evidence, and the strength of the evidence may depend primarily upon the credibility and prestige of the expert. In the law, there is an assumption that juries will be able to understand and evaluate the significance of complex information. Indeed, this is a basic premise of the jury system, that a group of fact finders, lay persons who are peers of the parties involved, is most likely to result in justice.
In contrast, in health care, evidence is more often judged by those who are most expert, and those who are less expert are often satisfied to receive the wisdom of the experts, hearing how the experts would undertake a particular challenge, treat a particular patient, or make a particular diagnosis. Few physicians read much of the original information from the literature, but rather depend upon review articles, material in textbooks and other sources of information. AHRQ has developed a web-based clearinghouse for clinical practice guidelines to help decision makers get quick and easy access to evidence-based guidelines.2 This clearinghouse is a practical companion to the evidence-based practice centers that analyze the evidence that undergirds clinical decision making.
It is ironic that the better-educated recipient of scientific information (the physician with an advanced degree and postgraduate training) generally seeks synthesized information or secondary sources, whereas the generally less-well-educated members of a jury must draw their own conclusions about the validity of the technical and scientific material being presented to them.
In these ways and others, those who explore evidence in health care and those who are interested in ways in which the law can improve its use of evidence have much to gain by interchanges such as this one. Open-minded discussions and thoughtful conceptual and empirical analyses can bridge our fields and identify areas of miscommunication. The AHRQ/Institute of Medicine workshop was a first step. AHRQ is interested in continuing a dialogue that can provide opportunities to seek reconciliation between those who come to evidence from health care and those who come to it from the law.
We also want to explore other avenues, such as potential support for health services and legal research into issues that demand attention of policy makers, judges, legislators, providers, payers, and health plans. AHRQ's mission is to enhance the quality and effectiveness of health care and access to such care. Inherent in this broad mandate are issues such as quality of life, medical errors and patient safety, cost, and use and coverage of new technologies as compared with established ones. All of these issues may become the grounds of legal suits, and all frame contentious policy debates in the nation today. The courts need solid scientific evidence on these issues before reasoned decisions can be made. Beyond needing research that provides evidence, lawyers and clinicians need, at a minimum, to recognize the differences in their professional dialects and underlying intellectual constructs that impede constructive dialogue on these fundamental issues.
Convening leaders, thoughtful debate, and carefully designed empirical research can be the basis of a better understanding between the legal and medical-scientific communities. These investments will afford the American public better-informed choices in health care, and ultimately better health. We at AHRQ intend to continue this dialogue, to identify and frame critical research issues, and to grapple with key applications such as health systems improvement to strengthen the quality of patient care and enhance patient safety. We intend to support research that will inform this discussion, such as elucidating patients' expectations of health care, examining the relative effectiveness of various ways to achieve these goals, studying how decision makers make their decisions in the court and the clinic, evaluating the impact of clinical practice guidelines, supporting critical syntheses and analyses of the available evidence, and reducing barriers to the translation of evidence into improved health care. We will continue to seek ways of articulating the levels of evidence and sponsor research that develops new methods to assess outcomes of medical care. We anticipate that this agenda will foster evidence-based approaches to resolve debates in addressing such public policy concerns as medical necessity and utilization review.
The basic tensions between these professional cultures will not readily dissipate. But, by supporting focused research and dissemination of research findings, and by brokering opportunities for vigorous debate of sensitive issues in access and provision of health care, AHRQ can continue to be a voice for informing decision makers at the level of clinical as well as management and policy decision making.
Anderson, G., M. Hall, and T. R. Smith. 1998. When Courts Review Medical Appropriateness. Medical Care 36:1295-1302.
Berg, A. O. 1998. Dimensions of Evidence. Journal of the American Board of Family Practitioners 11:216-223.
Fienberg, S. E. 1989. The Evolving Role of Statistical Assessments as Evidence in the Courts. New York: Springer-Verlag.
Guyatt, G. H. 1991. Evidence-Based Medicine. ACP Journal Club (Annals of Internal Medicine 114 [suppl. 2]): A-16.
Holmes, O. W. 1997. The Path of the Law. Harvard Law Review 10(8):457-478.
Sackett, D., and G. H. Guyatt. 1992. Evidence-Based Medicine: A New Approach to the Teaching of Medicine. Journal of the American Medical Association 268:2420-2425.
U.S. Preventive Services Task Force. 1996. Guide to Preventive Services. 2d ed. Baltimore, MD: Williams and Wilkens.
Walsh, J. T. 1997. The Evolving Standards of Admissibility of Scientific Evidence. The Judge's Journal (summer): 33-37.
*The views and opinions expressed in this article are those of the author and do not necessarily represent those of the Agency for Healthcare Research and Quality or of the U.S. Department of Health and Human Services.
1. A type-one error is making the mistake of concluding that there is a difference between two alternatives when the observed difference is actually due to chance. A type-two error is making the mistake of concluding that there is no difference when, in fact, there is one.
2. Available on-line at http://www.guideline.gov.
Internet Citation:
What Does Evidence Mean? Can the Law and Medicine Be Reconciled? Journal of Health Politics, Policy and Law, 26:2, April 2001. Copyright 2001, Duke University Press. All rights reserved; posted with permission. For information on the journal or to order a hard copy, go to http://www.dukeupress.edu/jhppl/