Reliability of Current Physician Report Cards Questioned

Press Release Date: June 8, 1999

A new study sponsored by the U.S. Agency for Health Care Policy and Research (AHCPR) questions whether the current generation of report cards on how well individual doctors manage specific health conditions can really help people select doctors, or help doctors improve their own performance. A growing number of health plans and hospitals write report cards on their doctors.

The study looked at how doctors in three types of medical practices in different parts of the country managed patients with type II diabetes—a disease the researchers selected because of its prevalence in primary care practice and because the way doctors manage diabetes can affect outcomes in their patients.

The researchers found that report cards, or physician profiles, for diabetes were unable to reliably detect true practice differences among doctors at the three medical practices studied—a large, West Coast staff-model health maintenance organization, an urban medical school teaching clinic located in the Midwest, and a group of private-practice physicians in New England. Differences in how the physicians managed their patients' diabetes—their practice styles—contributed only four percent, at the most, to the overall variance in their patients' hospitalization, office visit, laboratory use, and blood sugar level control rates. The difficulty in using these outcomes to evaluate physician performance was due, in large part, to the relatively small number of diabetics managed by each doctor.

"Report cards on doctors may one day help Americans make informed choices, but as this study so clearly shows, the current technology for profiling physicians is not reliable enough to detect practice differences with the number of patients a doctor normally sees for a given condition. We need more research to improve physician evaluation," said John M. Eisenberg, M.D., AHCPR's Administrator.

Sheldon Greenfield, M.D., a co-author of the study and chairperson of the Diabetes Quality Improvement Project (1)—a collaborative effort by the American Diabetes Association, National Committee for Quality Assurance, Foundation for Accountability, and the U.S. Health Care Financing Administration—said the findings send a wake-up call to health plans and hospitals evaluating their physicians, and to individual doctors.

"Our findings do not mean that we physicians should not be accountable for our patients' outcomes, nor that attempts to evaluate the medical and quality-of-life outcomes of our care are futile because the average doctor doesn't see enough patients for a specific condition to make analysis meaningful. Instead, the findings suggest that health plans and hospitals take a better look at the science they're using to do report cards," said Dr. Greenfield.

According to Dr. Greenfield, who is with the New England Medical Center in Boston, each of the physicians in the study would have had to have managed over 100 diabetic patients for the researchers to detect any meaningful differences among them. "Apart from hypertension, it is difficult to imagine that there would be enough cases per primary care physician to construct disease-specific profiles for almost any other chronic condition," said Dr. Greenfield, who added that doctors sometimes receive evaluations based on as few as four patients. "The science does not support such reckless use of numbers for judging physicians. More accurate judgements could be made if physicians were evaluated in groups for the purpose of treating specific diseases."

The lead author of the study, Timothy P. Hofer, M.D., of the VA Ann Arbor Healthcare System and the University of Michigan, said another problem with the current method of profiling individual doctors is that the process can backfire if it is linked to powerful incentives not to have patients whose conditions are hard to manage because they may fail to follow instructions or for other reasons, and who incur high costs. "The easiest way a doctor can avoid becoming a "deselected provider,"—that is, not have a bad report card—is to "deselect," or drop problematic patients. In our opinion, those implementing such profiling systems would be as much to blame as the doctors who deny care to the most vulnerable people needing medical help," said Dr. Hofer.

Sherrie H. Kaplan, Ph.D., a co-author of the study and an expert on patient satisfaction, added that at least 40 or 50 patients per doctor are needed to accurately evaluate satisfaction with an individual physician's care. The current study did not report findings about patient satisfaction evaluation, but the issue was addressed in the AHCPR-funded Type II Diabetes Patient Outcomes Research Team Study (AHCPR grant HS06665), a five-year research project, from which the current study is derived. Dr. Kaplan, who is with the New England Medical Center, and the other authors, were members of the study team, which was under the direction of Dr. Greenfield.

Funding for today's study was also provided by the Veterans Affairs Health Services Research and Development Service—a component of the U.S. Department of Veterans Affairs. Details are in "The un-reliability of individual physician 'Report Cards' for assessing the costs and quality of care of a chronic disease," published in the June 9, 1999 issue of the Journal of the American Medical Association.

Note to Editors: For interviews of Dr. Greenfield, call Catherine Bromberg (617) 636-0206; for interviews of Dr. Hofer, contact Kate Durham (734) 769-7026. For interviews of Dr. Eisenberg, contact Karen Migdail (301) 427-1855 .

For additional information, contact the AHCPR Press Office (301) 427-1364: Karen Migdail (301) 427-1855 (KMigdail@ahrq.gov).

(1) The Diabetes Quality Improvement Project, which also involves the American Academy of Family Physicians, American College of Physicians, and the U.S. Department of Veterans Affairs, has developed measures considered critical to keeping diabetics as healthy as possible.

---

Internet Citation:

Reliability of Current Physician Report Cards Questioned. Press release, June 8, 1999. Agency for Health Care Policy and Research, Rockville, MD. http://www.ahrq.gov/news/press/pr1999/reptcard.htm

---