Your browser doesn't support JavaScript. Please upgrade to a modern browser or enable JavaScript in your existing browser.
Skip Navigation U.S. Department of Health and Human Services
Agency for Healthcare Research Quality

AQA Invitational Meeting Summary

Report of the Performance Measurement Workgroup

Kevin Weiss, American College of Physicians

Kevin Weiss opened his remarks with a comment about the workgroup's process. He said that the workgroup tries to discuss a document or a concept until there is little left to discuss. As a result, when the workgroup votes on an item there are generally only a handful of abstentions (usually because of a conflict of interest, because a person did not have time to review the materials, or because a person was not comfortable speaking for his or her organization on the issue) and a handful of dissenting votes. He added that there are usually about 65-70 people on each call.

Weiss thanked the members of his workgroup, and noted that they have accomplished much. These accomplishments include approval of the Ambulatory Care Starter Set (26 measures), cardiology and cardiac surgery measures, and development and approval of a document on principles of efficiency measurement. In addition, he said the workgroup has held six cost-of-care physician education WebEx seminars and two meetings with specialty societies to review measure development activities.

Next, Weiss outlined his workgroup's meeting objectives:

  • Review and endorse the registry principles document.
  • Review and endorse the Guide for the Selection of Performance Measures for Medical Subspecialty Care.
  • Review and endorse the Surgery/Procedure Workgroup Mission and Principles.
  • Review and endorse the quality metrics for dermatology, radiology/neurology, rheumatology/clinical endocrinology, ophthalmology, surgery, and orthopedic surgery.
  • Review and endorse the CAHPS® Clinician and Group Survey, including composite measures.
  • Review and endorse the candidate list of conditions for cost-of-care work.
  • Review and discuss work in progress.

Finalizing Principles and Guidance Documents

Weiss started with Principles in the Use of Registries for Enhancing Quality of Care Through Performance Measurement. He noted that the document has been revised to clarify that the principles refer to registries that are created for the purpose of performance measurement.

There was no discussion.

Motion: To review and endorse the Principles in the Use of Registries for Enhancing Quality of Care through Performance Measurement.

Result: The motion was adopted unanimously.

Next, Weiss turned to the Guide for the Selection of Performance Measures for Medical Subspecialty Care. The purpose of the document is to refine the general guidelines to reflect what should be included for subspecialty care.

A participant opened the discussion with a comment about adding language addressing the needs of consumers and purchasers. He also commented on guideline 4 ("Measures should consider the importance of good comprehensive care for a condition even if only one or two of the measures will ultimately be used for public reporting. The complete set will be useful for physician practice quality improvement efforts."), saying that he believed language should be added that says the set "should be important and relevant to consumers and purchasers." He also suggested adding language that says the "collection of measures should be feasible."

Weiss asked for the sense of the group about adding the proposed language to each of the proposed amendments. In each case, there was consensus on doing this (with only a few dissenting votes).

One participant asked whether the amended phrases were redundant because a broad, general principles document already exists. Another participant also referred to the broader principles document. Weiss said he would check the general principles document to see whether there was existing language to cover these points, and rework this document if necessary.

A second participant noted that he did not really care if the language was redundant because it bears repeating that measures must be feasible and relevant to purchasers and consumers.

One participant wondered whether the language in guideline 6 ("Measures of appropriateness of care should be utilized, whenever possible") should be changed to say "whenever appropriate." Another person said he was fine with the concept, but preferred to say "whenever relevant."

Another participant stressed that it is important to make sure from the outset to create measures that are relevant to consumers. In response, someone else suggested that the solution is to identify for whom a measure is being developed.

We need to create a roadmap for better health care, said one participant, and the specialty society she represents has limited resources. She noted that if her organization has to prioritize between a measure that will make a big difference in quality of care for patients and a measure that will emphasize quality to the consumers, then it will opt in favor of the measure that physicians believe will improve quality of care.

Carolyn Clancy said that the issue of capacity across all physician societies is one that the AQA Steering Committee has been discussing at length. She said that while there are clearly measures that physicians recognize as important to quality, it is important to recognize that what consumers understand is a fast-moving target.

One participant stressed that involving consumer advocacy organizations in the process is very important.

Another person suggested that the document under discussion and the AQA Surgery/Procedures Workgroup Mission and Principles should be harmonized. Referring to the latter document, he said that language should be added to principle 4 ("The measure must address one of the IOM's six dimensions of quality care [safe, effective, patient-centered, timely, efficient, and equitable].") to say that the measure "collectively should try to address all six elements."

Motion: To review and endorse the Guidelines for the Selection of Performance Measures for Medical Subspecialty Care, as amended (to address feasibility and relevance).

Result: The motion was adopted.

Next, a member of the workgroup discussed the Surgery/Procedure Workgroup Mission and Principles, and asked for discussion on whether to include language in principle 4 regarding "collectively try[ing] to address all six elements."

One participant noted that the overarching principles criteria include feasibility. Another person said that it is important to acknowledge that there are two separate tracks that will need to be brought together at some point.

A third person said that while the initial steps to provide measurement are useful, there is also pressure to produce measures that consumers and purchasers can use. He suggested that the health plans have expertise in this area and can help the AQA in this regard. The immediate concern is whether the measures being proposed are ones for which there is no way that health plans can collect the needed information (e.g., he said that it is not possible to capture whether or not someone receives an antibiotic after surgery). There has to be a way to move more quickly than we are moving with the current measure set, he concluded.

One participant pointed to two critical elements that need to be considered: the feasibility (cost) of collecting and analyzing data, and the fact that the two tracks are not equally weighted and may or may not intersect. From a health care point of view, said the participant, we have to see what will be best for the consumer. He added that it was important to come to terms with an explicit bias in favor of consumers/patients and improving health care for them.

Another person noted that everything that comes before the AQA includes technical specifications and said that he thought some of the concerns were, in fact, being addressed.

We have given a lot of thought to implementation of these measures, said one participant. He said that there will have to be work done on the measures to add in a cost-of-care component, adding that there is a need to look at new Category 2 codes. Another person said that it is necessary but not sufficient that codes be created and assigned to these measures. He also raised a related question: What is the path by which all physicians will use these codes, and how do we ensure that all payers will use them? This, he said, stands in the way of implementation.

One participant stressed that health care is a vital part of the entire economy, affecting people and jobs. Business, he said, is looking to the AQA to bring forward a set of measures to drive efficiency and quality for purchasers and consumers of health care.

Kevin Weiss noted that the proposed language says that the measure "collectively should try to address all six elements," not that it must. We need to work to get where we want to be, he said, but, most importantly, we must keep moving as quickly as possible.

Finally, a participant noted that some measures are fully specified down to how to measure at an individual physician level with administrative data that we could use tomorrow, and some measures will be relatively feasible (and can be coded and reported). For others, however, it will take more time to get physicians to use other codes, and the path to implementation will be more difficult.

Motion: To review and endorse the Surgery/Procedure Workgroup Mission and Principles.

Result: The motion was adopted.

Proposed New Quality Measures for Adoption

Kevin Weiss discussed new quality measures for:

  • Dermatology (from the American Academy of Dermatology/Physician Consortium for Performance Improvement)
  • Rheumatology/clinical endocrinology (from the American College of Rheumatology/ American Academy of Clinical Endocrinology)
  • Radiology/neurology (from the American College of Radiology/American Academy of Neurology)
  • Ophthalmology (from the American Academy of Ophthalmology)
  • Perioperative surgery (from the American College of Surgeons, in collaboration with the Surgical Quality Alliance)
  • Orthopedic surgery (from the American Academy of Orthopedic Surgeons)

Weiss noted that some of the proposed measures are very simple and others are far more complex. He said the differences reflected where different specialty societies are in terms of their ability to develop measures. The differences also reflect the recognition that there are some very real gaps in health care quality, he said. Weiss noted that while some of the measures are basic, they come from a sense that physicians are not getting something right and that this is unacceptable. He added that his workgroup had made it clear that basic measures are a good first step, but that they could not be a means of avoiding more difficult issues. The specialty societies understand this, he said.


A representative from the American Academy of Dermatology briefly outlined the three proposed measures for physicians caring for patients with a current diagnosis of melanoma or a history of cutaneous melanoma. This outline was followed by a discussion of whether or not the measures are too simple to be acceptable.

One participant said he could not support the first two measures (addressing patient history and the complete physical skin examination) because there should be a level of basic competency. He also argued that, from an equity point of view, there needs to be a level playing field regarding performance measurements.

Another person, however, saw the matter differently. She said that it was tragic that the two measures were necessary, and it points to the fact that the quality chasm is greater than the medical community has generally acknowledged. We must look at and close that chasm to get to better quality, she argued. She also urged participants to look at "always" events, and that having measures means that plans and networks can look at these and know they are being done. To presume these are being done is not fair to patients, she said. Another participant thanked the American Academy of Dermatology for bringing forward an issue of major concern and said there is an urgency to act.

Another participant noted that it is important that the AQA see itself as improving the field of medicine and pushing the envelope. On the other hand, she said, she preferred to wait on approval until the American Academy of Dermatology can present a more comprehensive set of measures.

One participant said that when the measures were first brought before the workgroup she thought they were too basic. However, she said, there is a documented gap in care, and we need to address it and improve patient care. She added that not all measures adopted have to be used for pay for performance; some, she said, may just reflect a basic level of care. A second participant echoed the previous comments, adding that she thought perhaps one solution is to label the measures as first phase, temporary, or provisional. This would signal that these are only a first step, she said, and send a message that this measure set is not commensurate with the other sets of measures that the ACA has endorsed.

There also was discussion about the language "any physician," and whether the measures are meant for dermatologists or physicians in general. The answer is dermatologists. A question was raised about whether these measures have been coded (yes), and about to whom they apply. The answer is that the measures apply to patients seen within a practice within a calendar year.

Wrapping up the discussion, Weiss noted that the performance measurement workgroup had held pretty much the same discussion. Ultimately, the workgroup voted with only one abstention to move the measures forward, he said, recognizing that they are very basic and that if we close this gap then we have done well for improving care.

Motion: To approve the proposed dermatology measures.

Result: The motion was adopted with a handful of dissenting votes and a few abstentions.

Weiss noted that he recognized that many voting yes were doing so on the understanding that these are very basic measures—and that the American Academy of Dermatology needs to move forward rapidly with additional measures.


Next, Weiss turned to the measures for rheumatology/endocrinology. The five measures are intended for physicians who are (1) treating patients aged 50 years and older with a hip, spine, or radial fracture or (2) managing the ongoing care of patients with a diagnosis of osteoporosis. There was no discussion.

Motion: To approve proposed rheumatology measures 2-5.

Result: The motion was adopted with one dissenting vote and one abstention.

Neurology/Radiology Measures

The neurology measures (1-6) are designed for any physician caring for patients with a diagnosis of stroke or transient ischemic attack in the hospital setting. The radiology measures (7-8) are designed for radiologists and other physicians reading the imaging studies of patients with a diagnosis of stroke or transient ischemic attack in the hospital setting.

There was a question about coding. The answer was that this has not happened yet but is in the works. Another participant noted that coding specifications were sent out during the public comment period. There was also a comment about the "any physician" language, seeking clarification that it refers to neurologists/radiologists and not to all physicians.

Motion: To approve proposed neurology measures 1-6.

Result: The motion was adopted.

Motion: To approve the proposed measurement set for radiologists (measures 7-8)

Result: The motion was adopted.


The eight measures proposed by the American Academy of Ophthalmology are intended for ophthalmologists caring for patients age 18 years and older with primary open-angle glaucoma, age-related macular degeneration, cataracts, and diabetic retinopathy.

Echoing the earlier comments on the dermatology standards, there was again a discussion about whether some of the measures are too basic. Like the earlier discussion, the comments revolved around whether the bar was set too low or whether it was important to address core competencies and documented gaps in care, no matter how basic. A couple of participants suggested that perhaps these measures should be labeled a starter set, and one participant summed up the discussion by noting that there is a large quantity of data suggesting that collecting data in and of itself improves performance. Thus even the most basic standards are important, he said, and he supports the proposed standards.

In concluding the discussion, Weiss proposed that a short synopsis of the discussion be placed on the AQA Web site along with these measures in advance of a more formal AQA discussion on the matter.

Motion: To approve the proposed measurement set for ophthalmology.

Result: The motion was adopted.

Surgery (Perioperative)

These measures apply to physicians caring for patients undergoing a surgical procedure, as specified in each measure.

There was a suggestion to extend the use of prophylactic antibiotics to children. The decision was no, because the use in children was quite different and needed a separate guideline.

Motion: To approve the proposed starter set for surgery.

Result: The motion was adopted.

Orthopedic Surgery

The proposed measurement set applies to all physicians treating patients age 50 years and older with a hip, spine or radial fracture or managing the ongoing care of patients diagnosed with osteoporosis.

Motion: To approve orthopedic surgery measures 1 and 5.

Result: The motion was adopted.

CAHPS® Clinical and Group Survey

Next, Weiss discussed the proposed CAHPS® Clinical and Group Survey. He noted that the first step was to approve the measure as a whole. The next step, he said, was to figure out how to translate the items into composite measures.

There was a question about the status of the assessment tool in the National Quality Forum (NQF). Weiss replied that a final recommendation from the NQF Steering Committee was due the following week, after which the measure would be put out for public comment.

There was also a question from a participant about how the CAHPS® Clinical and Group Survey was being presented to this forum. Was it being presented as a consumer, experience-of-care transparency tool? In response, Weiss noted that it was meant for implementation in consumer and physician reporting environments.

Motion: To adopt the CAHPS® ambulatory composite measures that can be used with the survey.

Result: The measure was adopted.

Cost-of-Care Measurement

Turning to the candidate list of 20 conditions as a starter set for cost-of-care measures work, Weiss started with a brief review of the process to date. He noted that much of the urgency on quality has come from the recognition that there is not always an efficient use of care. Thus, he said, the AQA came to the conclusion that it has to look at both efficiency measures and cost-of-care measures.

The cost-of-care subgroup has looked at:

  • Drilling down a framework on cost-of-care measures.
  • Actionable moments in the physician community to address cost of care.

Referring to the 20-member candidate list of conditions, Weiss said that 20 was not a magical number, but rather that the subgroup drew a line at a point where at least half of the people participating felt that these items were of the highest priority.

Weiss stressed that the conditions are not measures. Rather, he said, they are procedures for which AQA will try to press for application to measure development. Weiss added that in compiling the list the subgroup looked at whether this set of procedures reflects both the human lifespan and a broad range of physical issues.

Opening the discussion, one participant expressed concern that the AQA was biting off too much at once and would be better off with a smaller group of conditions that could be moved forward more quickly. He also expressed concern that some conditions had practice guidelines of varying qualities. A second participant expressed concern about number 19 (spine: lumbar), suggesting that it is too broad.

One participant said that he was concerned about where the discussion was headed. He noted that the real goal is to find the true cost-of-care drivers, and noted that it does not necessarily have anything to do with the efficiency with which one person manages an episode. Instead, it is about the number of episodes. Another participant expressed concern that several of the conditions and procedures on the list (including gastroesophageal reflux disease, hip fracture, and sinusitis) did not have a quality measure that has been approved by NQF, AQA, or any other organization. She urged that they be removed from the list until quality measures have been approved.

One person said he was concerned about having a patient-centered denominator rather than a physician-centered one (i.e., cost of care per patient rather than per practice). In response, Weiss said that addressing this question is an essential next step.

One participant expressed concern that the cost-of-care subgroup was very small and not broadly open to stakeholders. She urged the AQA to table the discussion until there was more discussion (and a broader discussion) at the workgroup level.

On a separate issue, the same participant said that the list was ambitious and could lead to a significant amount of work. It should be narrowed down to areas where there are already robust quality measures in place, she said.

Another participant suggested adding a section that lists specialty cost of care (e.g., dermatology cost of care), endorses some type of episode grouping (to get at continuum of care issues), and addresses unit grouping.

Yet another participant said that dealing with the needs of patients may involve higher costs in certain populations or at various points in order to lower the overall costs to the system. So what is the measurement: The cost to the system of a diabetic patient? The cost of the entire system? He added that it is important to address the needs of the uninsured. Finally a participant, also citing the high percentage of uninsured Americans, said that tabling the measure was not an option.

Motion: To approve the Cost-of-Care Proposed Starter Set of Conditions and Procedures.

Result: The motion was adopted, with a number of people dissenting.

Weiss stressed that the issues raised would be addressed by his workgroup.

Next, Weiss addressed the Cost-of-Care Measurement Discussion Paper. He noted that the paper merely offered a cost-of-care framework, and said the draft document would need to come back as a formal document. He asked whether people would be comfortable in moving the draft document forward.

Motion: To approve the Performance Measurement Workgroup Cost-of-Care Measurement Discussion Paper as a draft working document.

Result: The motion was adopted.

Work in Progress

Weiss noted that ad hoc workgroups in the areas of test measures, individual v. physician group analysis, and appropriateness criteria have been established.

Wrapping up the discussion, Weiss noted that the workgroup on performance measurement has much left to do. The agenda moving forward, he said, includes continuing to review measures submitted by medical and surgical specialty groups, defining and developing principles for test measures, and establishing a subgroup on special populations.

Previous Section Previous Section        Contents         Next Section Next Section

AHRQ Advancing Excellence in Health Care