AQA Invitational Meeting Summary (continued)

You Are Here: AHRQ Home > Quality Assessment > Quality Information & Improvement > AQA Invitational Meeting Summary

AQA Invitational Meeting Summary

Report of the Performance Measurement Workgroup

Kevin Weiss, American College of Physicians

Kevin Weiss opened his remarks by thanking the members of the workgroup for their diligence and support. He then briefly outlined the Performance Measurement Workgroup's meeting objectives:

Discuss proposed revisions to AQA Parameters for Selecting Measures for Physician Performance.
Review and approve Defining Cost of Care Measures.
Review and approve Appropriateness Criteria Principles.
Review and approve the quality metrics for acute otitis externa, otitis media with effusion, hepatitis C, and end-stage renal disease.
Review measures for future consideration, including pathology, anesthesiology, prostate cancer, and general thoracic surgery.
Provide a status report on work in progress, including individual versus physician group analysis, special populations, and structural measures.

AQA Parameters for Selecting Measures for Physician Performance

Opening the discussion on the AQA Parameters for Selecting Measures for Physician Performance document, Weiss noted that the proposed amendments are designed to address what elements a measure should include. He indicated that the amendments had been voted out of the workgroup without objection.

Proposed amendments:

Measures that are submitted to the AQA for review should include confirmation that they were developed in accordance with the following principles: a thorough review of relevant evidence, involvement of a broad representation of interested stakeholders, an input phase that includes a public comment period, and detailed specifications of the measures.
Measures should have a significant linkage to outcomes.

Discussion

A participant asked whether soliciting public comments was a role better suited to AHRQ. In response, Weiss indicated that the workgroup had discussed operationalization and that the amendment addresses only the fact that the AQA expects that those measures that come to the forum have a public comment period identified (whether that is an internal AQA process or a more formal process).

Motion: To approve the amendments to the AQA Parameters for Selecting Measures for Physician Performance.

Result: The motion was adopted unanimously.

Cost of Care Measurement

Opening the discussion on cost of care, Weiss discussed three revisions to the Defining Cost of Care Measures document.

Amendment #1

The first amendment would revise Section I, B, to change the Episode Grouping title to add [to add what? Please clarify.] (if applicable).

Motion: To approve the revised language.

Result: The motion was adopted unanimously.

Amendment #2

The second amendment would delete Bullet #3 from the Episode Grouping category and replace it with the following language:

Adjustments for clinical factors affecting the relative cost of care such as illness severity, patient comorbidity, and provider case mix should be made, as possible. Adjustments for other factors such as variation in patient demographics and benefit plans should also be made, if possible. The methodology and types of adjustments utilized must be fully disclosed.

Weiss indicated that the language had been approved by the workgroup with only a few abstentions.

Discussion

One participant asked about the "adjustments for other factors...if possible" language. If it's not possible, he asked, then why are we doing the measure? He suggested deleting the qualifying language ("such as variation in patient demographics and benefit plans"). He later added that the goal here was to measure cost, not quality.

Several people spoke in support of keeping the qualifying language. One participant said that the language addresses the need to do the best job possible and use what is available. Another participant said that the language on patient demographics was important but also cautioned that the AQA health care community needed to be careful not to accept a different standard of care for minorities or socially disadvantaged groups.

Another participant noted that it was important not to judge providers of patients on cost alone in cases where, in fact, people do not have access to the same coverage. If you cannot adjust for patients, said another person, then it is important to compare like physicians.

Kevin Weiss noted that the key was full disclosure. Carolyn Clancy agreed, saying that this document (and amendment) deals with principles. Operationalizing then becomes a matter for the Reporting Workgroup.

One participant suggested changing "if possible" to "if appropriate." Another participant spoke against the idea, saying that it was the intent of the workgroup that all of the factors have an impact on performance and cost of care. Weiss then polled the room; participants voiced no interest in changing the language.

Motion: To approve the revised language.

Result: The motion was adopted, with two "No" votes.

Amendment #3

Turning to the section on Disclosure and Transparency, Weiss indicated that the workgroup was recommending changes to Subbullet #2 in the first bullet so that it reads as follows:

Those users of episode grouping products should provide a publicly available set of technical specifications and standardized rules for their methods of grouping, risk assessment, case mix, and other adjustments prior to use of these measures for public reporting.

Motion: To approve the revised language.

Result: The motion was adopted, with two "No" votes.

The Document as a Whole

Kevin Weiss then opened the full document for discussion.

One participant suggested adding "patient's illness, and comorbidities" to the end of sentence two under Section II, to make the sentence consistent with the rest of the document. As an alternative, Weiss suggested amending it to say "adjusted as noted elsewhere in the document." Bullet #3 in that section would then be similarly amended.

Another participant asked for a point of clarification regarding the phrase "standardized costs" in Section II, Bullet #1. She asked if it meant that if a health plan wants to report using these measures, it must be based on their actual costs (versus a fee schedule). In response, Weiss said that the language means that, for AQA-approved measures, there must be a standardized cost. You wouldn't be able to report only on actual costs, he said.

A participant asked whether Section II, Bullet #4, included liability insurance. This is a major factor in cost of care, he said. Weiss said that while the workgroup did discuss liability, there was a unanimous vote not to move the language forward.

Another participant suggested changing the language in Section III, Bullet #3, to read "additional cares or procedures."

One participant suggested adding language to Section I, C, on volume consideration for episodes to be attributed to a provider. This led to a discussion about statistical validation and sample sizes. Several people argued that many studies have a sample size that is too small; others suggested that small sample sizes are often adequate (and often a starting point), and wondered why the AQA would hold itself to a standard that was higher than the industry standard. One participant voiced support for explicitly including language requiring disclosure of sample sizes.

In response, Weiss said that the workgroup would drill down regarding what the layers of statistical consideration should be.

Another participant referred to the many items under Section IV (Disclosure and Transparency). "How do I prepare a report for my employees?" he asked, expressing concern that the key findings of a report might be overwhelmed by the disclosure requirements. This is important, he said, because we want these reports to be meaningful. In response, Weiss said that the issue of how to handle disclosures (and whether they might be treated akin to the fine print in credit card reports) was the proper purview of the Reporting Workgroup.

Motion: To approve Defining Cost-of-Care Measures, as amended.

Result: The motion was adopted, with one abstention.

AQA Principles for Appropriateness Criteria

Kevin Weiss put on the table for discussion and final approval the AQA Principles for Appropriateness Criteria. The document was developed to provide guidance for measure developers on how best to construct and implement appropriateness criteria for related measures that will meet multistakeholder needs.

Discussion

There was considerable discussion about the phrase "not be overly weighted" in the final bullet on page 1 of the document. The full bullet reads as follows:

In using this process, the panel must be reflective of a wide range of experts and stakeholders, including clinicians (from each relevant specialty or subspecialty), purchasers, patients, payers, and industry. The panel makeup should be balanced; not be overly weighted by clinicians who perform the procedure being evaluated, nor by other stakeholders who have a vested interest in the outcome.

One participant said that, where possible, all groups should follow the same pattern of representation. Another participant suggested that perhaps the language should read "not be overly weighted by any single stakeholder." A third participant suggested that the language read "the panel makeup should be balanced and not be overly weighted by any single stakeholder." A fourth suggested that the word balance implies that no stakeholder should be overly weighted; he suggested that the second part of the sentence be stricken.

Another participant, however, defended the existing language, noting that the language is inclusive of all stakeholders but that physicians should be highlighted and put up front.

It is not always apparent what the position of all the parties is, said another participant who spoke in favor of keeping the second part of the sentence. Another suggested changing the language in the second part to conclude "and any single stakeholder who has a vested interest in the outcome."

As part of the discussion, the issue of conflict of interest was raised by a participant who noted that every stakeholder has a vested interest-but the real issue was to be careful about conflicts of interest. One participant pointed out that the idea was to achieve transparency (including full disclosure of conflicts) and balance (including a focus on evaluative science). Another participant suggested language to deal with "potential conflicts of interest."

One participant noted that the conflict of interest in this case is about who gets paid for the procedure. He said that while payment may involve a conflict of interest, that does not mean the person is doing anything wrong.

Another participant stressed that there must be disclosure and transparency in the measures that come to the AQA. We have to know whether industry was involved in the development and so forth, he said. If there was any support, that's fine, but it must be declared.

Finally, a participant pointed out that the AQA already has a conflict-of-interest statement.

Wrapping up the discussion on conflict of interest, Weiss said that the matter should be addressed in the primary AQA principles document. He pledged to bring the matter to the Performance Measurement Workgroup.

Motion: To amend the language in the second bullet in the AQA Principles for Appropriateness Criteria that starts "In using this process," to say "the panel must be balanced and reflective..." and to substitute for the second sentence, "The panel makeup should be balanced and not overly weighted by any single stakeholder group. Potential conflicts of interest need to be disclosed (to be modified as consistent with the AQA's conflict-of-interest policy in general)."

Result: The motion was adopted unanimously.

Motion: To approve AQA Principles for Appropriateness Criteria, as amended.

Result: The motion was adopted unanimously.

Proposed New Quality Measures for Adoption

Kevin Weiss opened the discussion on new quality measures for the following:

Acute Otitis Externa and Otitis Media with Effusion.
Hepatitis C.
End-Stage Renal Disease.

He said that the debate was ongoing as to when a measure is too basic, and noted that the National Quality Forum had rejected the dermatology measures approved by the AQA at its October 24, 2006, meeting. The measure will come back for review by the AQA at a later meeting.

Weiss asked the group whether there were any specific measures that needed to be considered separately. The group then separated out Acute Otitis Externa Measure #2 and Otitis Media with Effusion Measures #4 and #5.

Motion: To approve the Acute Otitis Externa measure set, excluding Measure #2.

Result: The motion was approved with one abstention.

Motion: To approve the Otitis Media with Effusion measure set (excluding Measures #4 and #5).

Result: The motion was approved with two abstentions.

Motion: To approve the Hepatitis C measure set.

Result: The motion was approved with one abstention.

Motion: To approve the End-Stage Renal Disease measure set.

Result: The motion was approved with two abstentions.

Acute Otitis Externa Measure #2

One participant said she was concerned that, while the measure addressed pain assessment, it did not address pain management. She thought it was missing the mark. A second participant echoed the concern about addressing pain management. He also expressed concern that the measure was only measurable by doing a chart review.

Responding to the concerns, one participant said that the measure was designed to go hand-in-hand with Measure #3 (on Systemic Antimicrobial Therapy)-that pain should be both assessed and treated. She added that the measure is per patient visit and is designed to continuously assess pain during every patient visit.

Finally, a participant suggested that, like the dermatology measures, this was designed to address gaps in care. If there are gaps in care, even in basic procedures, he believed we should support these measures.

Motion: To approve Acute Otitis Externa Measure #2.

Result: The motion was adopted with three "No" votes and one abstention.

Otitis Media with Effusion Measure #4

A participant expressed concern about how the lower end of the age range (patients aged 2 months) was determined. She said the age range should be based on evidence, not on an "I'm able to do it" anecdotal reason. The latter, she said, affects the credibility of the measure.

In response, Weiss suggested that that the otitis and pediatric communities collaborate on this discussion.

Motion: To approve Otitis Media with Effusion Measure #4.

Result: The motion was approved with one abstention.

Otitis Media with Effusion Measure #5

One participant spoke in opposition to the measure, saying that he was concerned that tympanostomy tube insertion was controversial and not based on evidence. A hearing test with loss is not enough to justify tympanostomy tube insertion in and of itself, he said, and thus the measure should be deleted.

Another participant spoke in support of the measure, saying that it was important to conduct a hearing test as part of an evaluation. He said that part of overall patient care is to make a judgment about what is appropriate. A second participant indicated that the measure was a quality measure designed to reduce inappropriate surgeries and to get surgeons and primary care physicians to understand that this procedure should be documented.

Motion: To approve Otitis Media with Effusion Measure #5.

Result: The motion was adopted, with four "No" votes and one abstention.

Measures for Future Consideration

Next, Weiss highlighted measures for future consideration. These include measures for the following:

Pathology.
Anesthesiology.
Prostate cancer.
General thoracic surgery.

Weiss indicated that there have been informal discussions with officials from the Centers for Medicare & Medicaid Services (CMS) on how to manage the process. He noted that there were groups that wanted to put forth measures for consideration as part of the quality reporting initiative-but that time was of the essence. CMS needs to know what the candidate measures might be, said Weiss, and we have said that the AQA would be willing to consider them and put them in front of the public.

Weiss indicated that performance measures for pathology, anesthesiology, and prostate cancer will be voted on shortly by the Physicians Consortium for Performance Improvement; the measures for general thoracic surgery will be put forth soon as well. Weiss indicated that the AQA will vote on these measures at its October meeting.

Finally, Weiss indicated that the AQA must be notified by the end of July 2007 if other groups have measures they would like the AQA to consider this year.

Status Report: Individual versus Physician Group Analysis

Next, Weiss turned to the workgroup's extensive to-do list for 2007, and provided an update on the work in progress. One issue before the workgroup is the development of a document to promote discussion, concerning when it is appropriate to look at measurement at the level of the individual physician and when it should be considered at the group level. Weiss indicated that as the document has not yet been vetted by the workgroup, it will be brought before the AQA as a whole in either draft form or final form at the October meeting.

On the topic of individual versus physician group analysis, Weiss noted that one key issue is whether it is possible to create a process where one can formally decide, up front, whether or not a measure can be reported on at the individual level (i.e., do a hearing test before...) as opposed to its application to more complex issues (such as obstetric care, where patients see a team of physicians) that involve multiple handoffs or multiple physicians involved in care. Weiss said that the challenge is that employers and health plans are committed to individual physician analysis, but physician groups have concerns about it. We need to find a common space for us to meet, he said.

Next, Weiss indicated that the Performance Measurement Workgroup would form a subgroup to look at the issue of special populations. He said that, at the request of CMS, the AQA would also look at structural measures.

Next, Weiss noted that the workgroup also has the following on its 2007 agenda:

Reviewing measures submitted by measure developers.
Working to define principles for test measures and measure maintenance.
Pursuing cost-of-care measurement (by trying to get measure development on the prioritized list and composite measures for cost of care).
Working with the NCQA and the PCPI on cost-of-care implementation rules.
Looking for a taxometry for moving from competency to high-value measures.

Discussion

One participant asked about the possibility of developing templates for measure developers. In response, Weiss said that the NCQA, the NQF, and the PCPI were doing good work on this front.

[The issue of guidance for measure developers arose again later in the meeting. A participant representing the National Quality Forum noted that the NQF has posted on its Web site (http://www.qualityforum.org) a very detailed set of guidelines for describing items that must be included when submitting measures. She noted that the process will be much easier if everyone uses the same standardized form.

Regarding instances when measures are submitted and fail, the participant added that the NQF is working through a process whereby measure developers can address shortcomings (or gaps in the committee's understanding), and resubmit the measures in a timely manner. Finally, in response to another question, she indicated that the NQF is looking to standardize its internal procedures—starting with what happens when a measure is submitted. The next step is to pilot test an internal ratings system before measures go to the Technical Advisory Panels, she said, adding that standardization was a work in progress.]

Finally, in response to a query from a participant about how to comment on draft measures, Weiss urged participants to participate in the Performance Measurement Workgroup.

Previous Section Contents Next Section