Health Care Efficiency Measures, Appendix E

You Are Here:AHRQ Home > Quality & Patient Safety > Measuring Health Care Quality > Health Care Efficiency Measures > Appendix E

Appendix E: Blinded Reviewer Comments

Contents - By Section:
Explanation of Interest in Efficiency Measures
General
Executive Summary
Chapter 1 - Introduction
Chapter 2 - Methods
  Typology
Chapter 3 - Results
Chapter 4 - Assessing Measures
Chapter 5 - Discussion
Appendix
  Editorial Comment
  Which measures are ready for use?
  Are there published measures not included?
  Are there vendor developed measures not included?

Explanation of Interest in Efficiency Measures

Section	Comments	Response
Explanation of Interest in Efficiency Measures	Primarily as the national coordinator of an effort called Prometheus Payment which is looking at a way of paying providers that will reward them for both efficiency and quality. In addition, BTE and the Leapfrog Group collaborated on two reports that dealt with how to measure provider efficiency.	No response necessary.
Explanation of Interest in Efficiency Measures	The health plan I work for is using efficiency measures to tier specialist physicians for one large employer account at present. That account may eventually require us to use them to tier primary care physicians as well. Different visit co-pay amounts are tied to the different tiers.	No response necessary.
Explanation of Interest in Efficiency Measures	Interest in efficiency measures: In my role as a medical director at the Rochester Individual Practice Association (RIPA), I continue to work with our panel members and insurer (Excellus Blue Cross Blue Shield) to reduce waste in the provision of health care. As you will recall, we used an ETG-based efficiency index from 1999 through 2006 as an individual performance measure in our PFP program. In response to our practitioners’ questions, concerns, and needs, we developed a tool to analyze medical practice patterns on a specialty-condition basis. Late in 2005 two of us (Greg Partridge, the senior RIPA data analyst, and I) spun Focused Medical Analytics off from RIPA to bring our tools to a wider audience. As of January 1, 2007 RIPA no longer contracted with the insurer to provide a network; the insurer moved to direct contracts. With that change the RIPA PFP system ended. We continue to advocate for our panel and work on a consultative basis with the insurer. In that role we are moving towards using our analytic tool as the basis of quality improvement programs. Because it enables us to find the specific, key cost drivers and variation for a specialty’s care of a given condition we can then have a medical appropriateness conversation, understand if the variation represents underuse at the low cost end or overuse at the high cost end, and then develop quality improvement programs. One lever toward changing physician behavior would be in such a program would be through direct measurement of the cost driver utilization at a physician or group level, and then tie that in to the larger QI project. We are working to use these physician performance measures in place of an efficiency index in any future reporting and PFP system.	No response necessary.
Explanation of Interest in Measures	For BCBSIL, primary interests regarding efficiency measures are: ~To assess efficiency at the physician/physician group level, in order to identify efficient practitioners ~To assess efficiency at the hospital group level, in order to identify efficient hospitals ~To utilize information about efficient hospitals and physicians in pay-for-performance programs ~To utilize information about efficient hospitals and physicians in public reporting ~To utilize information about efficient hospitals and physicians (along with information about quality) in the development of high performance networks ~To utilize information about efficient hospitals and physicians (along with information about quality) in the development of high performance networks ~To meet employer expectations for identifying efficient providers	No response necessary.
Explanation of Interest in Measures	I have a very high interest in efficiency measures, both as a health plan manager/actuary and as a MedPAC commissioner. Since there is strong evidence from researchers (Wennberg, Fisher and Wennberg, et al) that the current system operates with a large amount of unnecessary or inappropriate care, policy and actions should be considered that will address this issue. In addition, any changes to the Sustainable Growth Rate (SGR) mechanism in current law for Medicare should, in my opinion, include actions that will lead to a more efficient system and more efficient providers. Measuring efficiency is clearly an important part of any change. Any course of policy that ignores or defers actions to address this part of the health care financing and Medicare solvency issue is greatly flawed. Just because current methods are less than "perfect" does not mean that policymakers and payors can avoid the need for immediate action. Any policy that changes the direction to move towards greater efficiency is likely to be helpful, even if we are "learning on the job" as we develop the methodologies.	No response necessary.
Explanation of Interest in Measures	In August 2006, the President issued an Executive Order, “Promoting Quality and Efficient Health Care In Federal Government Administered or Sponsored Healthcare Programs,” that called upon all Federal agencies to make the 4 cornerstones of value-driven health care a reality in government-run healthcare programs. The 4 cornerstones are: interoperable health information technology, transparency of quality information, transparency of price information, and the use of incentives to promote high-quality and cost-efficient care. To achieve this vision, the Centers for Medicare & Medicaid Services (CMS) has begun laying the foundation for aligning consumer and provider payments and other incentives in support of quality and value. Value, as defined by CMS, includes quality and price. If value can be measured, it can drive payments to more effective providers, more appropriate settings, and more proactive treatments. Higher quality, not quantity, can be rewarded. Thus, to truly achieve value in the healthcare system, quality information should be provided along side price information to the extent feasible. There are ongoing efforts by CMS to align payment policy with the delivery of high quality, cost efficient care through our various pay-for-performance initiatives, such as the hospital pay-for-performance program. One of the core tenets of such programs will be an ongoing process for developing, selecting, and modifying measures of quality and efficiency. In addition, CMS is actively in engaged in efforts to promote the use of an increasingly broad range of consistent, valid quality physician measures. It is expected that in the future, these measures will eventually include episode-based quality and cost measures for common conditions and procedures which provide patients with an overall picture of a providers’ care.	No response necessary.
Explanation of Interest in Measures	The Integrated Healthcare Association (IHA) is interested in developing and implementing measures of cost efficiency as soon as possible for use in the Pay for performance Program (P4P) for future payment and public reporting at the physician organization level. This effort is strongly backed by the IHA Board, as well as the Pay for performance Planning and Steering committees. Given rises in health costs as evidenced in increased premiums, both health plans and purchasers have emphasized the importance of adding efficiency to the measurement portfolio to secure future funding for the Pay for performance Program. The area IHA is concentrated on is physician group cost-efficiency (per AQA definition), which most closely aligns with provider level efficiency measures, using episode and/or population based approaches.	No response necessary.

Return to Top

General

Section	Comments	Response
General	Although I may have read the document too quickly, I didn't see any mention of administrative efficiency. It would be good to mention this at least in short version as you know there is little reflection on how to improve this part of the health care system.	Administrative efficiency is discussed in the revised version of the report.
General	And finally, the AQA, PCPI, and soon NQF definitions of efficiency also include the term "value" to reflect the utilities placed on any measure of efficiency as seen from different stakeholders. Your document discusses this issue, but you may want to consider using this similar approach to definitions.	We considered this but decided to leave the definition as is.
General	Useful exploration of efficiency as pursued by academics and others (purchasers, plans, and vendors). Would suggest you include a simple example of the types of variables and formula in the academic vs the vendor/purchaser/plan models in use.	We added these examples.
General	My main reaction was that the report did not cover some of the more recent additions to the efficiency measure area (David Wennberg-Health Dialogue) and NCQA's (not sure where Cave Associates is on the resource use side)- nor did it address some of the issues related to the use of episode groupers-as MedPAC and David have pointed out, there are some major problems with the grouper technology-not only it is "black box" to most of us-but it distorts the total resource use issue. Finally, I think it is really critical to sort through the distinction between efficiency- and the more practical linkage of measures of benefit (or quality as a proxy to benefit) to measures of resource use- yielding what one might call quality to resource use ratio (or practical efficiency?).	Added Health Dialog and Cave. Also added discussion of MedPAC work.
General	Although the report details the methodological problems of existing measures of efficiency in the academic literature such as DEA and Stochastic Frontier Analysis, it could add references to critiques of those methods that are in the literature. Two papers that readily come to mind are the short pieces by myself (focusing on the partialness of outcome measures) and Jon Skinner (focusing on sensitivity to the normality assumption in Stochastic Frontier Analysis) in the October 1994 Journal of Health Economics. Undoubtedly there are other articles as well. Although the problems that these articles focus on are discussed in the report, I think it would be helpful to indicate that there is an academic literature that is highly skeptical, to say the least, of the ability of existing methods in the academic literature to make a contribution.	These references have been added. We have also noted that there is skepticism about the likelihood that measures of efficiency can be developed.
General	Several of the bullets on future research in the discussion section seem to assume that either there are or will be validated measures of efficiency, a premise that I do not think is consistent with much of the material in the report. These bullets include: “Identifying characteristics of efficient health care providers.” “Studying the relative contributions of prices, input mix, and input quantities to the efficiency of providers or health plans.” “Testing the feasibility of existing data sources in constructing efficiency measures.” “Identifying best practices that demonstrate enhanced efficiency and improved quality of care.” If one cannot measure efficiency in a meaningful way, why would future research on these questions be useful? I think this criticism applies to other bullet points in this section as well, but I won’t belabor the point.	See above.
General	Overall -- I found the report somewhat "frustrating" in that the methods in the peerreviewed literature seemed almost a theoretical exercise in "how effiicent" some parts of the delivery system might get in a hypothetical universe of medical care. Instead, many analysts, policymakers and health plan managers need something that works today -- on an urgent basis. Being able to measure "relative efficiency" of one provider to her peers or of a hospital-physician group system to another in the same market, OR (this is the longest term goal) measuring the relative efficiency of, say, Minnesota best practices to those in NYC Metro or S. California is what we really need.	We have tried to provide a better balance in the final report between some of the drivers in the academic literature (theory, measurement science) and the real world need for tools to help purchasers and plans manage costs.
General	Last -- it strikes me that this report is only partially complete (no doubt, to your great dismay). As stated in the Conclusions section (p. 55), there is little overlap with peerreviewed literature and those methods in the grey literature. Since most analysts and managers are using the grey literature methods, a lot more work should be done on these methods, vendors, techniques. As you probably know, MedPAC has had nearly two years of work ongoing in various tests and evaluations of these vendor efficiency measures. In a similar way, the Society of Actuaries has evaluated actual risk adjustment systems, using real claims data. This type of evaluation is what is really needed, and perhaps your report should recommend this follow-on effort.	We considered this but decided to leave the definition as is.
General	One final comment -- there is recent (March 2007) GAO testimony about the feasibility of using efficiency measures by CMS. Perhaps mention of this would be a worthwhile addition to the report.	We have noted that GAO is working on studies of this issue.
General	I apologize again for perhaps having over stated the obvious, and for editorializing beyond the scope of the project. I am very enthusiastic about this subject, and that leaks out at times! Please accept these comments in the collegial spirit in which they are meant. I cannot over emphasize my admiration for your work. It is an immense privilege to participate on the Technical Expert Panel and to be asked to comment. Thank you again.	No response necessary.

Return to Top

Executive Summary

Section	Comments	Response
Executive Summary	Page 1, first paragraph: “Quality” is missing in the presentation. The authors assume it is included in the notion of output, but this may be too subtle to the average reader. A distinction between cost of care measures and efficiency measures would be of value – our position paper should be attached to this critique to potentially shape this opening statement. It appears that efficiency is equated with economic profiling.	The relationship between quality and efficiency has been expanded on in this revision of the report.
Executive Summary	Page 1, Typology: I very much like the notion of perspective – they might add a sentence to enrich the nuance on how this could change the nature of the measurement. Intermediaries – they lump plans and employers as if they have the same “perspective”. This may be misleading as plans have a greater profit motive while employers have a cost reduction motive (premiums and productive work force).	The example of the physician performing CT scans in the revised report makes this point. We modified this statement to note that intermediaries may act on their own behalf as well.
Executive Summary	Page 1, outputs: Is there a place for discussing “desired outcomes?” That would make the quality connection. Where we know the desired outcome we can start using lean six sigma techniques.	We are providing some perspective on this in the discussion of health outcomes in the typology
Executive Summary	Also, it would be useful to footnote who the "four vendors and four stakeholders" were (pg3). Did these stakeholders include the provider community, and if so which organizations?	In the final version of the report, we list all 12 vendors and stakeholders we contacted.
Executive Summary	Page 3, literature, third paragraph: first real statement about quality – this kind of state should appear at the start or the end of the executive summary document.	The executive summary was revised.
Executive Summary	Page 3, last paragraph: this section needs expansion for the average reader to have better conceptual understanding of DEA and SFA --- examples and/or implications of data availability should be apparent to the reader. In addition, in their examination of regression based programs – a methodologic assessment of prospective validation would be a useful item for summarization.	We added additional text to better explain this.
Executive Summary	Page 3, second to last paragraph, the observation that ratios were more common for physician efficiency measures: My general comment above applies.	We are working in references to cost drivers and to quality improvement uses.
Executive Summary	p.3 This section should provide a succinct synopsis of stakeholder feedback (i.e. areas of concern see p 46). Descriptions of episode groupers in this amount of detail not needed for an ES.	Done
Executive Summary	P4, paragraph 3 - It might be helpful to briefly discuss that different types of efficiency measures (i.e. population-based vs. episode-based) are better suited to measuring efficiency of different types of entities (e.g. PCPs vs. hospitals) or under different financing models (e.g. capitation vs. FFS).	Reference to this is made in the body of the report.
Executive Summary	Page 5, first paragraph: “because of their clinical and statistical homogeneity, episodes of care have been widely used …” re: this homogeneity – is this presumed or confirmed by study?	This sentence was deleted in the final version.
Executive Summary	Page 5,paragraph 3: discusses use of procedural codes for CRGs – if they are the basis of defining an episode and if the procedure was unnecessary, would this system be selfconfirming of care with marginal economic benefit?	We discuss this in this revision.
Executive Summary	Page 5, Evaluation: here we have use of the term efficiency in the framework of quality as if quality can be disassociated from efficiency …… Missing from the evaluation framework is the notion of transparency of the methods of determining efficiency	This version discusses at length the relationship between quality and efficiency measures.
Executive Summary	Page 5: The last sentence on page 5 is a major finding that deserves greater prominence in the executive summary	That sentence is now included in the executive summary.
Executive Summary	P5, paragraph 2 - Add DxCG	Added DxCG.
Executive Summary	P5, paragraph 4 - 1st sentence is awkward. Consider “We suggest that measures of health care efficiency…”	We made this change.
Executive Summary	P5, paragraph 4-8 - The Evaluation section proposes 3 criteria for evaluating efficiency measures. I would suggest a 4th: Is the proposed measure suitable for the intended purpose? Your typology includes “Perspective”—are some efficiency measures inconsistent with some perspectives? Just as approved drugs end up with off-label uses, some of which may be inappropriate, efficiency measures are at risk of the same fate. You allude to this in bullet 2 under Future Research.	Added a fourth criterion for actionability.
Executive Summary	p.5 Lots of detail here for an ES on evaluation criteria (left out stakeholder’s “attributes”). Suggest compressing and thus allowing for some discussion on applications for efficiency measurement.	Done
Executive Summary	Page 6, end of first paragraph: I think I like the multi-input, multi-output measure concept (although I only have a general understanding of the ones you mention in the report). Why would it necessarily take more time and effort to convince various stake holders of their merits?	Removed this sentence
Executive Summary	Page 6, last two sentences of second paragraph: Yes! Equal quality outputs are assumed (among other things). That’s why we needed to find key cost drivers. Without quality outputs you need some way to judge if we’re asking physicians to do the right thing. Example: in several regions where we’ve looked at practice patterns in hypertension, the only thing that matters is prescribing patterns. (We’ve found about 4 different patterns by the way – 3 ways of being expensive and two ways of being less expensive.) Without blood pressure and side effect outcomes, however, we can’t say that more costly is worse. Some patients need only one med and some need 3 or 4. But we can say that your mix of medications should be weighted more towards thiazide diuretics and ACE-inhibitors than ARBs and calcium-channel blockers. That would decrease some overuse-type waste, and there are clinical guidelines to back that up. I think if we did have outcomes, however, we would get a great deall further. Then you could continue whittling away at the regimens as long as you were moving towards the desired outcome.	We have added this example with attribution.
Executive Summary	Page 6, first paragraph: the finding that no articles were found on successful use by policy makers deserves greater prominence in the summary	We made this change.
Executive Summary	Page 6: The last sentence in the first paragraph is vague – is this a good thing or a bad thing that it takes more time and effort to convince people about their merits?	This sentence has been deleted in the revised report.
Executive Summary	Page 6, Scientific Soundness: another key finding for greater prominence – the lack of testing of the scientific soundness of efficiency measures – there needs to be a greater definition of what are stochastic vs. deterministic models for efficiency	We have prominently highlighted this in the executive summary and discuss in more detail the strengths and weaknesses of various approaches. We believe this is best explained in the Explanation of Methods box in the Results chapter.
Executive Summary	Page 6: The last sentence also needs greater prominence – that many methods assume equivalent quality of all outcomes	This version discusses at length the relationship between quality and efficiency measures.
Executive Summary	Some of the points of emphasis listed above should appear here The notion of social efficiency appears here but not earlier in the text …Why?	We reworked the presentation of social efficiency in this version and have now introduced and defined social efficiency prior to this reference.
Executive Summary	Page 7: Last bullet in discussion fails to mention proprietary measures as a cause of disagreement in the different stakeholders	This sentence has been deleted in this version.
Executive Summary	Page 7, Future research: no mention of evaluating teams or units vs. individual providers	We agreed this is something useful for future research but did not add it to our already full list of future research items.
Executive Summary	P7, heading - The heading “Discussion” in the Executive Summary is labeled “Conclusions” in the main report (see page 55). The contents read more like Conclusions.	This has been changed in the Executive Summary to “conclusions”.
Executive Summary	Page 7, last bullet point: I would add poorly defined outcomes to the list of possible causes.	This has been added.
Executive Summary	p.7-8 Ditto on discussion section & research agenda as recommended below for Ch 5.	Done
Executive Summary	Page 8: third bullet on page 8 again assumes quality is independent of efficiency.	The relationship between quality and efficiency has been more extensively detailed in this revision.
Executive Summary	Page 8: In bullet one there is also the sensitivity to various methodological differences. The second bullet refers to various objectives, but these are all varieties of making a judgment (does this doctor deserve higher payment, three stars, inclusion in the network, etc.) rather than improving quality or efficiency (it seems to me). Re: the third bullet I think the way to incorporate quality of care into efficiency is a combination of waste reduction through decreased overuse and misuse, and lean-six sigma processes; can that be worked in some how? In the fifth bullet, identifying characteristics of efficient health care providers will require details and behaviors to change. Finally I suggest adding to the list of critical research topics one about defining in a measurable, quantifiable, and reproducible way, desired outcomes condition by condition, so that quality improvement techniques can be applied.	We have incorporated many of these comments in the body of the report.
Executive Summary	On page 8, one of the bullets for future research states that we should study "relative contributions of prices, input mix, and input quantitites to the efficiency of providers or health plans." The real question is "Why is this important?" If the quality and dollar-costdenominated effficiency are the same, why do we care if one part of the country delivers care differently than another? Is one type of delivery system inherently better? Probably not.	This has been deleted from the Executive Summary.
Executive Summary	P8, future research - Suggestions for additional bullets: ~Methodologies for establishing peer groupings of providers ~Attribution methodologies for physician efficiency measurement ~Which inputs should be included for which providers (e.g. who controls what?) and how should those inputs be priced? For example when using actual costs for measuring physician efficiency, should neutral pricing be applied to hospital room and board expenses?	We have added these in the body of the report.
Executive Summary	I remain concerned that there is still a little too much weight given to the industry-developed measures of "efficiency," which are just ratios of input costs (individual to group comparisons). This is reflected by nearly a page (half of pg 4 and half pg 5) out of the total 8 pages being devoted to describing some of the main proprietary products. I would suggest just a mention of these in no more than a short para.	We attempted in this revision to achieve a better balance between the published measures and the industry developed measures.
Executive Summary	In the discussion section last bullet suggest that there is disagreement between various stakeholders... however the AQA has a consensus document on definitions of efficiency. (this document was voted on by more than 100 stakeholders). This same definition is being used as part of the PCPI policy document on cost-of-care measurement. Also, this same definition is being used in the draft NQF document on efficiency and episodes of care. However this is not reflected in the exec summary, or the main document.	We acknowledged in more detail the AQA’s position in this revision but also note that among stakeholders and peer reviewers in this report there was still disagreement about the definition of efficiency.
Executive Summary	Also, important that while the industry vendors have been using are cost-of-care ratios rather than measures of input to outputs, and thereby do not even meet your document's definition of efficiency.	We disagree and believe that the industry vendors’ measures do meet our definition of efficiency.

Return to Top

Return to Report Contents
Return to Previous Section
Proceed to Next Section