Executive Summary

A. Focus of the Report

This report summarizes the results from the Mathematica Policy Research, Inc. (MPR) evaluation of the National Health Plan Collaborative (NHPC) to reduce racial and ethnic disparities, which is cosponsored by the Agency for Healthcare Research and Quality (AHRQ) and the Robert Wood Johnson Foundation (RWJF). The Collaborative began in July 2004 and ended in September 2006; the evaluation began in June 2005. A second phase is planned, but this evaluation focuses on the initial Collaborative.

The Collaborative involved nine firms working to address racial and ethnic disparities in health care that may exist within health plans. The participating firms were large national and regional organizations that covered millions of lives; over half of them sponsored health plans in more than one location. The Collaborative's work focused primarily on disparities that may exist among firms' commercially enrolled members. Participants in the Collaborative agreed to focus on diabetes and to measure disparities using common HEDIS measures. Several organizations supported the work of the Collaborative—the two dominant support organizations are RAND and the Center for Health Care Strategies (CHCS).

The evaluation sought answers to several questions:¹

1. How was the Collaborative structured, and what did it do?
2. What did the Collaborative accomplish and how sustainable will these efforts be?
3. Did the support provided by the Collaborative process contribute to firms' ability to make progress in addressing issues related to disparities, and how valuable did firms view their participation to have been?
4. What can AHRQ learn about whether or how to engage in similar collaboratives in the future?

Drawing on a conceptual framework, we sought to understand what the Collaborative did to help firms (1) support firm leadership in building support for work on disparities; (2) collect or estimate the race and ethnicity of their membership to better identify potential disparities; (3) develop and test pilot interventions to reduce disparities; and (4) communicate the outcomes of the work to others outside the Collaborative. This summary focuses on what we learned; readers will find additional detail on all of these activities within the text of the report.

B. Data Sources

This was a qualitative evaluation that involved little primary data collection. We received Collaborative documents and sat in on the Collaborative's telephone calls and meetings as a silent observer. We also conducted three rounds of interviews with the lead staff of participating organizations and a broader set of staff from among the nine firms. In round two, we asked all participants to complete a "network feedback form" to support a formal network analysis of the Collaborative. All 15 participants responded to this request, although responses for some items were incomplete.

C. Findings and Conclusions

1. How Was the Collaborative Structured and What Did it Do?

From the start, the NHPC was structured as a learning collaboration that convened participating firms through meetings and calls to discuss activities in the area of disparities. Firms also received technical assistance from support organizations.

The Collaborative involved several diverse organizations whose interests and internal styles of operation differ. A key point of contention for the Collaborative was how much to emphasize broad-based efforts to build national and firm infrastructure for addressing disparities versus small-scale, specific pilot interventions designed to reduce such disparities. The focus of the Collaborative's work evolved over time, a factor important to understanding its accomplishments.

A key focus of work was on developing insights into existing disparities; RAND provided support to firms seeking assistance with geocoding and surname analysis so that they could better learn about disparities in care among racial and ethnic minority members with diabetes (and other conditions, should firms elect to include them). The Collaborative also provided an opportunity for firms to learn more about the activities of other participating firms that are considered leaders in primary data collection efforts on race and ethnicity.

The Collaborative also encouraged firms to develop pilot interventions to reduce disparities and to focus on them during the Collaborative's second year. At firms' request, such interventions were defined broadly at the organizational, member, provider, or community level.  However, sponsor and support organizations appear to have encouraged relatively small-scale interventions that could be assessed with HEDIS measures before and after implementation. This model was more relevant to some participants—for example, regional firms with relatively small total membership—than others.  Collaborative sponsors have engaged a communications contractor, GMMB, to work with firms' communications staff on dissemination plans as the Collaborative progresses.

The success of the Collaborative needs to be assessed in context. For both sponsors and support organizations, work with large commercial health plans around disparities was a new and risky endeavor, as these organizations are complex and often difficult to understand.  Participants had varying views of the Collaborative's goal and therefore what constituted success. Moreover, the scale of participating firms influenced what they could accomplish. Firms' scale has proved both a major strength (they touch millions of lives) and weakness (more barriers to change) for the Collaborative.

In the first round of interviews, sponsors reported seeing value in small steps that made disparities a more legitimate focus of quality improvement work among firms. They also saw the value of small-scale efforts that help firms understand information needed to measure disparities, and of using the Collaborative to improve communication within firms to increase support for addressing disparities.

2. What Did The Collaborative Accomplish, And How Valuable Do Firms Perceive Their Participation In It?

Despite adjustments to their efforts over time, the Collaborative maintained the participation of all firms that were involved at the start. Sponsors and support organizations can take pride in that outcome, since many events could have shattered the Collaborative. To a certain extent, it is not surprising that firms remained in the Collaborative—whatever its demands, firms probably perceived the costs of participation as relatively low in relation to the risks associated with dropping out. As one firm participant remarked, "No one wants to be left behind. That's a strategic disadvantage." Hence, although firms' continued interest is a positive sign, it is important to look more substantively at the accomplishments of the Collaborative. The main ones include:

• Increased organizational attention and commitment to disparities as part of the quality agenda for health plans.
• Firms' growing recognition that their ability to generate primary data on race/ethnicity is critical to making progress.
• Increased awareness among sponsor and support organization staff about how firms work, in ways that are relevant to understanding firms' contribution to disparities.

The Collaborative had less success in sharing lessons about caring for patients in ways that reduce disparities and applying that knowledge to alter care delivery. We summarize below the findings on the Collaborative's progress in each area.

Organizational Commitment. All of the firms participated with the support of their senior leadership, designated well-placed senior staff to serve as liaisons, and involved their traditional reporting structures to keep executives aware of their efforts. Thus, participants in the Collaborative did so as official representatives of large organizations, a factor that contributed to their ability to influence organizational commitments to disparities. Most firms used their existing organizational channels to address concerns related to disparities, but the Collaborative also encouraged some firms to enhance their organizational structures to more effectively deal with this issue, including creating disparities task forces and the like. These structures—together with increased recognition of the issue, generated partly by firms' participation in the Collaborative—should sustain interest. While the Collaborative focused only on diabetes, firm responses suggest that any insights firms gain about disparities are influencing their thinking about care delivery in general.

However, there are challenges to sustainability, particularly stemming from the environment and the instability within the industry. All firms viewed the tight fiscal constraints imposed by the health care market as influencing their decisionmaking, although some are better positioned fiscally than others. Leadership turnover and change is also common in the industry.  Among national firms, for example, one had limited participation in Phase I because of a merger and staff turnover, two others are now dealing with turnover of the chief executive for their corporation. To the extent that firm commitments have translated into permanent change—for example procedures for data collection, or the inclusion of interventions in standard operating systems—firms are likely to be better positioned to maintain the progress they have made already.

Primary Data to Better Identify Disparities. As a result of the Collaborative, firms are much more aware of the value of race/ethnicity data in supporting quality improvement efforts targeting racial and ethnic disparities. All but one of the firms now say the goal is to capture race/ethnicity for all their members; the exception is capturing it for selected patients in disease management programs. The geocoding/surname analysis experience in Phase I was important in helping firms develop a broader-based acceptance of the existence of disparities. It also highlighted to firms what geocoding/surname analysis could do (general patterns) and what it could not (member-specific identification to support interventions, or identify patterns when residential patterns are not highly concentrated by subgroups).

Despite the accomplishments, there remains a gap between what firms have done and what they ultimately seek to do. For example, one leading firm has primary data for only a small proportion of members, despite trying for several years to collect them. Two of the firms committed to collecting race and ethnicity data have not yet determined how they will do so, and a third will not start until at least 2008, when its new IT system is in place.  Firms also face additional barriers. Because participating organizations are large, even those that have data may not store it in a way that is accessible for various uses across the firm.

Because of the time it takes to generate useable primary data on race/ethnicity, some firms are planning to use geocoding/surname analysis into the future to benchmark change by geographic area or further identify locations for disparity-oriented interventions. While some tools will continue to be made available to firms by RAND in Phase II, firms seeking individual assistance from RAND will have to enter into individual contracts, as AHRQ will not fund it. The transition poses a structural barrier to sustainability. In retrospect, it could have been valuable to consider earlier how to institutionalize firm capacity to deal with these issues, although firms seem to be making their own arrangements.

Identification and Implementation of Interventions.  Firms' efforts to pilot interventions to reduce disparities generally took a backseat to data collection. As firms gained insight on disparities, they began to think more concretely about what they, as firms sponsoring diverse health plans, could do to reduce disparities. By the end of the Collaborative, seven of the nine firms had either completed or were in the process of completing pilot interventions, and two were developing them. During this evaluation, it was too early for most to know the outcomes of their interventions; however, most thought their efforts created a framework and base for future expansion and learning, and planned to continue related interventions after Phase I ended.

Firm progress in pursuing interventions was challenging.  These challenges included: 1) uncertainty about how to begin, and how best to intervene; 2) lack of data on race/ethnicity of particular members; 3) implementing effective interventions that could leverage the diverse functional systems in the firm and the split between corporate and regional responsibilities; and 4) logistical issues, such as recruiting physicians to participate in provider-based interventions.

The Collaborative led firms to view their work on disparities as a part of their quality improvement effort, rather than an additional or separate activity. This linkage allowed firms to create leverage to address disparities. Still, firms were constrained by the tight fiscal environment in which they operated and the competition for resources. The ability to build a business case for working on disparities was viewed as important to getting resources to address this and the quality improvement agenda in firms.

Enhanced Industry Knowledge in Staff from Sponsor/Support Organizations.  While not a stated objective, participation in the Collaborative helped sponsors and support organizations learn more about large commercial health plans. Although some key staff in sponsor and support organizations were experienced in this area, others openly said they learned a great deal about the industry through their participation in the Collaborative.

3. Did the Support Provided By the Collaborative Process Contribute to Firms' Progress in Addressing Issues Related to Disparities and How Valuable Did Firms View Their Participation As?

Overall Value. Firm responses to the network analysis clearly paint a positive picture of the Collaborative overall, as an effort that contributed to their goals. In the round three interviews, all of the firms responded positively to a question about whether they viewed their participation as worthwhile relative to its costs. Consistent with their hopes at the start of the Collaborative, firms articulated this value as allowing them to leverage firm resources, enhance firm awareness of disparities, fuel internal efforts, and ensure momentum. Firms appreciated the sponsors' willingness to provide resources to support their needs. The fact that the Collaborative was sponsored by an important federal agency and a major health foundation enhanced its credibility and provided added value in the eyes of participating firms. Moreover, sponsors' decision to continue with a second phase of the Collaborative (as discussed more below) takes advantage of existing momentum, and the creation and institutionalization of disparities task forces (or similar) by several participating firms improves chances for longer-term sustainability.

Contribution of Collaboration. On a more concrete level, however, firms did not appear to necessarily benefit as much from collaboration as they might have, had they been willing to more openly share information or had the Collaborative been better structured to facilitate substantive learning, particularly with respect to evidence on reducing disparities. The network analysis indicated that sponsor and support organizations were seen as the "glue" that held the Collaborative together. Although termed a Collaborative, there was much more communication between firms and support organizations than from firm to firm. This finding was included in the interim report (which was shared with all participants), giving them an opportunity to consider it. From firms' discussion at the final Phase I meeting, it appears that they agreed with this conclusion. To some extent, limited sharing is a function of the culture of the firms and the markets in which they operate. As one firm noted in our interviews, "It [communication] is a double-edged sword. To learn, you have to tell." When AHRQ requested more information on this to aid in planning Phase II, firms thought the more specific focus of their work in the next phase (discussed below) would facilitate better communication, as would the experience they had working with one another and the trust developed during Phase I.

Also relevant to shared learning were the firm responses about their biggest disappointment: the Collaborative did not address their interest in knowing about "what works," especially in terms of interventions that might reduce disparities. While some of this could be a reaction to the lack of a solid evidence-based knowledge in this area, it appears that more could have been done to connect firms with sources and people who could provide insight on this issue and also to structure agendas so that they could learn more from one another. The effort required of CHCS to coordinate the complex structure of the Collaborative probably came at a cost in resources that could be devoted to more substantive support in this area. The fact that many firms did not want to focus on implementing pilot interventions may have further discouraged attention to this content, which it appeared firms wanted even if they did not want to use the Collaborative to talk about what they might do with the information.

Contribution of Communications. The communications and dissemination infrastructure was an important component of Phase I. While many participating organizations agreed that there was relatively little to communicate in the first phase, the communications work undertaken by GMMB was important in presenting a standardized and consistent message externally about the Collaborative. Moreover, much of the communications activity in Phase I—such as the development of a logo and other NHPC materials and the establishment of a core message—provides a foundation for Phase II, when the Collaborative may have substantively more to report on its activities in the area of reducing disparities.

Firm Requirements for Participating in the Collaborative.  The most contentious issues for firms were the structure and requirements the Collaborative sought to impose. Reporting requirements were a particular concern, and at least some firms viewed the cumulative number of requests from sponsor-affiliated groups to be burdensome. At the final meeting of Phase I, firms' rejection of externally imposed reporting requirements was explicit—they said they wanted to be responsible for defining any measures of progress that would be used in Phase II and were uneasy about ways in which efforts could be monitored. While firms acknowledged that Phase I deadlines were valuable in pushing their efforts forward, they felt that responding to standardized reporting requirements provided more value to sponsors and support organizations with contractual requirements than to firms themselves which were not funded to participate in the Collaborative. This is consistent with the fact that for firms, a major cost of collaboration was the demands made on the busy senior staff whose involvement was essential in generating the stature and commitment from firms that the Collaborative sought.

4. What can AHRQ learn about whether or how to engage in similar collaboratives in the future?

The evaluation findings provide insight both on issues relevant to future efforts with large firms sponsoring health plans and, specifically, for Phase II of the Collaborative.

General Lessons. In designing an initiative similar to this, with large firms sponsoring health plans, sponsors would do well to be clearer from the start about the goals of collaboration. They should also be sure that the goals are shared by all participants, and adapt participation and structure accordingly.

Assuming a given set of goals, there are at least three generic questions that warrant consideration:

• Who Participates? There are not many firms that play a major role sponsoring health plans nationally or regionally. Those that do meet this criterion are diverse in structure (ranging from quite centralized to very decentralized), investment in quality improvement, linkages with provider systems based on ownership or history, geographic coverage, and other dimensions.
• What Model for Collaboration? There are a variety of ways to structure a collaboration. Ultimately, the form chosen should support the overall goals (neither these goals nor the structure seemed to have been given appropriate consideration at the outset of the Collaborative). The decision to have RWJF sponsor a support organization (CHCS) to complement RAND's work for AHRQ was a significant one that probably had more influence over the Collaborative than has been recognized. The Collaborative was structured on a model of traditional quality improvement work with smaller, less complex organizations—typically providers or small health plans with strong links to provider groups. Other structures may be more appropriate, depending on the goals. For example, if the goal is to inspire firms to prioritize work on disparities and to leverage firm scale to remove environmental barriers to doing so, it might be appropriate to use a workgroup model that includes politically savvy expert facilitators with deep knowledge of firms' workings—a former chief executive officer (CEO) who is well respected by firms and has a good grasp of public policy concerns, for example—and the support of consulting content experts. The Learning Network or Laboratory that some participants suggested could be another model.
• How to Leverage the Private Sector Effectively?  Working with large private sector organizations that function in highly competitive markets is different from working with grantees beholden to the sponsor and financially motivated to cooperate. Sponsors seeking to engage large private sector organizations in group efforts should understand the reasons (business, political, personal) that drive a firm to participate, the constraints that are likely to limit their response, and the processes required to link the external work within the Collaborative to the firm's infrastructure and decisionmaking processes.
• How to Encourage Sustainability? Because Turnover in staff can be anticipated, sponsors need to think about how change can be institutionalized and instability within participating firms. The other side of sustainability involves doing as much advance thinking as possible about how to sustain work in firms after external support is over. AHRQ may want to consider building more formal requirements for technology transfer into RFPs to help leverage the work funded through AHRQ's support contracts.

5. Insights on the Next Phase of the Collaborative

To sustain attention on reducing disparities, sponsors have decided to proceed to a Phase II for two more years. While many details remain to be determined, the intent is to increase the specificity and clarity of objectives in Phase II, with a focus on particular activities that firms agree are important. While not all of the firms participating in the Collaborative will necessarily be involved in each of the activities, the foci for attention in Phase II are (1) developing approaches to primary data collection on race/ethnicity; (2) collective work on ways to enhance language access at the national and local market level; (3) developing the business case for work on disparities, both nationally and within firms; and (4) continuing information exchange both among participating firms and with other stakeholders (which includes a communications component that builds on Phase I accomplishments).

While some might view the specific activities of Phase II as a more narrow scope of work—and perhaps more constraining—than Phase I, the fact that these activities were defined by participating firms, rather than sponsors or support organizations, should improve buy-in in Phase II. Some firms expressed discontent with being told what to do in Phase I. From our perspective the more collaborative approach in Phase II—at least in determining what activities to pursue—holds promise and allows firms more flexibility to focus on those activities of greatest interest to them. Further, it is beneficial to narrow the scope of work potentially of interest to firms when resources and time are limited. Focusing attention on a limited number of priority areas—while giving firms flexibility to participate in them or not—is an efficient approach to generate substantive change.

Our evaluation suggests that extending the Collaborative will be valuable to firms in sustaining and expanding the accomplishments to date.  As one participating firm observed, the Collaborative serves as "the external cattle prod that keeps us moving." Moreover, by the end of Phase I, the Collaborative appeared to be gaining momentum. Given the external pressures on firms and the competition for resources, the Collaborative will encourage firms to continue to focus on the area of disparities and provide a platform from which they can share their experiences, successes, and, if they choose to, failures. This alone will be valuable to firms seeking insight and support.

However, the challenges should not be underestimated, particularly if Phase II success is to be measured in terms of concrete accomplishments. While the plan for this phase may appear more concrete and defined than that of Phase I, there are in fact many remaining ambiguities.  From our observations of the process through which the Collaborative chose specific foci for Phase II works—primary data collection, language access, and the business case—we believe it will take strong leadership to move participating firms in a direction that is both useful to them and substantively clear and feasible. After tasks are better defined, support organizations may also find that they need to draw on additional expertise and organizations to achieve specific goals, such as the use of expert facilitators (such as former CEOs) or consulting content experts to lead collaborative sessions on particular topics as necessary.

Sponsors and support organizations may need to be more realistic about what they can accomplish with their own resources and the internal energy firms can devote to specific issues. On the one hand, keeping all stakeholders engaged requires a broad focus because each firm has its own priorities. On the other, to the extent that the focus is on collective accomplishments rather than communications support to firms, only so much can be done. Although there was a conscious effort to limit the number of activities in Phase II, we are concerned that the successful completion of each task may be complicated by defining these activities to include many interrelated tasks. For example, some at this stage appear to have national and market components (data or language access) and others national and firm-specific estimates (in terms of the business case for work on reducing disparities). The Collaborative will also have to invest in enhanced information sharing and, potentially, strengthening the substantive content of support. If many of the same staff (within firms and within the Collaborative) are expected to support each of these things, there is a risk that none will be done well.

We are also concerned that too high a share of the resources available to the Collaborative have, in the past, been devoted to coordination rather than substantive analysis linked to other external efforts and scientific knowledge of the available evidence/state of the work in each target area. For example, for the primary data collection activities of Phase II, the Collaborative will need to identify how its efforts interface (if at all) with providers and/or purchasers and how they relate to existing governmental efforts at standardization.

Our recommendations for Phase II include developing more targeted goals for each Phase II activity and encouraging an outcomes-oriented (rather than process- and logistics-oriented) approach to Collaborative meetings. Careful focus to targeted activities can help ensure that the Collaborative's attention is not spread too thin or in too many directions.

6. Conclusion

In conclusion, the Collaborative has enhanced firms' interest in effective interventions to measure and address disparities. However, there remain many substantive issues about how to design and support such measures and interventions, and many political, organizational, and market factors that must be considered. We encourage participants in the Collaborative to carefully assess priorities and lessons from Phase I as they continue to work on the important issue of racial and ethnic disparities in health care.

Return to Contents
Proceed to Next Section