Chapter I. Introduction and Purposes

A. Overview of the Evaluation

The Agency for Healthcare Research and Quality (AHRQ) contracted with Mathematica Policy Research, Inc. (MPR) to evaluate the National Health Plan Collaborative on Racial and Ethnic Disparities (NHPC), which was cosponsored by AHRQ and the Robert Wood Johnson Foundation (RWJF).  The evaluation covers what participants view as "phase one" of the NHPC, as the Collaborative now has been funded for an additional two years of work following its original end in September 2006.  In this first phase, the Collaborative involved nine firms working with associated support organizations to address racial and ethnic disparities in care that may exist within health plans.  The firms are national and regional organizations; over half sponsor health plans in more than one location.  The work of the Collaborative was focused primarily on disparities that may exist among commercially enrolled members because that is the dominant membership in health plans and there has been less work on disparities in this area than in Medicare and Medicaid.

Several organizations supported the work of the Collaborative in phase one; the two dominant organizations were RAND and the Center for Health Care Strategies (CHCS). Other involved support organizations included the Institute for Healthcare Improvement (IHI), subcontracting with CHCS, and GMMB, a communications firm hired by RWJF). Overall, sponsors view the Collaborative as important to their priorities, but also challenging because it requires that they develop relationships in new ways with large national firms sponsoring health plans.  Not surprisingly, Collaborative goals have evolved over time and may not be consistently viewed by all participants, a theme we discuss later in the report.

After reviewing the background and status of the Collaborative (Gold et al. October 26, 2005), AHRQ agreed that the MPR evaluation would address five questions:

1. How was the Collaborative structured, and what did it do?
2. What did the Collaborative accomplish, and how valuable do firms perceive their participation in it to have been?
3. To what extent did the support provided by the Collaborative process contribute to firms' ability to make progress in addressing issues related to disparities?
4. Will the work on disparities be sustainable after the Collaborative concludes its assignment, and what tools may help measure such sustainability at that time?
5. What can AHRQ learn about whether or how to engage in similar collaboratives in the future?

The evaluation builds on a framework that highlights the various ways in which the Collaborative could contribute to significant progress in addressing racial and ethnic disparities (Figure I.1).  The framework highlights steps that health plans can take to reduce racial and ethnic disparities, and ways in which the Collaborative may contribute to plans' success.

The evaluation framework helps conceptualize the Collaborative and identify short-term indicators of progress that can help all stakeholders to assess the potential merits of the Collaborative and its contributions.  Such short-term indicators of progress include, for example, participating firms' continued commitment of resources to reducing disparities, progress in developing improved data on disparities, the ability to identify and take action on evidence-based interventions, and what firms report (and we observe) as the added contribution of the Collaborative to their efforts to address disparities.  Additional spillover effects could include stronger organizational support for the Collaborative by work that shares each others' knowledge across organizations, generic lessons for capturing data by race or ethnicity, stronger networks across organizations involving individuals who share similar functional responsibilities, and expanded knowledge of AHRQ's research by those in the field who can translate it into practice.

The evaluation was originally tasked with measuring, among other things, the Collaborative's ability to reduce racial and ethnic disparities.²  As noted previously however, our initial review of the Collaborative at the start of the evaluation indicated that this was not a realistic goal, and AHRQ concurred with this assessment. Addressing racial and ethnic disparities is a major undertaking whose success depends on much more than just the actions of participating firms and whose execution, even among Collaborative participants, will require considerable time beyond the life of the Collaborative.  The evaluation therefore focused on operationally feasible, mid-term outcomes that firms can reasonably control.  In particular, we sought to understand what the Collaborative did to help firms (1) enhance efforts by firm leadership or others to pursue work in the area of disparities; (2) collect data or use geocoding/surname analysis to improve their ability to measure disparities or monitor the effects of pilot interventions to reduce disparities; (3) develop and test pilot interventions dealing with patients, providers, or the community to reduce disparities; and (4) communicate the outcomes to others outside the Collaborative.

The evaluation began in June 2005 and covered the period though the end of the original Collaborative (September 30, 2006).  Both budgetary constraints and agency preferences resulted in an evaluation with little primary data collection.  Instead, we acquired Collaborative documents and sat as a silent observer of the Collaborative's telephone calls and meetings.  We also conducted three rounds of interviews with the lead staff of participating sponsor and support organizations and a broader set of staff from among the nine firms.  At the start of the evaluation in summer 2005, we interviewed the lead staff from each participating organization (round one interviews) and reviewed documents in order to prepare a summary of the Collaborative's history and plan for the evaluation (Gold et al. 2005).  Rounds two and three of the interviews were more extensive and focused on diverse staff from participating firms.³  The second round, the most extensive of the three, was completed between December 2005 and February 2006. We also asked all participants to complete a formatted "network feedback solicitation" to support AHRQ's interest in a formal network analysis of the Collaborative (described further in Appendix C). The third round took place from August to September 2006.

Table I.1 summarizes the data collection plan and topics we sought to address in each round of the interviews.  (Readers seeking more detail on the full evaluation design should see Gold et al. October 26, 2006.)

Return to Contents

B. Purposes of this Report and Data Sources Used

This report provides AHRQ and other Collaborative participants with feedback on the Collaborative including what it was, what participants did, and what was accomplished.  Building on the framework described above, we consider the Collaborative's work in four areas:

• Contributions to the firms' efforts to address disparities.
• How it helped firms collect and analyze disparities (especially the value of geocoding and surname analysis).
• What pilot and other interventions were tested during the Collaborative.
• How communications were used to support the Collaborative's goals. 

This final report builds on the interim report, which covered initial findings in the first two areas listed. This final report updates, where relevant, the interim report and includes more detail on the pilot interventions and communication goals (as these activities were just starting at the time of our round two interviews).  The report also provides a comprehensive analysis of the overall contribution of the Collaborative and the lessons for AHRQ.

The data sources used for the evaluation were developed by balancing evaluation needs with Collaborative firms' willingness to respond to questions or requests. Firms participating in the Collaborative did so voluntarily and their efforts were self-supported; they did not receive grants or other funding for participating.  From their perspective, the Collaborative's demands needed to be consistent with each organization's own goals and particularly the competitive marketplace in which they operated.  They saw benefits to collaboration, as discussed later, but were less convinced there were benefits to be gained from a substantial investment by their firms' staff in responding to data collection requests. However, with one possible exception, all of the firms participating in the Collaborative gave generously of their time and staff resources to support this evaluation.  Additional detail on interviews is included in Appendix A.

Return to Contents

C. Content of the Final Report

This final report of the evaluation covers the following areas of interest:

• A summary description of the Collaborative that builds on and updates the descriptions in the Interim and October 2005 reports  (Chapter II).
• An analysis of firms' rationales for participating, including how they view disparities and whether the staff assigned to work with the Collaborative and other related structures appear to occupy an appropriate organizational locus from which to influence the organization's work on disparities (Chapter III).
• An analysis of and update on the status of disparities data collection, particularly for the commercial population within the nine participating firms, the potential contributions of the geocoding and surname work involving RAND and many of the participating firms, and the experience with common measures (Chapter IV).
• An analysis of the status of activities/interventions carried out by the firms to address racial and ethnic disparities among their members, as well as an examination of results available and any known implications for the firms' future work (Chapter V).
• An analysis of how communications were used to support Collaborative goals, including the experiences and perspectives of firm staff working with GMMB to address these issues (Chapter VI).
• The final report concludes (Chapters VII and VIII) with an examination of participants' perspectives on the Collaborative as a whole, as well as our analysis of the overall contribution of the Collaborative and lessons for AHRQ.
• The Appendix C documents what we found in our network analysis about how the Collaborative works and is viewed by participants (as reported in the interim report).

We mask the identity of firms in the report, as it bears on what firms told us in interviews or reported on the network feedback form.  However, we identify firms by name in a few cases where the information is generally known to Collaborative participants or more broadly.

Return to Contents
Proceed to Next Section