Chapter I. Introduction and Purposes
A. Overview of the Evaluation
The Agency for Healthcare Research and Quality (AHRQ)
contracted with Mathematica Policy Research, Inc. (MPR) to evaluate the
National Health Plan Collaborative on Racial and Ethnic Disparities (NHPC),
which was cosponsored by AHRQ and the Robert Wood Johnson Foundation (RWJF).
The evaluation covers what participants view as "phase one" of the NHPC, as the
Collaborative now has been funded for an additional two years of work following
its original end in September 2006. In this first phase, the Collaborative
involved nine firms working with associated support organizations to address
racial and ethnic disparities in care that may exist within health plans. The
firms are national and regional organizations; over half sponsor health plans
in more than one location. The work of the Collaborative was focused primarily
on disparities that may exist among commercially enrolled members because that
is the dominant membership in health plans and there has been less work on
disparities in this area than in Medicare and Medicaid.
Several organizations supported the work of the Collaborative
in phase one; the two dominant organizations were RAND and the Center for
Health Care Strategies (CHCS). Other involved support organizations included
the Institute for Healthcare Improvement (IHI), subcontracting with CHCS, and
GMMB, a communications firm hired by RWJF). Overall, sponsors view the
Collaborative as important to their priorities, but also challenging because it
requires that they develop relationships in new ways with large national firms
sponsoring health plans. Not surprisingly, Collaborative goals have evolved
over time and may not be consistently viewed by all participants, a theme we
discuss later in the report.
After reviewing the background and status of the Collaborative
(Gold et al. October 26, 2005), AHRQ agreed that the MPR evaluation would
address five questions:
- How was the Collaborative structured, and what did it do?
- What did the Collaborative accomplish, and how valuable do firms
perceive their participation in it to have been?
- To what extent did the support provided by the Collaborative process contribute to firms' ability to make progress in addressing issues related to disparities?
- Will the work on disparities be sustainable after the Collaborative
concludes its assignment, and what tools may help measure such sustainability
at that time?
- What can AHRQ learn about whether or how to engage in similar collaboratives in the future?
The evaluation builds on a framework that highlights the
various ways in which the Collaborative could contribute to significant
progress in addressing racial and ethnic disparities (Figure I.1). The
framework highlights steps that health plans can take to reduce racial and
ethnic disparities, and ways in which the Collaborative may contribute to
plans' success.
The evaluation framework helps conceptualize the Collaborative
and identify short-term indicators of progress that can help all stakeholders
to assess the potential merits of the Collaborative and its contributions.
Such short-term indicators of progress include, for example, participating
firms' continued commitment of resources to reducing disparities, progress in
developing improved data on disparities, the ability to identify and take
action on evidence-based interventions, and what firms report (and we observe)
as the added contribution of the Collaborative to their efforts to address
disparities. Additional spillover effects could include stronger organizational
support for the Collaborative by work that shares each others' knowledge across
organizations, generic lessons for capturing data by race or ethnicity,
stronger networks across organizations involving individuals who share similar
functional responsibilities, and expanded knowledge of AHRQ's research by those
in the field who can translate it into practice.
The evaluation was originally tasked with measuring, among
other things, the Collaborative's ability to reduce racial and ethnic
disparities.2
As noted previously however, our initial review of the Collaborative at the
start of the evaluation indicated that this was not a realistic goal, and AHRQ
concurred with this assessment. Addressing racial and ethnic disparities is a
major undertaking whose success depends on much more than just the actions of
participating firms and whose execution, even among Collaborative participants,
will require considerable time beyond the life of the Collaborative. The
evaluation therefore focused on operationally feasible, mid-term outcomes that
firms can reasonably control. In particular, we sought to understand
what the Collaborative did to help firms (1) enhance
efforts by firm leadership or others to pursue work in the area of disparities;
(2) collect data or use geocoding/surname analysis to improve their ability to
measure disparities or monitor the effects of pilot interventions to reduce
disparities; (3) develop and test pilot interventions dealing with patients,
providers, or the community to reduce disparities; and (4) communicate the
outcomes to others outside the Collaborative.
The evaluation began in June 2005 and covered the period
though the end of the original Collaborative (September 30, 2006). Both
budgetary constraints and agency preferences resulted in an evaluation with
little primary data collection. Instead, we acquired Collaborative documents
and sat as a silent observer of the Collaborative's telephone calls and
meetings. We also conducted three rounds of interviews with the lead staff of
participating sponsor and support organizations and a broader set of staff from
among the nine firms. At the start of the evaluation in summer 2005, we interviewed
the lead staff from each participating organization (round one interviews) and
reviewed documents in order to prepare a summary of the Collaborative's history
and plan for the evaluation (Gold et al. 2005). Rounds two and three of the
interviews were more extensive and focused on diverse staff from participating
firms.3
The second round, the most extensive of the three, was completed between
December 2005 and February 2006. We also asked all participants to complete a
formatted "network feedback solicitation" to support AHRQ's interest in a
formal network analysis of the Collaborative (described further in Appendix C).
The third round took place from August to September 2006.
Table I.1 summarizes the data collection plan and topics we
sought to address in each round of the interviews. (Readers seeking more
detail on the full evaluation design should see Gold et al. October 26, 2006.)
Return to Contents
B. Purposes of this Report and Data Sources Used
This report provides AHRQ and other Collaborative participants
with feedback on the Collaborative including what it was, what participants
did, and what was accomplished. Building on the framework described above, we
consider the Collaborative's work in four areas:
- Contributions to the firms' efforts to address disparities.
- How it helped firms collect and analyze disparities (especially
the value of geocoding and surname analysis).
- What pilot and other interventions were tested during the
Collaborative.
- How communications were used to support the Collaborative's
goals.
This final report builds on the
interim report, which covered initial findings in the first two areas listed.
This final report updates, where relevant, the interim report and includes more
detail on the pilot interventions and communication goals (as these activities
were just starting at the time of our round two interviews). The report also
provides a comprehensive analysis of the overall contribution of the
Collaborative and the lessons for AHRQ.
The data sources used for the evaluation were developed by
balancing evaluation needs with Collaborative firms' willingness to respond to
questions or requests. Firms participating in the Collaborative did so
voluntarily and their efforts were self-supported; they did not receive grants
or other funding for participating. From their perspective, the
Collaborative's demands needed to be consistent with each organization's own
goals and particularly the competitive marketplace in which they operated.
They saw benefits to collaboration, as discussed later, but were less convinced
there were benefits to be gained from a substantial investment by their firms'
staff in responding to data collection requests. However, with one possible
exception, all of the firms participating in the Collaborative gave generously
of their time and staff resources to support this evaluation. Additional
detail on interviews is included in Appendix A.
Return to Contents
C. Content of the Final Report
This final report of the evaluation covers the following areas
of interest:
- A summary description of the Collaborative that builds on and
updates the descriptions in the Interim and October 2005 reports (Chapter II).
- An analysis of firms' rationales for participating, including how
they view disparities and whether the staff assigned to work with the
Collaborative and other related structures appear to occupy an appropriate
organizational locus from which to influence the organization's work on disparities
(Chapter III).
- An analysis of and update on the status of disparities data
collection, particularly for the commercial population within the nine
participating firms, the potential contributions of the geocoding and surname
work involving RAND and many of the participating firms, and the experience
with common measures (Chapter IV).
- An analysis of the status of activities/interventions carried out
by the firms to address racial and ethnic disparities among their members, as
well as an examination of results available and any known implications for the
firms' future work (Chapter V).
- An analysis of how communications were used to support
Collaborative goals, including the experiences and perspectives of firm staff
working with GMMB to address these issues (Chapter VI).
- The final report concludes (Chapters VII and VIII) with an
examination of participants' perspectives on the Collaborative as a whole, as
well as our analysis of the overall contribution of the Collaborative and
lessons for AHRQ.
- The Appendix C documents what we found in our network analysis
about how the Collaborative works and is viewed by participants (as reported in
the interim report).
We mask the identity of firms in the report, as it bears on
what firms told us in interviews or reported on the network feedback form.
However, we identify firms by name in a few cases where the information is
generally known to Collaborative participants or more broadly.
Return to Contents
Proceed to Next Section