NIH Guide: CAHPS II

CAHPS II

Release Date: August 28, 2001

RFA: RFA-HS-02-001

Agency for Healthcare Research and Quality

Letter of Intent Receipt Date: September 24, 2001
Application Receipt Date: November 14, 2001

PURPOSE

The Agency for Healthcare Research and Quality (AHRQ) invites applications
for cooperative agreements to build on and expand the project known as
Consumer Assessment of Health Plans Study (CAHPS). AHRQ funded the initial
CAHPS cooperative agreements from October 1995 through September 2001. In
this document, we refer to that project period as “CAHPS I.” The current RFA
will be referred to as “CAHPS II.” The purposes of CAHPS I were to conduct
research to: 1) produce reliable, valid and rigorously tested survey
protocols for collecting information from consumers regarding their
assessments of health plans and services; 2) develop and test the
effectiveness of different formats for conveying resulting information to
consumers; 3) evaluate the use of resulting survey protocols in applied
settings; and 4) evaluate the usefulness of this information in assisting
consumers, and purchasers acting on their behalf, in making informed
selections of health care plans and services. Detailed information on CAHPS
I including all products developed under CAHPS I are available at
http://www.ahrq.gov/fund/grantix.htm.

The present RFA builds upon the earlier work to enrich the evidence base
concerning consumer assessment and reporting and enhance the usefulness of
CAHPS products. Its specific purposes are to conduct research to 1) maintain
existing CAHPS products, 2) build upon three existing CAHPS products relating
to nursing homes, group practices and persons with mobility impairment, 3)
develop new products, including innovative ways of reporting CAHPS data, 4)
assess the usefulness of CAHPS data for quality improvement purposes and 5)
evaluate the effectiveness of CAHPS in applied settings.

HEALTHY PEOPLE 2010

The Public Health Service (PHS) is committed to achieving the health
promotion and disease prevention objectives of "Healthy People 2010," a
PHS-led national activity for setting health improvement priorities for the
United States. AHRQ encourages applicants to submit grant applications with
relevance to the specific objectives of this initiative. Potential
applicants may obtain a copy of "Healthy People 2010" at
http://www.health.gov/healthypeople.

ELIGIBILITY REQUIREMENTS

AHRQ strongly encourages all eligible and qualified organizations to respond
to this RFA. Status as a CAHPS I grantee is not a prerequisite for this RFA.
Though each applicant organization or team must have experience in all areas
described in the “Objectives” section, this experience need not be based on
CAHPS products. We have referenced as many published and unpublished
documents and reports as possible for the purpose of describing the CAHPS I
experience to potential new CAHPS to this RFA. This RFA includes areas of
effort not addressed in CAHPS I: the use of CAHPS data for quality
improvement purposes and a greatly expanded version of the reporting task
(developing a comprehensive research approach to the development and testing
of reporting formats). Both these areas call for different types of
expertise above and beyond those required by CAHPS I.

Applications may be submitted by domestic and foreign, public and private
non-profit organizations, including universities, clinics, units of State and
local governments, and eligible components of agencies of the Federal
Government. Under recently enacted reauthorization legislation, AHRQ is
authorized to enter into cooperative agreements with for-profit organizations
as well as with public and not-for-profit entities. Thus, for-profit
organizations are invited to respond to this notice with research
applications for cooperative agreements. Such applications will be
administered in accordance with Subpart E of 45 CFR Part 74 and 42 CFR Part
67 Subpart A. The latter regulation has not yet been amended to reflect
these changes in Agency name and authority. (See December 6, 1999
reauthorization at http://www.ahrq.gov/hrqa99a.htm). AHRQ encourages women,
members of racial and ethnic minority groups, and persons with disabilities
to apply as Principal Investigators.

MECHANISM OF SUPPORT

The administrative and funding instrument to be used for this program will be
a cooperative agreement (U18), an "assistance" mechanism (rather than an
"acquisition" mechanism), in which substantial AHRQ scientific and/or
programmatic involvement with the awardee is anticipated during performance
of the activity. Under the cooperative agreement, the AHRQ purpose is to
support and/or stimulate the recipient's activity by involvement in and
otherwise working jointly with the award recipient in a partner role, but it
is not to assume direction, prime responsibility, or a dominant role in the
activity. Details of the responsibilities, relationships and governance of
the study to be funded under cooperative agreement(s) are discussed later in
this document under the section “Terms and Conditions of Award.”

The total project period for applications submitted in response to the
present RFA may not exceed 5 years. The anticipated award date is Spring
2002.

This RFA is a one-time solicitation. AHRQ has not determined whether or how
this solicitation will be continued beyond the present RFA.

We have chosen the cooperative agreement mechanism for CAHPS because it
allows AHRQ to have a greater involvement in facilitating the collaborative
efforts of the consortium members. Continuing involvement in CAHPS research
also enables AHRQ staff to serve more effectively as liaisons with other
governmental and non-governmental agencies who are interested in using CAHPS
products.

FUNDS AVAILABLE

AHRQ expects to award from $3 million to $4.5 million total costs (direct
costs plus facilities and administrative costs) in fiscal year 2002 to
support the first year of $1 million to $1.5 million to approximately three
grantees under this RFA. The total award amount is dependent on the final
fiscal year 2002 budgets for both AHRQ and the Center for Medicare and
Medicaid Services (CMS, formerly the Health Care Financing Administration).

The actual number of applications funded is dependent on the number of high
quality applications and their individual budget requirements; it is not the
intent of AHRQ that awards be equal in size. Should additional funds become
available and if AHRQ receives a sufficient number of meritorious
applications, AHRQ reserves the right to fund additional applications under
this RFA.

Although this program is provided for in the financial plans of AHRQ, awards
pursuant to this RFA are contingent upon the availability of funds for this
purpose. Funding beyond the initial budget period will depend upon annual
progress reviews by AHRQ and the availability of funds.

Definitions

The work of the CAHPS I grants and the Survey User Network (SUN) contract has
been performed by dividing team members into groups that perform specific
tasks or work towards defined goals. Since the names of these working groups
are mentioned in resource documents referenced in this RFA, we identify these
groups by name and describe their composition in this section.

Since the division of team members as described in this section served us
well in CAHPS I, it makes sense to discuss their use in CAHPS II. After
awards are made, the grantees will meet with the Project Officers to discuss
how the work can best be conducted. They will then decide together on the
best way to subdivide the grantee teams to facilitate the completion of
agreed-upon goals.

The SUN or Survey User Network contract is a five year contract for the
provision of various types of support to the CAHPS consortium. This support
includes preparing CAHPS products for dissemination to potential users;
providing technical assistance to users; providing communication support to
facilitate the work of the CAHPS consortium; providing technical and
conference support for meetings; and providing support for a National CAHPS
Benchmarking Database (NCBD) described below. During CAHPS I, the SUN
contractor played a critical role in identifying the needs of current users
of CAHPS products, responding quickly and efficiently to those needs, and
developing effective strategies for communicating with users on a regular
basis. The ways in which the grantees will interact with the SUN contractor
will be described in greater detail in the Study Organization and Budget and
Related Issues sections. Additionally, details regarding the scope of work of
the SUN contract can be found on the AHRQ website at http://www.ahrq.gov.

The NCBD contains two major data sets: CAHPS survey response data and health
plan descriptive data. The primary purpose of the NCBD is to facilitate
comparisons of CAHPS results among various types of CAHPS survey sponsors,
including Medicaid agencies, public and private employers, and individual
health plans. These comparisons are intended to provide both purchasers and
health plans with benchmarking information useful for evaluating and
improving performance as measured by the CAHPS survey. The NCBD also offers
an important source of primary data for conducting specialized research that
will enable improvements in future survey design and a better understanding
of health plan and enrollee characteristics that influence performance. All
sponsors of CAHPS surveys that are administered independently according to
CAHPS specifications are invited to participate in the NCBD. Survey sponsors
include public and private purchasers (employers, state Medicaid agencies,
coalitions) and individual health plans. The confidentiality of individual
respondent information is maintained. The NCBD also has several working
groups including the NCBD Advisory Board, Technical Advisory Panel, and
Executive Research Committee. Grantees will participate in each of these
workgroups. Further information regarding grantee responsibilities will be
presented in the Budget and Related Issues section. Additional information
about the NCBD can be found at http://ncbd.cahps.org.

“The CAHPS consortium” refers to the entire line-up of people who work on
CAHPS, direct its work or advise the grantees. This includes all grantees,
AHRQ CAHPS staff (Project Officers and others), the SUN contractor, the CAHPS
Steering Committee (to be described in the Study Organization section) and
the CAHPS Grantee Advisory Committee (to be described in the Study
Organization section). Throughout this RFA, the term “CAHPS consortium” is
used interchangeably with “the CAHPS team.”

Each grantee and their consortium/contractual partners/collaborators are
referred to as a “grantee team.” However, when the term “grantee team” is
used throughout this solicitation, it is understood that the primary grantee
is responsible for the project, which includes the programmatic and fiscal
management of third parties.

We refer to the CAHPS Survey and Reporting Kit, or CSRK, throughout this RFA.
The CSRK contains all CAHPS products as well as detailed advice on their
use. Portions of the kit are available through the AHRQ website in addition
to instructions for obtaining a complete kit, if desired.

Finally, the name of this project has been updated to reflect its expanded
focus. In the final years of CAHPS I, the team began development of tools
through which consumers and patients can assess care given to them by
providers (as members of group practices) and care they receive in nursing
homes. These efforts will be built upon in CAHPS II. Since the focus of our
product development has expanded beyond health plans, the name “Consumer
Assessment of Health Plans” no longer accurately describes the team’s work.
However, CAHPS has become a recognized product name in the industry. In view
of this, we are keeping the project name “CAHPS” but will no longer use the
original name for which this used to be an abbreviation.

RESEARCH OBJECTIVES

1. Project Goals and Objectives
A brief summary of the development of products during CAHPS I and some
information about CAHPS users can be found at
http://www.ahrq.gov/fund/grantix.htm. We encourage applicants to consult them.
Our main reason for referencing these reports is to inform potential applicants
who were not CAHPS I participants about team experiences and decisions made
during that time frame. In addition, one of the objectives of this RFA is the
further development of products of the three CAHPS I projects: nursing home
CAHPS (NH CAHPS), group practice level CAHPS (GCAHPS) and CAHPS for people with
mobility impairments (CAHPS PWMI). Applicants who were not CAHPS I
participants may need this additional background information and information
about the status of these three efforts to prepare an application.

Though the background documents represent the current “best thinking”
regarding development and testing of CAHPS products, applicants should not
feel constrained by their conclusions or recommendations. We encourage
applicants to “think beyond” these materials and present different strategies
and new approaches to the research challenges stated in this RFA. In this
section we describe the goals for CAHPS II.

This RFA includes five categories of work to be performed across two phases.
The categories are: 1) refinement of current CAHPS products; 2) continued
development of NH CAHPS, G-CAHPS and CAHPS PWMI; 3) development of new
products; 4) quality improvement studies; and, 5) evaluation studies of the
use of CAHPS products. Each category is discussed below. Work in these areas
will occur across two time spans or phases: phase 1 includes years one and
two of the cooperative agreement; phase 2 includes years three, four and
five. The phase or phases within which a product is to be completed are
listed below.

With the new emphasis on quality improvement and evaluation research in CAHPS
II, it is unlikely that any single organization wishing to respond to this
RFA will contain in itself people with expertise necessary to develop and
test questionnaires, develop and test reporting templates, modify existing
CAHPS products, plan and conduct QI studies and perform evaluations. We
encourage applicants to team with individuals or organizations who add
expertise in important areas. Applications that respond only to individual
elements of this RFA rather than the entire range of activities will be
returned as nonresponsive.

Refinement of Current CAHPS Products:
Ongoing through Phases 1 and 2

For CAHPS surveys, refinement involves periodic updating of all survey-
related information in the CAHPS Survey and Reporting Kit and includes work
on such elements as item wording, administration procedures, sampling
guidelines, data preparation, analytic strategies, guidance on interpretation
of results, as well as any other information that would facilitate use of the
products on the kit. Grantees are required to maintain existing as well as
questionnaires under development.

For CAHPS reporting templates, refinement involves periodic updating of all
print and computer reports-related information contained in the CSRK. This
includes such elements as development and testing of explanatory material in
the templates; development and testing of different data displays;
reconfiguration (if necessary) of reporting composites; development and
testing of graphic designs and navigational aids.

Grantees will also be required to update all other CAHPS products as the
consortium determines necessary. These products include but are not limited
to the booklet “Frequently Asked Questions about CAHPS” and the Survey
Submitter’s Kit. The current versions of these can be found on the AHRQ
website.

For all CAHPS products, revisions may also be based on feedback from current
and potential users, changes in technology, changing requirements of
accreditation organizations or other sources of information identified by the
Project Officers.

Further development of G-CAHPS, NH CAHPS and CAHPS PWMI
G-CAHPS: Phase 1
NH CAHPS: Phase 1
CAHPS-PWMI: Phase 1

In addition to product refinement, grantees will build upon the prior
development and testing of three CAHPS products (G-CAHPS, NH CAHPS and CAHPS
PWMI). The former grantees and SUN contractor will deliver to the CAHPS II
grantees all relevant materials.

Group Practice Level CAHPS (GCAHPS): The CAHPS team in collaboration with the
Pacific Business Group on Health and funded jointly by the California Health
Care Foundation and AHRQ developed and tested a GCAHPS instrument in
California, Eastern Massachusetts, Denver, Knoxville, and St. Louis. The
resulting instrument is in a beta test stage. Several other sites around the
country are administering the survey. Additional questions remain about
sampling and data collection. The CAHPS II goal is to continue development
and address these issues so that the GCAHPS survey can be added to the set of
CAHPS products. The final report on the developmental efforts to this point
and suggested next steps are included on the AHRQ website.

Nursing Home CAHPS (NHCAHPS): This survey is jointly supported by CMS and
AHRQ. A draft questionnaire is being cognitively tested with nursing home
residents with varying levels of cognitive capacity to assess how well they
can respond to alternate sets of response options. Also a preliminary
assessment of sampling strategies for use in nursing homes for short stay and
long stay residents has been prepared. A description of the work completed
and planned for the current effort is included on the AHRQ website. The goal
for work under this new CAHPS project will be to prepare a fully field tested
instrument; sampling strategy and data collection protocol. Applicants are
urged to offer approaches to achieving this goal by building on the current
work.

CAHPS for people with mobility impairments (CAHPS-PWMI): This project is
still at the conceptual stage of development. The only activity that has
been performed thus far is the convening of an expert group to look at the
issues and challenges related to development of a survey for PWMI. A summary
of the April 2001 meeting of this panel can be obtained through the AHRQ
website. In the development and testing of CAHPS-PWMI products, grantees
will collaborate with staff from the National Rehabilitation Hospital Center
for Health and Disability Research (NRHCHDR), staff from the National
Institute for Disability and Rehabilitation Research (NIDRR) and members of
an expert panel convened by AHRQ, NRHCHDR and NIDRR.

Development of new CAHPS products

Development and testing of PPO supplemental items: Phase 1
Development and testing of individual provider level CAHPS: Phase 1
Translation of questionnaires and reporting templates into other languages:
Phase 1
Research on cultural comparability: Ongoing throughout phases 1 and 2
New data collection technologies: Ongoing throughout phases 1 and 2
Development of OMB clearance packages: Ongoing throughout phases 1 and 2

Potential new CAHPS products include (1) development and testing of
questionnaires, screeners, and supplemental questions; (2) development and
testing of data collection technologies; and (3) development and testing of
reporting templates. Below we describe the types of products that may be
developed though the budget supporting CAHPS II will not permit development
of all of them. Products will be selected after awards are made based on
recommendations from the grantees, AHRQ assessment of need for the product,
feedback from the CAHPS Steering Committee, expertise within grantee teams,
and other sources of information. Products described below are not listed in
priority order.

o Questionnaire development and testing:

--Perform complex literature searches and reviews (along with ability to
track down relevant unpublished or in-progress studies);

--Conduct feasibility studies to determine whether prospective questionnaires
are desired by consumers in addition to identify the challenges of collecting
the data in a cost effective manner;

For example, consumers tell us that they are most interested in CAHPS related
quality of care information for individual providers. Challenges exist in
being able to collect these data in a cost effective manner. Applicants
should propose studies which will respond to these challenges. The study
should also address data collection issues to ensure that survey results can
be examined at the individual provider, group practice, and health plan
levels. If the results of the feasibility support the development of the
questionnaire then the actual instrument may be developed and tested.

--Develop questionnaires for consumer assessment of health plans, services
and providers. This includes strategies for specification of content areas;
preparation of psychometrically equivalent self- and telephone-administered
survey versions;

During CAHPS II, CAHPS supplement items for PPOs will be developed. CAHPS is
designed to be applicable across different types of health care plans.
However, it is desirable to pursue the development of supplemental items to
address the specific requirements for assessing PPOs.
--Develop and test screener questionnaires to identify special populations
such as mobility-impaired individuals;

--Develop and test sampling plans for testing purposes;

--Develop analytic plans for testing data;

--Perform psychometric testing of questionnaires; and,

--Translate questionnaire into languages other than Spanish;

Applicants may also need to re-evaluate the current Spanish language version
to assess what changes, if any, need to be made in the translation to
accommodate differences in Spanish dialects and make those changes.

--Evaluate the cultural comparability of questionnaires; and,

--Develop OMB clearance packages.

o Data collection technologies:

--Explore the feasibility of collecting data via alternate methods such as by
the internet, email, and other technologies. There are a number of challenges
to collecting data in a way that will be cost effective and keep respondent
burden at a reasonable level. Applicants should offer studies to investigate
ways in which technologies such as the internet might be used to meet the
challenges.

o Report template development and testing:

Develop and conduct five year program of research for CAHPS reports. This
would require grantees to:

–Identify and review the relevant literature, which may include health
education, consumer choice, social psychology, decision-making research,
marketing research, research on the design of electronic reports and other
areas. The goal for this activity is to identify what we know and need to
know about the characteristics of decision-makers, format and other features
of print and non-print materials, decision situations and other areas that
affect the way people make health care choices.
–Identify and obtain information about projects that are still in progress or
would not be in the published literature. Locate consumer choice projects for
which results may not have been published or which may still be in progress.
Review these projects for relevant information and continue to check on those
that are incomplete. Web sites such as http://www.talkingquality.gov may be
helpful for this activity.
–Formulate research questions and development research plan. Based on this
information, identify variables that affect consumer choice and develop
research questions related to these variables. The reports team should use
the development of new reporting templates (listed below) as opportunities to
test research questions. Since some research questions may be more
appropriate for study in a laboratory setting, the team will propose and
implement lab studies as well.
–Plan how report development projects can fit into this framework
–Keep in mind that some questions may best be explored through lab studies
–Prepare final report in year five which details which questions were tested,
what the results were, what we now know about consumer use of this
information, what we have left to learn.

Develop and test reporting templates based on:
–Group practice level CAHPS, (one report for providers, one for consumers)
–Individual provider level CAHPS, (one report for providers, one for
consumers; if the feasibility study supports this)
–CAHPS for PWMI
–Supplemental PPO items
–CAHPS for children with special health care needs
–Spanish language CAHPS
–CAHPS in languages in which questionnaires are translated (to be determined
by the Steering Committee)

This task includes preparation of material for the CSRK, including the
development of complete instructions for analyzing data, developing
composites and inserting the data into reporting templates. All reporting
templates will need to be developed in electronic and print formats.

Develop and test reporting templates in different languages.

Develop and test reporting templates for purchasers from small or medium-
sized organizations or purchasing coalitions. In many cases, the target
audience is the benefits manager or the broker contracted to evaluate health
plans for employee benefits purposes. About 50% of Americans with health
insurance can choose among two or more health plans. For others, choice
among plans occurs at the level of the purchasing coalition, benefits manager
or other entity that makes the purchasing decision. People who make these
decisions, especially those from small or medium sized organizations, may not
fully understand health plan quality information and so may not be able to
use this information to inform their selection decision. Additionally, value-
based purchasing requires both cost and quality data. Cost data are
difficult to integrate into consumer reporting templates since this
information is specific to the purchaser. However, purchasers would
significantly benefit from having both cost and quality information
integrated in a report. Therefore, cost information could be integrated into
this reporting template.

Develop templates for reporting CAHPS data for various audiences such as
consumers, patients, health care benefits purchasers, different types of
health care providers, health care quality improvement specialists,
accreditation specialists, and researchers;

Reporting development and testing includes:

--Perform cognitive testing of questionnaires and reports for users;

--Recruit people who have poor cognitive skills (rather than relying
exclusively on convenience samples) for cognitive testing purposes,

--Conduct usability testing on products such as reports, booklets of
frequently asked questions and future editions of the CAHPS Survey and
Reporting Kit;

--Use graphic design and information design principles in the development of
CAHPS products to “let the people [who use them], working in their own
physical, social and cultural environments, accomplish their goals and tasks
effectively and efficiently” (Hackos and Redish, 1998);

Prepare reports based on discussions that recommend different approaches for
the development of reporting tools and strategies for CAHPS templates. For
example, applicants may consider gathering examples of existing documents,
reports in other than paper media and decision tools the purpose of which is
to provide health quality information to consumers, patients, and providers
and developing a list of experts from which to convene a group (or groups) to
review, discuss and critique a thoughtfully-chosen subset of these documents.
These experts should represent diverse disciplines and points of view,
including information design, graphic arts, web page design, health
communications, decision theory, marketing, social marketing, health
education and campaign development, “plain language” experts and people who
prepare materials for those with poor communication or reading skills. It
may help to include people who have developed tools in areas other than
health quality; for example, those who have developed systems to explain
retirement options and related tax consequences to employees; those who have
designed information that explains the pro’s and con’s of clinical trials
participation to candidates; those who have designed successful user manuals
and/or on-line support for the operation of mechanical or technical
equipment.

The second part of the reporting task is to develop templates in the context
of a carefully-considered program of research that will advance our knowledge
about effective ways of preparing and disseminating usable quality reports
for patients, purchasers (consumers, benefits managers and others), health
care providers and other audiences. Though we continue to see the
publication of individual studies which are well-planned and executed,
research on consumer choice as a whole is not organized or systematic and
thus has not proceeded as efficiently or effectively as it could. A planned
program of research which identifies a series of critical research questions
and builds on their results over a five year time span could improve this
situation. This research program will be implemented over the five year span
of the project. The reports team will revise it annually to incorporate
findings from the previous year and feedback from the Steering Committee and
to include development of different templates as we learn of the need for
them. At the end of CAHPS II, this program of research should result in
systematic advances in what we know about purchaser and provider use of
health quality information and should make significant contributions to that
literature.

Quality improvement studies.

CAHPS began with a primary goal of providing high-quality information which
consumers and purchasers would use to support their health plan selection
decisions. As CAHPS has evolved, there has been growing interest in
exploring the usefulness of CAHPS results for designing strategies to improve
the quality of health care from the consumer’s or patient’s perspective.
Grantees will design and conduct studies to investigate how CAHPS can be used
for quality improvement purposes.

As an example, consider the CAHPS group practice level questionnaire. As
discussed above, this questionnaire will require the development of two
reporting products: one for providers (to summarize survey results and
identify areas for improvement) and one for consumers (to summarize survey
results so that consumers can identify group practices that will best meet
their needs). Even if providers had these data, they may not understand how
to make the leap from identification of problem areas (from their patients’
point of view) to implementation of improvements. A CAHPS QI study could
work with these providers to:
--Understand how providers would like for these data to be reported to them,
--Translate study findings into recommendations for improvement or changes in
practice behaviors,
--Develop strategies and procedures designed to help change the identified
behaviors,
--Help implement these change strategies,
--Provide mid-course corrections to the strategies if necessary,
--Develop a process evaluation system to confirm that change strategies were
correctly implemented,
--Develop an outcome evaluation to determine if the changed behaviors had the
expected impact, and
--Determine how and when to communicate to patients that concerns they
conveyed through the survey were recognized and that changes were made.

As this sample task list implies, grantees may need to develop different and
new CAHPS products to develop and implement a QI study. All CAHPS
questionnaires, including the one in development to assess care given at the
group-practice level, were constructed to enable comparisons across different
plans in different settings. An intended and positive consequence of this is
that the surveys produce data that allow comparisons between a PPO in
California and a fee-for-service plan in Virginia. A negative consequence
(at least for the purpose of quality improvement) is that these data reflect
views of the plan (or health care services obtained through the plan) from
“50,000 feet” rather than at “ground-level,” i.e., that the results are not
clearly linked with reproducible steps for improvement. But the ability to
develop an effective QI plan depends on specific, detailed ground-level
information about the quality of services delivered. This suggests that,
though items from CAHPS surveys can serve as “red flags” for areas that need
to be improved, it is likely that the developer of a QI program would need to
collect additional qualitative or quantitative data to pinpoint specific
behaviors that need to be changed.

For example, if consumer ratings on the CAHPS group-provider level instrument
identify a group practice with low scores in the Communication/Interpersonal
Care domain, a researcher would need to know what specific provider behaviors
led to this assessment. Did the physician not explain the purpose of ordered
tests or test results well enough? Was the physician not sufficiently aware
of the health issues that most concern the patient? How did this affect
treatment? For responses to these and similar questions, the researcher may
need to develop a qualitative data collection protocol such as focus groups
or individual interviews. Once the researcher has a good understanding of
behaviors that prompted low ratings in this domain, he can then present the
information to the providers in that group practice and work with them to
develop strategies for change.

Like all CAHPS products, all interview protocols, focus group discussion
guides and other such materials developed for QI purposes will be available
to users free of charge.

In CAHPS II, grantees will design and implement quality improvement (QI)
studies based on CAHPS data. Each applicant to this RFA should propose a QI
study which includes at least these elements:
–Use of an existing CAHPS instrument to identify an area which requires
improvement;
–Development and use of additional qualitative or quantitative data
collection tools, if necessary;
–Design and implementation of an intervention; and
–Evaluation of the effectiveness of the intervention. The evaluation should
include a readministration of the CAHPS instrument used for problem
identification to see if the instrument is sensitive enough to detect changes
in plan/provider behavior.

GCAHPS should be completed in time for use in proposed QI studies.
Therefore, applicants are encouraged to consider either the health plan or
group level for proposed studies. Additionally, applicants may also want to
consider collaborations with Peer Review Organizations to conduct these
studies.

--Identify and establish partnerships with appropriate sites for QI studies;

--Develop strategies for using CAHPS data for quality improvement purposes.
This may involve development and testing of additional qualitative or
quantitative instruments;

--Develop strategies for translating these data into recommendations for
improvement tailored to the specific target audiences;

--Develop programs for assisting plans and care providers in using CAHPS data
to improve the quality of their programs and services;

Evaluation studies

The development of CAHPS products will require both process and outcome
evaluation studies. Grantees will perform two types of process evaluation
studies: The first (sometimes called “formative evaluation”) requires
thorough testing of materials with the target audience while materials are
still in draft status. “Materials” include report segments, reports in their
entirety, questionnaire items and draft questionnaires, elements of the
Survey and Reporting Kit, quality improvement plans and strategies and other
materials. The second type of process evaluation refers to obtaining user
feedback about how final versions of CAHPS products and procedures work in
their intended settings. We are particularly interested in process
evaluations of new products and procedures but need to collect additional
data for existing products (like the Survey and Reporting Kit) so that we can
continue to make data-based improvements to them.

In general, outcome evaluation seeks to answer the question, “Did the
intervention achieve the expected change?” In CAHPS II, applicants will need
to design and implement evaluations that demonstrate the effectiveness of: a)
QI interventions based on areas of concern identified by patients through a
CAHPS instrument and b) the effectiveness of CAHPS report in supporting
consumer choice of health plans or providers appropriate for their needs and
circumstances.

--Specify appropriate evaluation questions, identify appropriate outcome
measures and appropriate respondents for each question, and specify times at
which these questions should be asked.

--Develop an appropriate evaluation design. The design should: a) eliminate
as many threats to validity as possible; and b) result in a clear
demonstration of usefulness of the data-based reports in assisting consumer
decision-making in an open enrollment situation, as well as identification of
factors related to product effectiveness or ineffectiveness.

--Identify and use appropriate qualitative and quantitative data collection
instruments.

--Analyze data appropriately. Applicants should pay particular attention to
strategies for analysis of qualitative data, as well as the issue of
conveying information from quantitative evaluation analyses in a manner that
is understandable by and useful to personnel at sites implementing the survey
protocols.

--Develop recommendations based on evaluation data that can be used to modify
CAHPS projects or procedures. One component of the CAHPS evaluation plan
should be the development of clearly written, user-friendly recommendations
designed to promote data-based changes to products and procedures.

-Identify and develop partnerships with sites appropriate for evaluation
studies.

Examples of the kinds of questions that may be explored through this research
include the following:

We know that people need more than statistics to support their selection of a
health plan or provider. To make proper use of comparison data, people need
to understand critical differences among health care delivery systems, the
importance of a quality-based decision, and how to organize and prioritize
diverse pieces of information for each of what may be several alternatives.
How do we produce a report that covers all these bases but is brief enough to
appeal to users who will spend only minutes reviewing the data and making
their decision?

A related issue concerns people’s diverse needs for information. Is it
possible to design a single report that “layers” information to accommodate
people who want only a listing of the top three plans available to them as
well as people who want individual item scores for each variable?

About 50% of Americans who have health insurance have the choice of two or
more plans. Benefits managers, other institutional purchasers, and brokers
contracted by employers also often need to evaluate and choose among several
plans. A recent study by Iyengar and Lepper (2000) shows that increasing the
number of alternatives among which one can choose can easily overwhelm the
chooser and produce the opposite of the desired behavior. How many health
plans can consumers or purchasers evaluate before they feel overwhelmed? How
can we structure materials and the decision-making task to avoid overwhelming
users?

How do we design non-print modes of reporting (video, interactive video, web-
based reports) that are low cost and relatively easy to install and maintain?
How can these alternative materials be disseminated or made accessible to
users?

Consumers want information about health care quality and how plans and
providers compare to one another but are often intimidated by displays of
statistical data. How can we present quantitative data in a manner that
allows reliable and valid comparisons but is also as easy to understand and
user-friendly as narrative reports?

People with extremely poor cognitive skills or those who have mobility,
cognitive or sensory impairments may be unable to complete paper-and-pencil
questionnaires or to read or understand data in typical reports. How do we
produce reports that are useful to these people?

Are high users of health care services (e.g. children with special health
care needs, people with mobility impairments) likely to use quality reports
in different ways than people who are not high users? If so, how should we
design report templates for these audiences?

What are the best modes of dissemination for each of our target audiences?

Recent research has shown that quality reports that emphasize the opportunity
to protect oneself from risk are more compelling to purchasers than those
that emphasize potential gain or benefit (Hibbard, 2000). Is this finding
solid enough that we should incorporate risk-based messages into the design
of CAHPS reporting templates? How can we develop risk-based messages that
motivate consumers but do not alienate the plans whose data we are reporting?
Are there other ways of reframing key messages in quality reports that will
increase the likelihood that purchasers will pay attention to and use the
information?

Development of experiments that respond to these research questions as well
as development of quality reports that consumers will read and find useful
are tasks that require multiple types of expertise. For example, these tasks
require researchers who have: the methodological expertise to structure
rigorous experiments both in laboratory and real world settings; knowledge of
decision theory especially as it relates to consumer decisions; experience
applying social marketing principles to the design of materials for
consumers, patients and health care providers; experience with graphic
design; knowledge of research which shows how existing quality reports have
succeeded and failed; experience with design of electronic reports; expertise
in document design and design of materials in plain language; experience with
the design and evaluation of health messages and health education campaigns.
In their applications, applicants should identify the expertise required to
support their approach to report development and testing and include people
experienced in these areas as part of their project team.

3. Study Organization

Each grantee must propose a Principal Investigator as well as a co-Principal
Investigator who will be responsible for all aspects of project management
when the lead person is unavailable. These individuals should coordinate
their schedules so that one or the other is always available.

The Principal Investigator, or the person she designates, serves as the
official voice of that grantee team. Communication from the grantee team to
the AHRQ Project Officer should come from the Principal Investigator rather
than from multiple members of that team.

Applicants should propose staff with a range of experience and skill levels.
Our goal is to avoid the situation where extremely qualified or senior level
staff are performing tasks that a person with less experience could perform
just as well.

Grantees should also propose a lead staff person (and alternate) for each of
four “cross-grantee teams” (instrumentation, reports, quality improvement
studies and evaluation studies). During the course of the project, other
cross-grantee teams may be created. “Cross-grantee teams” are composed of
members of each grantee team organized to perform defined tasks. During
CAHPS II, we will require that one staff person from the SUN contract
participate in each cross-grantee team to prepare and publish call notes and
sometimes to participate in the content of the work. In CAHPS I, an
instrumentation team was organized, as well as teams for reports, evaluation,
survey and reporting kit development, user group meeting agenda development
and other purposes. The length of time cross-grantee teams are needed is
dependent upon the work being performed by the team. For example, the
instrumentation and reports teams of CAHPS I were formed early in the project
and continued to meet (by conference call and in-person) throughout its life.
Other groups are formed for shorter term goals (e.g., development of an
agenda for a user group meeting) and disband when those goals have been met.
Sometimes certain tasks can be performed most efficiently through the
development of sub-groups. For example, a sub-group of the survey and report
kit team was developed to proof-read and edit the final version. The
Objectives and Methods sections contains information about the substance of
work for four of the cross-grantee teams.

Some of the long-term cross grantee teams (such as instrumentation and
reports) have “conveners.” The convener moderates conference calls and
meetings; works with the designated SUN staff person to arrange the call,
publish a call agenda and meeting materials on the CAHPS team web site; helps
the team develop a long-term agenda including a task list and time line;
reminds the team of impending due dates; and meets with the Project Officers
on a regular basis to discuss progress toward goals, barriers and how to
resolve them.

For critical periods during CAHPS I, the Principal Investigators met weekly
by phone and sometimes in person with the AHRQ project officers,
representatives from the SUN contractor, team conveners and other people as
required. There will be two major components to the organizational structure
of the CAHPS II project–the CAHPS Steering Committee and the CAHPS Grantee
Advisory Committee. Members of the CAHPS Steering Committee will be: the
AHRQ Project Officers, at least one representative from each of the
organizations which partner with AHRQ to fund the CAHPS grants, the CAHPS
Prinicpal Investigators and their alternates. The project director of the
Survey User Network contract will attend these meetings to provide
information about CAHPS user needs but will not have an official voice in
determining direction of the work of the cooperative agreements. At this
time, CMS is expected to be a CAHPS funding partner. Other funding partners
may be identified in the future. Additional members will be added as
necessary as decided upon by the existing Committee members.

The Steering Committee will set the overall direction of the CAHPS project
and decide on project priorities from year to year. The Committee will
establish a five year plan with broad goals at the beginning of the project.
Each year it will review progress towards these goals, review emerging needs
for new or different CAHPS products and adjust project direction if
necessary. More detailed information regarding grantees’ roles on the
Steering Committee and in setting the project direction can be found in the
“Process for Setting Project Direction subsection under the Special
Requirements section below.

The CAHPS Grantee Advisory Committee will be comprised of representatives of
the key stakeholders affected by the implementation of CAHPS products. These
might include representatives of consumers/patients; health care providers,
purchasers; health plans; accrediting organizations and others. The role of
the Grantee Advisory Committee will be to advise the CAHPS grantees about the
needs and concerns of the organizations who use CAHPS products. The grantees
will select the members of this group, set meeting agendas, make all meeting
arrangements, and convene the group as often as necessary. One
representative from each organization on the Steering Committee are ex
officio members of the Grantee Advisory Committee.

Interactions among the Steering Committee, the Grantee Advisory Committee and
other CAHPS teams are discussed in the Special Requirements section.

4. Budget and Related Issues

As mentioned earlier in this RFA, the CAHPS Steering Committee will determine
the research and product development agenda on a year-by-year basis. This
approach constitutes the best strategy for meeting user needs as they evolve.
But since applicants need to prepare five-year budgets which must include
these as yet unspecified activities, we offer the following assumptions.

-For budgeting purposes, applicants need to develop budgets devoting
approximately 1/3 effort for each of the three areas listed below.

o Continued development of GCAHPS and Nursing Home CAHPS (for completion of
both GCAHPS and NH CAHPS over years 1 and 2)
o PWMI CAHPS (over years 1 and 2)

o Individual provider level CAHPS (over years 1 and 2)

-Applicants can find project summaries and a listing of remaining tasks for
completion of Nursing Home CAHPS and for GCAHPS at
http://www.ahrq.gov/fund/grantix.htm. For the other two surveys, the CAHPS
Submitter’s Kit found at the same website is a good source of information
concerning critical characteristics of questionnaires that bear the CAHPS
trademark. As mentioned in the “Objectives,” subsection above applicants
should view this document as a background resource and are not required to
abide by its recommendations or suggestions.

-The three grantees will be working collaboratively on GCAHPS, Nursing Home
CAHPS, PWMI CAHPS, and individual provider level CAHPS so each applicant
should describe their ability to perform all tasks necessary to construct a
valid survey from item generation through development of complete and easy-
to-understand administration procedures.

-Applicants should assume that the activities listed in the Timetable Section
will occur during the five year time-span of this project.

-Each grantee should budget $10,000 for each of years one and two for
consultation with one staff person from NRHCHDR for CAHPS PWMI.

-Each reporting product must be developed and tested in print and electronic
form.

-Reports and questionnaires must be tested with users who have poor cognitive
skills as well as with those who have average cognitive skills.

-One purpose of this RFA is to develop and test tools (e.g., questionnaires,
reporting templates, analytic procedures) through which users can obtain
assessments of health care quality from the consumer or patient point of view
and report them to the appropriate audiences. An additional purpose is to
stimulate research that advances knowledge in the areas of: l) consumer and
patient-based health care quality assessment, 2) reporting quality
information to various users, 3) use of assessment data for quality
improvement purposes and 4) evaluation of quality improvement programs. The
“Objectives” section of this RFA contains many examples of research questions
that could be addressed or studies that could be performed which would
advance the knowledge in one of these areas. Though applicants need not use
these particular examples of research questions or studies, they should
devote some part of the budget to the design and execution of research that
will advance the field in one or more of these four areas.

o Meetings

For each year of the contract, the appropriate grantee representative(s) will
be required to attend the following meetings:
- weekly conference calls with Principal Investigators, their back-ups and
Project Officers
- cross-grantee team meetings as necessary (in-person as necessary; assume
ten person trips)
- one Expert Panel meeting (in person)
- two Steering Committee meetings (in-person)
- two Grantee Advisory Committee meetings (in-person)
- one CAHPS User Group annual meeting (in-person)
- one NCBD Advisory Committee meeting (in-person)
- one NCBD Technical Advisory Panel (TAP) meeting
- one NCBD Executive Research Committee meeting
- five online workshops for CAHPS users

–For in-person meetings (e.g., of the Grantee Advisory Committee, cross-
grantee teams, entire consortium meetings, etc.), grantees may use free of
charge the AHRQ conference facilities currently at 6010 Executive Boulevard,
Rockville, Maryland. Use of these facilities are subject to availability.

–The grantee teams will share the cost of transportation, lodging and per
diem for Advisory Committee member attendance at each meeting. All other
costs for these meetings (obtaining space, materials preparation, etc.) will
be borne through the SUN contract.

–The grantee teams will share the cost of transportation, lodging and per
diem for Expert Panel member attendance at each meeting. All other costs for
these meetings (obtaining space service, materials preparation, etc.) will be
borne through the SUN contract.

–The CAHPS consortium holds one meeting of CAHPS user per year. Though the
SUN contractor takes the lead responsibility for this meeting, each of the
grantees will contribute to planning the content of and making presentations
at the meeting. Each grantee will be allocated two registrations free of
charge. Additional grantee team members that attend the meeting must submit
registration fees.

o Interaction with the SUN contractor

A member of the SUN contract team will be present for all meetings. The SUN
Contractor will work with grantees to schedule meetings and to prepare
materials for those meetings. Additionally, the SUN contractor will summarize
all meetings. Grantees will be required to work with the SUN contractor for
the following:
- developing, testing, and revising the CSRK
- updating other CAHPS products as necessary
- answering technical questions that arise from the technical assistance to
CAHPS users, if necessary
- providing all meeting materials, reports, and other items to the SUN
contractor in a timely manner
- serving on the NCBD Advisory Committee, TAP, and Executive Research
Committee
- participating in five online user workshops

o Reports

The following reports will be required during CAHPS II:
- Summary report for each new survey instrument developed describing any
difficulties encountered in the course of planning, implementing and
evaluating the surveys
- Five year research plan for developing and testing reporting templates
- Summary report for each new reporting template developed describing any
difficulties encountered in the course of planning, implementing and
evaluating the templates
- Final report of quality improvement study
- Final report for all evaluation studies conducted
- Non-competitive renewal proposals at the end of each grant year
- Final report at the end of year five on the advances made to reporting
quality data through these cooperative agreements
- Two summary reports per grant year describing progress
- Timeline and table of activities for each cross-grantee project (updated as
necessary as decided by the Principal Investigators and the Project
- Notes and action items from annual AHRQ site visits
- Any other interim or final reports agreed upon during CAHPS II

5. Timetable

Phase one includes years one and two of the project; Phase two includes years
three, four, and five. The activities listed in this section describe AHRQ
and CMS expectations for projects to be conducted and may change based on
input from members of the Steering Committee. Also, the “Objectives” section
of this RFA contains topics that have been identified as important by CAHPS
stakeholders but are not included on this list. The Steering Committee may
determine that studies based on these (or other) topics are worth pursuing.

Phase I Activities
-GCAHPS survey
-NHCAHPS survey
-CAHPS-PWMI survey
-PPO supplemental items
-Individual provider level CAHPS (ongoing)
-New data collection technologies
-Translations of CAHPS into other languages (ongoing)
-Development of five year research plan on reporting quality information to
various audiences
-Research on the cultural comparability of CAHPS (ongoing)
-Spanish language CAHPS reporting template
-Translation of CAHPS reporting template into one other language
-OMB clearance packages (ongoing)

Phase II Activities
-Individual provider level CAHPS
-New data collection technologies
-Translations of CAHPS into other languages
-Research on the cultural comparability of CAHPS
-GCAHPS reporting template
-CAHPS-PWMI reporting template
-Translation of CAHPS reporting template into two languages
-Quality improvement studies
-Evaluation studies
-OMB clearance packages

Annual Meetings and Reports
-Progress reports (Months one and seven of grant year)
-Grantee Advisory Committee meeting (Months two and eight of grant year)
-Steering Committee meetings (Months three and nine of grant year)
-Non-competitive renewal proposal (Month eleven of grant year)
-CAHPS User Group meeting (currently anticipated for Spring 2002; may be
scheduled in the fall for subsequent grant years)

SPECIAL REQUIREMENTS

1. Reporting requirements

As soon as possible after the award date, the Principal Investigators and
will meet in person with the AHRQ project staff and SUN contractor to review
proposed project tasks for years one and two, make any agreed upon or
necessary changes to the list, identify the subgroups required to perform
various tasks, and determine which grantee will take the lead for which
product (e.g., Spanish language version of the CAHPS reporting template) or
project (e.g., feasibility study for an individual provider level
instrument). We will also determine who the government contact person will
be for each product and project as well as whether a consortium-wide, in-
person meeting is required at that point or at some later point. Remaining
decisions will be discussed during weekly conference calls.

In order to accomplish the Agency’s mission of translating research into
practice and policy, researchers must inform the Office of Health Care
Information when articles from the study are accepted for publication in
journals. Researchers should also discuss ideas about other dissemination
and marketing efforts with the Office of Health Care Information at AHRQ so
that efforts to disseminate and translate the research may be coordinated
with the Agency in order to maximize the knowledge and application of the
research by potential users, including clinicians, patients, health care
systems and purchasers and policymakers.

The work of this RFA requires close coordination with the SUN contract
activity. One major jointly developed product is the CSRK, CAHPS Survey and
Reporting Kit. The grantees’ have principle responsibility for development
of the first draft text for the CSRK. The SUN contractor and the AHRQ Office
of Health Communications Information (OHCI) are responsible for final
formatting, editing, dissemination and actual physical production of the kit.
Because so many team members from different parts of various organizations
are involved in the CSRK and because CAHPS users have time-sensitive needs
for information in the CSRK, failure by any single part of the team to meet
agreed-upon deadlines can have disastrous consequences. So it will be
critically important for designated writers on grantee teams to meet the
deadlines to which they agree.

The CAHPS project officers will perform annual site visits with each of the
grantees. During the site visits (which usually last a day and a half), the
grantees will present evidence of progress on each of the tasks for which
they have responsibility; discuss problems, questions or other issues with
the project officers; and work with the project officers to lay out
activities for the coming year.

Process for setting project direction

The process for setting the project direction is as follows:

--Drawing upon grantee work plans, the Steering Committee establishes the
overall direction of the project with a five year plan at the beginning of
the project. The Committee will meet twice during each grant year during the
life of the project to establish or refine priorities for the coming year.
The timeframe for these meetings can be found in the Timetable section.

--All Steering Committee members will participate in discussions of project
direction and the distribution of work among the grantees. Based on these
discussions, the grantee representatives on the Steering Committee will be
responsible for jointly proposing projects along with the distribution of
work among the grantees.

--Prior to the completion of each year of the grants the CAHPS grantee teams
and the SUN contractor will consult with the CAHPS Grantee Advisory Committee
to review project progress and secure their input into priorities for the
upcoming year.

--CAHPS grantee teams will develop proposals for the upcoming year reflecting
the Steering Committee’s priorities and input from the Grantee Advisory
Committee. These proposals may suggest modifications to the these priorities.

--The Steering Committee will review proposals and decide which activities
and products will be pursued.

--The grantee teams will modify proposals as necessary to address Steering
Committee’s priorities.

See the section below entitled “AHRQ Staff Responsibilities” for
circumstances under which AHRQ may terminate or curtail the cooperative
agreement activities.

Terms and Conditions of Award

These special Terms of Award are in addition to and not in lieu of otherwise
applicable OMB administrative guidelines, HHS grant administration
regulations at 45 CFR Parts 74 and 92, 42 CFR Part 67 Subpart A, and other
HHS, PHS grants administration policy statements. Applicants should be
familiar with the Agency’s grant regulations, 42 CFR Part 67 Subpart A, and
particularly sections 67.18-67.22.

Consistent with the U18 concept defined under Mechanism of Support, the
dominant role and prime responsibility for the activity resides with the
awardee(s) for the project as a whole, although specific tasks and activities
in carrying out the studies will be shared among the awardees and the AHRQ
Program Official(s).

1. Awardee Rights and Responsibilities

o General

Awardees will have primary and lead responsibilities for the project as a
whole including:

–Survey instrument development and appropriate testing;

–Feasibility studies as necessary to support the development of new CAHPS
products;

_The grantee will write up notes and follow-up actions discussed at this
meeting for submission to the Project Officers;

–Report template development and testing;

–Translation of instruments and report templates into other languages;

–Evaluation of the cultural comparability of questionnaires;

–Design, implementation, and analysis of quality improvements studies,
including evaluation of the impact of intervention(s);

–Design, implementation, and analysis of evaluation studies;

–Development of relationships with testing sites, QI study sites, and
evaluation sites;

–Formation of the Grantee Advisory Committee, including selection of members,
convening meetings, planning meeting agendas and documenting activities;

–Provision of AHRQ access to all data generated under this award as it is
collected; and

–Cooperation with other key parties in these cooperative agreements,
including but not limited to the SUN contractor; other CAHPS grantees; and
AHRQ Project Officers and other AHRQ staff.

o Meetings

–Grantees will need to cover all meeting costs (tape-recording, etc.) for in-
person meetings with the exception of the CAHPS User Meeting and the Advisory
Committee meetings. These costs include facility costs if AHRQ conference
facilities are not used.

–The grantee teams will share the cost of transportation, lodging and per
diem for Expert Panel member attendance at each meeting. All other costs for
these meetings (obtaining space, materials preparation, etc.) will be borne
through the SUN contract.

–The CAHPS consortium usually holds one meetings of CAHPS users per year.
Though the SUN contractor takes the lead responsibility for this meeting,
each of the grantees will contribute to planning the content of and making
presentations at the meeting. Each grantee will be allocated two
registrations free of charge. Additional grantee team members that attend
the meeting must submit registration fees.

o Interaction with the SUN contractor

Grantees will be required to work with the SUN contractor for the following:
- developing, testing, and revising the CSRK
- updating other CAHPS products as necessary
- answering technical questions that arise from the technical assistance to
CAHPS users, if necessary
- providing all meeting materials, reports, and other items to the SUN
contractor in a timely manner
- serving on the NCBD Advisory Committee, TAP, and Executive Research
Committee
- participating in five online user workshops

o Reports

The following reports will be required during CAHPS II:
- Summary report for each new survey instrument developed describing any
difficulties encountered in the course of planning, implementing and
evaluating the surveys
- Five year research plan for developing and testing reporting templates
- Summary report for each new reporting template developed describing any
difficulties encountered in the course of planning, implementing and
evaluating the templates
- Final report of quality improvement study
- Final report for all evaluation studies conducted
- Non-competitive renewal proposals in Month eleven of each grant year
- Final report at the end of year five on the advances made to reporting
quality data through these cooperative agreements
- Two summary reports per grant year describing progress (Months one and
seven of grant year)
- Timeline and table of activities for each cross-grantee project (updated as
necessary as decided by the Principal Investigators and the Project
- Notes and action items from annual AHRQ site visits
- Any other interim or final reports agreed upon during CAHPS II

2. AHRQ Staff Responsibilities

The AHRQ Project Officer and other AHRQ staff will have substantial
scientific and programmatic involvement during the conduct of this activity
through technical assistance, advice, and coordination beyond the normal
program stewardship for grants. Collaboration on the development of the
testing plan, evaluation design and survey protocol will occur after the
award is made. Specifically, AHRQ's role in the cooperative agreement is to
provide technical assistance, advice, and support to the PI and to ensure
that all related AHRQ-supported projects complement one another. AHRQ will:

--Participate in all key project decisions, including but not limited to:
identification of focal consumer groups, assessment of consumer information
needs, selection of items/question modules to be tested, identification and
resolution of methodological issues;

--Work with the grantee to identify and negotiate with sites for quality
improvement and evaluation studies, with the AHRQ project officer retaining
right of final approval;

--Establish collaboration with and obtain project-related information from
other Federal agencies and programs;

--Disseminate tested question modules and other research findings as they
become available; and

--Participate in or coordinate strategy sessions with the grantees on an as-
needed basis.

--AHRQ will require prior written approval for the addition or deletion of a
participating or collaborating institution, site, or other organizational
component.

These special Terms of Award are in addition to and not in lieu of otherwise
applicable PHS grants policies and Federal regulations.
Arbitration

Any disagreement that may arise with respect to scientific or programmatic
matters (within the scope of the award), between award recipients and AHRQ
may be brought to arbitration. An arbitration panel will be composed of
three members – one selected by the Steering Committee (with the AHRQ member
not voting) or by the individual awardee in the event of an individual
disagreement, a second member selected by AHRQ, and the third member selected
by the two prior selected members. This special arbitration procedure in no
way affects the awardee’s right to appeal an adverse action that is otherwise
appealable in accordance with the PHS regulations at 42 CFR Part 50, Subpart
D and HHS regulation at 45 CFR Part 16.

Data Privacy

Pursuant to section 924(c) of the Public Health Service Act (42 USC 299c-
3(c)), information obtained in the course of any AHRQ-supported study about
an identifiable individual or entity must be treated as confidential in
accordance with any promises made or implied regarding the possible uses and
purposes of the data collection. In the Human Subjects section of the
application, applicants must describe procedures for ensuring the
confidentiality of such identifying information. The description of the
procedures should include a discussion of who will be permitted access to the
information, both raw data and machine readable files, and how personal
identifiers and other identifying or identifiable data will be restricted and
safeguarded. Plans for compliance with the Privacy Rule as 45 CFR Part 164,
if applicable, should be set forth.

The grantee should ensure that computer systems containing confidential data
have a level and scope of security that equals or exceeds those established
by the Office of Management and Budget (OMB) in OMB Circular No. A-130,
Appendix III - Security of Federal Automated Information Systems. The
National Institute of Standards and Technology (NIST) has published several
implementation guides for this circular. They are: An Introduction to
Computer Security: The NIST Handbook; Generally Accepted Principals and
Practices for Securing Information Technology Systems; and Guide for
Developing Security Plans for Information Technology Systems. The circular
and guides are available on the web at
http://csrc.nist.gov/publications/nistpubs/800-12/handbook.pdf. The application of these
confidentiality and security standards to subcontractors and vendors, if any,
should be addressed.

These and other requirements applicable to AHRQ grantees as well as important
legal rights are set out or referenced in the Agency’s grants regulations at
42 CFR 67, Subpart A. (Available in libraries and from the GPO’s website
http://www.access.gpo.gov/nara/cfr/index.html).

Rights in Data

AHRQ grantees may copyright or seek patents, as appropriate, for final and
interim products and materials, material they developed individually:
including, but not limited to, methodological tools, measures, software with
documentation, literature searches, and analyses, which are developed in
whole or in part with AHRQ funds. Such copyrights and patents will be
subject to a worldwide irrevocable Federal government license to use and
permit others to use these products and materials for government purposes.
In accordance with its legislative dissemination mandate, and subject to
statutory confidentiality protections, AHRQ purposes may include, subject to
statutory confidentiality protections, making research materials, data bases,
results, and algorithms available for verification or replication by other
researchers; and subject to AHRQ budget constraints, final products may be
made available to the health care community and the public by AHRQ or its
agents, if such distribution would significantly increase access to a product
and thereby produce public health benefits. Ordinarily, to accomplish
distribution, AHRQ publicizes research findings through the media and other
appropriate channels in coordination with AHRQ’s Office of Health Care
Information but relies on grantees to publish research results in peer-
reviewed journals and to market grant-supported products.

INCLUSION OF WOMEN, MINORITIES, AND CHILDREN IN RESEARCH STUDY POPULATIONS

It is the policy of AHRQ that women and members of minority groups be
included in all AHRQ-supported research projects involving human subjects,
unless a clear and compelling rationale and justification are provided that
inclusion is inappropriate with respect to the health of the subjects or the
purpose of the research.

All investigators proposing research involving human subjects should read the
UPDATED "NIH Guidelines on the Inclusion of Women and Minorities as Subjects
in Clinical Research," published in the NIH Guide for Grants and Contracts on
August 2, 2000
(http://grants.nih.gov/grants/guide/notice-files/NOT-OD-00-048.html).
A complete copy of the updated Guidelines is available at
http://grants.nih.gov/grants/funding/women_min/guidelines_update.htm. To the
extent possible, AHRQ requires adherence to these NIH Guidelines.

Investigators may obtain copies from the above sources or from the AHRQ
Publications Clearinghouse, listed under INQUIRIES, or from the NIH Guide Web
site http://grants.nih.gov/grants/guide/index.html.

AHRQ also encourages investigators to consider including children in study
populations, as appropriate.

AHRQ Program staff may also provide additional information concerning these
policies (see INQUIRIES).

LETTER OF INTENT

Prospective applicants are asked to submit, by September 24, 2001, a letter
of intent that includes a descriptive title of the proposed research, name,
address, and telephone number of the Principal Investigator, the identities
of other key personnel and participating institutions, and the number and
title of the RFA in response to which the application may be submitted.

Although a letter of intent is not required, is not binding, and does not
enter into consideration of any subsequent applications, the information
allows AHRQ staff to estimate the potential review workload and to avoid
conflict of interest in the review. AHRQ will not provide responses to
letters of intent.

The Letter of Intent is to be sent to Maria Wey (see address under
INQUIRIES).

APPLICATION PROCEDURES

The PHS 398 research grant application instructions and forms (rev. 5/2001)
available at http://grants.nih.gov/grants/funding/phs398/phs398.html are to
be used in applying for these grants. This version of the PHS 398 is
available in an interactive, searchable PDF format. Although applicants are
strongly encouraged to begin using the 5/2001 revision of the PHS 398 as soon
as possible, the NIH will continue to accept applications prepared using the
4/1998 revision until January 9, 2002. Beginning January 10, 2002, however,
the NIH will return applications that are not submitted on the 5/2001
version. For further assistance contact GrantsInfo, Telephone 301/435-0714,
Email: GrantsInfo@nih.gov.

State and local government applicants may use PHS 5161-1, Application for
Federal Assistance (rev. 5/96), and follow those requirements for copy
submission.

AHRQ is not using the Modular Grant Application and Award process.
Applicants for funding from AHRQ should ignore application instructions
concerning the Modular Grant Application and Award process, and prepare
applications according to instructions provided in form PHS 398.
Applications submitted in the modular format will be returned without review.

Beginning with applications submitted for the February 1, 2001 receipt date,
Institutional Review Board (IRB) approval of human subjects is not required
prior to peer review of an application. The “AHRQ Revised Policy for IRB
Review of Human Subjects Protocols in Grant Applications” was published in
the NIH Guide on September 27, 2000.
(http://grants.nih.gov/grants/guide/notice-files/NOT-HS-00-003.html).

Applications kits are available at most institutional offices of sponsored
research. They may also be obtained from the Division of Extramural Outreach
and Information Resources, National Institutes of Health, 6701 Rockledge
Drive, MSC 7910, Bethesda, MD 20892-7910, telephone (301) 435-0714, E-
mail:grantsInfo@nih.gov.

AHRQ applicants are encouraged to obtain application materials from the AHRQ
Publications Clearinghouse (see INQUIRIES).

The RFA label available in the PHS 398 (rev. 5/2001) application form must be
affixed to the bottom of the face page of the application. Type the RFA
number on the label. Failure to use this label could result in delayed
processing of the application such that it may not reach the review committee
in time for review. In addition, the RFA title and number must be typed on
line 2 of the face page of the application form and the YES box must be
marked. The RFA label is also available at:
http://grants.nih.gov/grants/funding/phs398/label-bk.pdf.

A complete, and signed, typewritten original of the application, including
the Checklist and three signed photocopies, must be submitted in one package
to:

Center for Scientific Review
National Institutes of Health
6701 Rockledge Drive, Room 1040 - MSC 7710
Bethesda, MD 20892-7710
(20817 for express/courier service)

At the time of submission, two additional copies of the application, labeled
“Advanced Copy (s)” must also be sent to:

Maria Wey
Center for Quality Improvement and Patient Safety
Agency for Healthcare Research and Quality
6011 Executive Blvd., Suite 200
Rockville, MD 20852
Telephone: (301) 594-6626
FAX: (301) 594-2155
E-mail Address: mwey@ahrq.gov

Applications must be received by November 14, 2001. An application received
after the deadline may be acceptable if it carries a legible proof-of-mailing
date, assigned by the carrier, and the proof-of-mailing is not later than 1
week prior to the deadline date. If an application is received after that
date, it will be returned to the applicant without review.

Application Preparation (for Using CMS Data)

For applications that propose to use Medicare or Medicaid data that are
individually identifiable, applicants should state explicitly in the
“Research Design and Methods” section of the Research Plan (form 398) the
specific files, time periods, and cohorts proposed for the research. In
consultation with Center for Medicare and Medicaid Services (CMS), previously
called Health Care Financing Administration (HCFA), AHRQ will use this
information to develop a cost estimate for obtaining the data. This estimate
will be included in the estimated total cost of the grant at the time funding
decisions are made.

Applicants should be aware that for individually identifiable Medicare and
Medicaid data, Principal Investigators and their grantee institutions will be
required to enter into a Data Use Agreement (DUA) with CMS to protect the
confidentiality of data in accordance with AHRQ’s confidentiality statue, 42
USC 299c3(c), the Privacy rules at 45 CFR Part 164, if applicable, and
standards set out in OMB Circular A-130, Appendix III–Security of Federal
Automated Information Systems. The use of the data will be restricted to the
purposes and time period specified in the DUA. At the end of this time
period, the grantee will be required to return the data to CMS or certify
that the data have been destroyed.

Unless AHRQ is able to negotiate exceptional arrangements, for the sole
purpose of assuring that data confidentiality is maintained, included in the
DUA is the requirement that the User agrees to submit to CMS, a copy of all
findings within 30 days of making such findings. The user further agrees not
to submit these findings to any third party (including but not limited to any
manuscript to be submitted for publication) until receiving CMS’s approval to
do so.

In developing research plans, applicants should allow time for refining,
approving, and processing any CMS data requests. Requests may take 6 months
from the time they are submitted to complete. Applications proposing to
contact beneficiaries or their providers require the approval of the CMS
Director and may require meeting(s) with CMS staff.

CMS data are provided on IBM mainframe tapes using the record and data
formats commonly employed on these computers. Applicants should either have
the capability to process these tapes and formats or plan to make
arrangements to securely convert them to other media and formats.

Questions regarding CMS data should be directed to the AHRQ program official
listed under INQUIRIES.

In carrying out its stewardship of research programs, the AHRQ, at some point
in the future, may begin requesting information essential to an assessment of
the effectiveness of Agency research programs. Accordingly, grant recipients
are hereby notified that they may be contacted after the completion of awards
for periodic updates on publications resulting from AHRQ grant awards, and
other information helpful in evaluating the impact of AHRQ-sponsored
research.

AHRQ expects grant recipients to keep the Agency informed of publications as
well as the known uses and impact of their Agency-sponsored research.
Applicants are to agree to notify AHRQ immediately when a manuscript based on
research supported by the grant is accepted for publication, and to provide
the expected date of publication as soon as it is known, regardless of
whether or not the grant award is still active.

To receive an award, applicants must agree to submit an original and 2 copies
of an abstract, executive summary, and full report of the research results in
the format prescribed by AHRQ no later than 90 days after the end of the
project period. The executive summary should be sent at the same time on a
computer disk which specifies on the label the format used (WP5.1 or WP6.0 is
preferable).

CONFERENCE FOR PROSPECTIVE APPLICANTS

AHRQ plans to present a webchat to respond to questions from prospective
applicants on September 20, 2001 at 4 p.m. EST. Participation not a
prerequisite to applying. Applicants who are interested in participating
should send a written request to Maria Wey (contact information can be found
in the INQUIRIES section). For further information, contact (301) 594-6626.

REVIEW CONSIDERATIONS

Upon receipt, applications will be reviewed for completeness by CSR and
responsiveness by AHRQ. Incomplete applications will be returned to the
applicant without further consideration. Applications that are complete but
are not responsive to the RFA will be returned to the applicant without
further consideration. Applications that are complete and responsive to the
RFA will be evaluated for scientific and technical merit by an appropriate
peer review group convened by the AHRQ in accordance with the review criteria
stated below.

As part of the merit review, all applications will receive a written
critique, and also may undergo a process in which only those applications
deemed to have the highest scientific merit will be discussed and assigned a
priority score.

All applications will be judged on the basis of the scientific merit of the
proposed project and the documented ability of the investigators to meet the
research objectives of the RFA. Although the technical merit of the proposed
project is important, it will not be the sole criterion for evaluation of a
study. Other considerations, such as importance and timeliness of the
proposed study and access to patients/data will be part of the evaluation
criteria.

General Review Criteria

The reviewers will comment on the following aspects of the application in
their written critiques in order to judge the likelihood that the proposed
research will have a substantial impact on the pursuit of these goals. Each
of these criteria will be addressed and considered by the reviewers in
assigning the overall score weighting them as appropriate for each
application. Note that the application does not need to be strong in all
categories to be judged likely to have a major scientific impact and thus
deserve a high priority score. For example, an investigator may propose to
carry out important work that by its nature is not innovative but is
essential to move a field forward.

1. Significance. If the aims of the application are achieved, how will
scientific knowledge reporting quality information be advanced? What will be
the effect of these studies on the concepts or methods that drive this field?

2. Approach. Are the conceptual framework, design, methods, and analyses
adequately developed, well-integrated, and appropriate to the aims of the
project? Are the proposed data sources appropriate and adequate? Does the
applicant acknowledge potential problem areas and consider alternative
tactics?

3. Innovation. Does the project employ novel concepts, approaches or
method? Are the aims original and innovative? Does the project challenge
existing paradigms or develop new methodologies or technologies?

4. Investigator. Is the investigator appropriately trained and well suited
to carry out this work? Is the work proposed appropriate to the experience
level of the Principal Investigator and other researchers (if any)? Is the
project (or workplace) well organized?

5. Environment. Does the scientific environment in which the work will be
done contribute to the probability of success? Do the proposed experiments
take advantage of unique features of the scientific environment or employ
useful collaborative arrangements? Is there evidence of institutional
support?

The initial review group will also examine: the appropriateness of proposed
project budget and duration; the adequacy of plans to include both genders,
children, and minorities and their subgroups as appropriate for the
scientific goals of the research and plans for the recruitment and retention
of subjects; the provisions for the protection of human and animal subjects;
and the safety of the research environment.

Special Review Criteria

The special review criteria include experience and skills in consumer
assessment and reporting of health plans and other providers. It is not
necessary that this experience be based on previous CAHPS experience.

1. Experience with similar activities outlined in the Objectives section.

2. Experience with the measurement of consumer and patient assessments of
health care plans, services and providers.

3. Experience with the needs of various audiences who may use health care
information (e.g., consumers, patients, purchasers, plans, providers, payers,
State and local-level health care organizations, etc.) and the ways in which
they may use this information (selection of plans and providers; comparison
of plan/provider performance; identification of local or multi-site quality
problems; etc.).

4. Experience with all aspects of fielding a large-scale, multi-site survey,
including but not limited to: formulation of a methodologically sound
design; performance of validity and reliability studies on large-scale survey
instruments; identification and resolution of methodological issues
associated with sample selection and sample size, and with administration of
surveys to the subpopulations of interest and within a variety of health care
delivery settings; identification and resolution of methodological issues
associated with mode of administration.

5. Experience with producing detailed survey operations manuals suitable for
use by groups with minimal as well as extensive experience in fielding large-
scale, multi-site surveys.

6. Experience with identifying and recruiting appropriate sites for survey
instrument testing, report template testing, QI studies, and evaluation
studies.

7. Experience with collaborating with, provide technical assistance or
training to, and/or monitor the progress of test sites, community-based
organizations, data supplying organizations and site representatives.

8. Experience with collaborating with government agencies in the performance
of research studies and demonstrations.

9. Experience with developing and testing understandable and useful print
and electronic reports which contain health quality data for use by a wide
range of audiences (patients, consumers, providers, health plans, national,
state and local policy makers, government agencies, etc.).

10. Experience in designing questionnaires and reports for people with poor
cognitive skills and/or poor reading skills.

11. Experience in designing, implementing and evaluating quality improvement
programs in health care settings.

12. Experience in designing and implementing process and outcome
evaluations, including: specification of evaluation questions and outcome
measures; use of quantitative and qualitative data collection methods; data
analysis; development of data-based recommendations and use of the
recommendations to improve products and procedures.

13. Experience in participating as a productive member of a multi-
organization team over a lengthy performance period in the development of
high-quality products and procedures for their use.

AWARD CRITERIA

Applications will compete for available funds with all other applications
under this RFA. The following will be considered in making funding
decisions: 1) quality of the proposed project as determined by peer review;
2) availability of funds; 3) program balance with respect to types insurance
structure and organizational delivery system features, enrollee versus
community studies, populations studied, geography including rural/urban mix,
regional representation, and targeting understudied areas with unmet
evaluation needs; 4) compatibility with other awardees with respect to
cooperative activities; and 5) policy relevance.

INQUIRIES

Copies of the RFA and copies of the grant application form PHS 398 (rev.
5/2001) are available from:

AHRQ Publications Clearinghouse
P.O. Box 8547
Silver Spring, MD 20907-8547
Telephone: 1-800-358-9295
TDD Service: 888-586-6340

The RFA is also available on AHRQ’s Web site, http://www.AHRQ.gov, and
through AHRQ InstantFAX at (301) 594-2800. To use InstantFAX, you must call
from a facsimile (FAX) machine with a telephone handset. Follow the voice
prompt to obtain a copy of the table of contents, which has the document
order number (not the same as the RFA number). The RFA will be sent at the
end of the ordering process. AHRQ InstantFAX operates 24 hours a day, 7 days
a week. For comments or problems concerning AHRQ InstantFax, please call
(301) 594-6344.

AHRQ welcomes the opportunity to clarify any issues or questions from
potential applicants who have read the RFA. Written and telephone inquiries
concerning this RFA are encouraged. Direct inquiries regarding programmatic
issues, including information on the inclusion of women, minorities, and
children in study populations to:

Direct inquiries regarding fiscal and eligibility matters to:

Al Deal

Grants Management Specialist
Agency for Healthcare Research and Quality
2101 East Jefferson Street, Suite 601
Rockville, MD 20852
Telephone: (301) 594-1843
FAX: (301) 594-3210
Email: adeal@ahrq.gov

AUTHORITY AND REGULATIONS

This program is described in the Catalog of Federal Domestic Assistance,
Number 93.226. Awards are made under Title IX of the Public Health Service
Act (42 USC 299-299c-7) as amended by P.L. 106-129 (1999). Awards are
administered under the PHS Grants Policy Statement and Federal Regulations 42
CFR 67, Subpart A, and 45 CFR Parts 74 or 92. This program is not subject to
the intergovernmental review requirements of Executive Order 12372 or Health
Systems Agency review.

The PHS strongly encourages all grant and contract recipients to provide a
smoke-free workplace and promote the non-use of all tobacco products. In
addition, Public Law 103-227, the Pro-Children Act of 1994, prohibits smoking
in certain facilities (or in some cases, any portion of a facility) in which
regular or routine education, library, day care, health care or early
childhood development services are provided to children. This is consistent
with the PHS mission to protect and advance the physical and mental health of
the American people.

REFERENCES

Website references: SUN Contract - http://www.cahps-sun.org,
NCBD - ncbd.cahps.com, Talking Quality - http://www.talkingquality.gov.

CAHPS Documents (available on the AHRQ website): CAHPS Survey and Reporting
Kit, GCAHPS Final Report, CAHPS-PWMI Expert Panel Meeting Summary, NH-CAHPS
Summary, Frequently Asked Questions about CAHPS booklet, CAHPS Submitter’s
Kit, Report of Input from Stakeholders on Design of CAHPS II, SUN contract
scope of work.

Hackos JT, Redish JC. User and Task Analysis for Interface Design. 1998; John
Wiley & Sons, Inc.:New York, NY.

Hibbard JH, Harris-Kojetin L, Mullin P, Lubalin J, Garkinkel S. Increasing
the impact of health plan report cards by addressing consumers' concerns.
Health Aff. 2000 Sep-Oct;19(5):138-43.

Iyengar SS, Lepper MR. When choice is demotivating: can one desire too much
of a good thing? J Pers Soc Psychol. 2000 Dec;79(6):995-1006.

Shaller DV, Sharpe PS, Rubin RD. A national action plan to meet health care
quality information needs in the age of managed care. JAMA.
1998;279(16):1254-1258.

Return to Volume Index

Return to NIH Guide Main Index

	Office of Extramural Research (OER)			National Institutes of Health (NIH) 9000 Rockville Pike Bethesda, Maryland 20892			Department of Health and Human Services (HHS)
Note: For help accessing PDF, RTF, MS Word, Excel, PowerPoint, RealPlayer, Video or Flash files, see Help Downloading Files.