Detailed Information on the
Health - Data Collection and Dissemination Assessment

Produce additional data for MEPS tables compendia. Specifically, MEPS will add tables on children's use of preventive health services for 2006 by the end of FY 2008, and for 2007 by the first quarter of FY 2009.

AHRQ has begun to address management deficiencies by adopting performance-based contracts that require superior performance toward achieving established goals.

CAHPS - Determine the feasibility of adopting a performance-based contract for the recompeted CAHPS contract in 2007.

Develop annual targets for long-term measures for out-years

2007 - Submit CAHPS RFC to AHRQ Contract Office and review to make determination regarding whether it could be made a performance-based contract.

Increase the number of consumers for whom CAHPS data are available by developing and placing in the public domain surveys that will cover clinicians, nursing homes, and home health.

Update 2008 targets.

Produce additional data of MEPS Household component for state and local area estimates to enhance the usability of data to the health care community.

Three new organizations will use HCUP/Quality Indicators.

Develop updates for new versions of the CAHPS Clinician and Group Survey.

Measure: Number of additional organizations per year that use Healthcare Cost and Utilization Project (HCUP) databases, products or tools in health care quality improvement efforts.

Explanation:As a non-regulatory agency, AHRQ relies heavily on the public to provide feedback on its use of and experience with AHRQ and HCUP products and tools. Through technical support contracts designed to assist organizations in using these data products and tools, AHRQ is able to collect (as a by-product of these activities) information on the number of organizations using HCUP and Quality Indicator tools in their quality improvement and public reporting efforts. These organizations self-report as having used the products and whether improvement has occurred. These organizations include hospitals, hospital systems, hospital associations, state data organizations, employer/business coalitions, and a major international organization (the Organization for Economic Cooperation and Development, OECD) which collects and compares data among dozens of developed countries. By increasing the number of organizations using these tools, attention and resources are being allocated towards improving quality issues within such organizations. The measures for future years are ambitious targets particularly given a fixed HCUP/QI budget which has remained constant for the last seven years. During the course of the project, inflation has affected labor and other project costs needed to achieve project goals. For example, the cost of data has increased by approximately 40% over the last four years.

Measure: The number of consumers who have access to customer satisfaction data from the Consumer Assessment of Healthcare Providers and Systems (CAHPS) to make health care choices.

Explanation:This metric indicates the extent to which consumers have access to consumer satisfaction information to help them identify health care providers to improve the quality and safety of the care they receive. As the CAHPS customer satisfaction survey program continues to develop new surveys, more organizations continue to participate in these surveys. Some organizations require health care providers or health insurance plans to participate; for example, the National Committee for Quality Assurance (NCQA) requires the use of the Health Plan CAHPS customer satisfaction survey for managed care health insurance plans who want to obtain NCQA certification. The Centers for Medicare and Medicaid Services (CMS) and many state Medicaid programs also require health insurance plans who serve Medicare or Medicaid beneficiaries to field and report data from this same Health Plan CAHPS survey; and OPM requires health insurance plans offered to Federal employees to collect and report this same data. CMS does not technically require hospitals to use the Hospital CAHPS customer satisfaction survey, but any hospital who fails to report this data to CMS will not receive their full "market basket" inflationary increase for service reimbursement. These and other organizations publicly report the CAHPS survey data they collect. This measure is tabulated by keeping track of the number of organizations using CAHPS survey data and asking these organizations for estimates of their "reach" (that is, how many people have the opportunity to consult these data before selecting a health plan, hospital, care provider or other health-related service or organization).

Measure: The number of months required to produce Medical Expenditure Panel Survey (MEPS) data files (i.e. point-in-time, utilization and expenditure files) for public dissemination following data collection.

Explanation:Releasing MEPS data to the public in a timely manner is important because it allows researchers and others access to the most current data for their projects. Between the end of data collection and publication, the data collection database needs to be transformed into analytic files. Some variables need coding. Analytic variables for sampling weights need to be prepared and the weights need to be produced. Some measures (expenditures) require augmenting household reports with medical provider reports. These data require matching and imputation. Finally, the data files are reviewed for respondent protection of confidentiality.

Measure: The average number of field staff hours required to collect data per respondent household for the Medical Expenditure Panel Survey (MEPS).

Explanation:This measure tracks the average number of field staff hours spent collecting data per household interviewed for the Household Component of the Medical Expenditure Panel Survey (MEPS). MEPS is a longitudinal survey in which cohorts or "panels" of households are interviewed 5 times over a two-and-a-half-year period. This measure tracks the average number of hours required to conduct the first of the five rounds of interviews per household interviewed, since this round typically takes the most number of hours to complete. Because the budget for the MEPS survey ($55.3 million) does not increase from year to year, any improvements in this measure will indicate that the MEPS field staff are able to collect data in a shorter average number of hours -- thus achieving efficiencies. Although the activities conducted during field staff hours include the actual interviewing of households in their homes, the duration of interviews is not expected to decrease, and will be monitored to ensure interviewers are following proper protocol and not skipping questions (duration of interviews is automatically recorded by the laptop software used by field staff while interviewing respondents). All other field staff activities associated with collecting household data will be encouraged to be completed in a shorter number of hours, including: setting up appointments with households; negotiating entrance into apartments and gated communities; refusal conversion; travel time; processing of paper documents such as medical provider permission forms; finding respondents who have moved; and having conference calls with field supervisors on case assignments. The number of hours spent on these activities is logged by field staff in an automated system. To reduce the number of hours spent collecting household component data, AHRQ will work with its contractors to identify additional efficiencies in the travel and administrative tasks performed by field staff. Quality is monitored in several ways: 1) Validation interviews are conducted for a sample of respondents, in which questions concerning the interviewing process are asked -- such as the date the interview occurred, how long it lasted, and what transpired during the interview -- to ensure that the interview was conducted according to protocol; 2) Response rates are monitored to ensure that they stay high; and 3) the duration of interviews are tracked to ensure that interviewers are following proper protocol and not skipping questions during the interview. Targets are expected to improve over 2007's actuals because in 2007 interviewers began using upgraded software called the Computer-Assisted Personal Interview (CAPI) data collection system, instead of a rudimentary DOS-based system. Targets for 2010 and 2011 increase at a slower rate than previous years because fewer potential economies are expected to be available to be implemented, since many economies will have already been attained.

Is the program purpose clear?

Explanation: The Public Health Service Act (PHS) states the purpose of AHRQ "is to enhance the quality, appropriateness, and effectiveness of health services, and access to such services through the establishment of scientific research and the promotion of improvements in clinical and health system practices." Such activities include: 1) "conduct[ing] a survey to collect data on a nationally representative sample of the population on the cost, use and, ... quality of healthcare, including the types of health care services Americans use, their access to health care services, frequency of use, how much is paid for the services used, the source of those payments, the types and costs of private health insurance, access, satisfaction, and quality of care..." (MEPS); 2) developing tools to collect data "the costs and utilization of, and access to health care..." (HCUP); and 3) "develop[ing] survey tools for the purpose of measuring participant and beneficiary assessments of their health care..." (CAHPS).

Evidence: Reauthorized 2000-2005 (P.L. 106-129) under the Healthcare Research and Quality Act, which amends Title IX of the Public Health Service Act (www.ahrq.gov/hrqa99.pdf).

Does the program address a specific interest, problem or need?

Explanation: The availability of national, representative data on the status of the health care delivery system and its costs and utilization are limited. Health care is both a national and local issue. As a result of HCUP, CAHPS, and MEPS data collection and dissemination tools researchers, institutions, and policy officials have ready access to a wide breath of national and state level data to accurately reflect the status of the health care system and expenditures for accessing/providing care in the system.

Evidence: 1) www.ahrq.gov/data/hcup/ 2) www.meps.ahrq.gov/ 3) www.ahrq.gov/qual/cahps/

Is the program designed to have a significant impact in addressing the interest, problem or need?

Explanation: Some of the data collected/disseminated for MEPS and HCUP are similar to that of data provided by the National Center for Health Statistics funded by the CDC. However, the MEPS sample sizes and HCUP databases are larger and more detailed. More complex and representative questions from researchers/policy officials may be answered using AHRQ's tools. AHRQ's tools are used to standardize information so that it may be compared across states and health care delivery systems. The MEPS Health Insurance Component Survey provides data regarding establishments' expenditures; this information is not collected elsewhere across government. MEPS also collects longitudinal data from households, information about linkages between employment and insurance, and medical expenditure and utilization data in an event-by-event manner. NCHS conducts snapshot household and person-based data.

Evidence: HCUP's standardized databases include nationwide inpatient samples and 29 state inpatient databases, 15 state ambulatory surgery databases, 7 pilot emergency department database, and the Kids' inpatient database. MEPS survey instruments are designed to collect national data on medical expenditures for more than 9,000 households; medical provider expenses for more than 23,000 physicians, 9,000 pharmacies, and 11,000 hospitals. NCHS documents the health status of the population and of important subgroups, describes our experiences with the health care system, monitors trends in health status and health care delivery, identifies health problems, and supports biomedical and health services research.

Is the program designed to make a unique contribution in addressing the interest, problem or need (i.e., not needlessly redundant of any other Federal, state, local or private efforts)?

Explanation: These instruments collect and disseminate large amounts of data that are more nationally representative than other tools. As a result, researchers/policy officials can use these data to capture uncommon conditions/procedures and population subgroups. These tools are designed to fill gaps in the availability of private sector, nationally collected and disseminated data.

Evidence:  

Is the program optimally designed to address the interest, problem or need?

Explanation: These instruments collect and disseminate large amounts of data that are more nationally representative than other tools. As a result, researchers/policy officials can use these data to capture uncommon conditions/procedures and population subgroups. These tools are designed to fill gaps in the availability of private sector, nationally collected and disseminated data.

Evidence:

Does the program effectively articulate potential public benefits?

Explanation: These programs do not effectively articulate potential public benefits. For the most part, the data from these tools are available for discrete groups (researchers/policy officials/Medicare beneficiaries/specific institutions) and not the general public. AHRQ has developed fact sheets for some of these tools, which indicate the inclusion of these data in Federal Employees Health Benefits Program materials (CAHPS), materials provided to Medicare beneficiaries/specific institutions (CAHPS), and papers provided to policy officials to make decisions on program changes (MEPS). These vehicles tend to provide access to but not necessarily use by these groups. These data are not used in a wide-scale way by the general public, likely because of the lack of a clear and effective explanation of the public benefit.

Evidence: 1) HCUP/Quality Indicators Fact Sheet (www.qualityindicators.ahrq.gov/data/hcup/prevqifact.htm). 2) CAHPS Fact Sheet (www.ahrq.gov/qual/cahpfact.htm). 3) Advantage of MEPS (www.ahrq.gov/data/mepsadva.htm).

If an industry-related problem, can the program explain how the market fails to motivate private investment?

Explanation: In the mid-1990s, attempts to encourage the private sector to build multi-state databases were not successful in large part due to lack of profit associated with such a project, and because of data confidentiality issues. Private organizations have few incentives to develop tools for assessment of health plans other than the type they manage (HMO vs. fee-for-service). MEPS has taken on the role to fill the gap left by market failure and makes the data available to the public.

Evidence:  

Does the program have a limited number of specific, ambitious long-term performance goals that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: OMB and AHRQ recently developed ambitious long-term outcome goals that link to the mission of the program. In some cases baseline data are to be determined, but AHRQ believes these data can be collected.

Evidence: AHRQ's newly developed long-term outcome goals are: 1) Data from the MEPS survey will be available within 12 months of completion of the survey by 2008 and 2) At least 5 organizations (e.g., federal organizations, state organizations, private associations, health plans, employers, employer groups) will use HCUP databases, products, or tools, to improve statewide health care quality for their constituencies by 10% as defined by the AHRQ Quality Indicators by 2010.

Does the program have a limited number of annual performance goals that demonstrate progress toward achieving the long-term goals?

Explanation: AHRQ's annual GPRA plan includes annual goals, many of which are process-oriented. OMB and AHRQ recently developed discrete, quantifiable, and measurable annual performance goals that demonstrate progress toward achieving the long-term goals.

Evidence: AHRQ's newly developed annual goals are: 1) "Point-in-time" data from the Household Survey and Insurance Component tables will be available within 12 months of collection, 2) Data from the Household Survey reflecting expenditures will be available within 12 months from the end of Medical Provider Component data collection, and 3) Develop implementation strategy for long-term goal related to HCUP databases, products, or tools to improve health care quality for organizations' constituencies.

Do all partners (grantees, sub-grantees, contractors, etc.) support program planning efforts by committing to the annual and/or long-term goals of the program?

Explanation: The MEPS contracts for data collection and production specify the same data release expectations as their performance goals. With some contracts these measures are a part of their performance based contract plans. HCUP requires contractors to commit to tasks contributing to performance goals and file reports by phone weekly, and written monthly and annual reports. CAHPS work plans include statements of tasks and sub-tasks required to achieve specific goals, identification of staff with responsibility for that activity, and dates by which tasks and sub-tasks must be completed. Project Officers also use these documents to measure progress toward completion of activities as they perform their annual site visits with each grantee. If progress is insufficient, the cooperative agreement may be terminated.

Evidence: 1) Work plan tasks and subtasks. 2) Grantee progress reports. 3) Grantee financial status reports.

Does the program collaborate and coordinate effectively with related programs that share similar goals and objectives?

Explanation: There are few programs with similar goals and objectives. AHRQ, as part of its MEPS activities, is a member of the Interagency Committee on Employment-Related health insurance surveys which considers and recommends collaborative efforts that will improve employment-related data collection activities. AHRQ also collaborates with sister agencies across HHS on HCUP-related items to provide evidence on cost and quality of particular treatments. The CAHPS team also collaborates with non-governmental agencies. Packard Foundation had funded a questionnaire to assess care given to children with special health care needs; CAHPS was also working on a similar questionnaire. To avoid duplication, AHRQ partnered with the Packard Foundation team and the National Committee for Quality Assurance to develop the Child and Adolescent Healthcare Measurement Initiative, a single instrument.

Evidence:  

Are independent and quality evaluations of sufficient scope conducted on a regular basis or as needed to fill gaps in performance information to support program improvements and evaluate effectiveness?

Explanation: MEPS surveys began in 1977. In 1987 a National Medical Expenditure Survey Planning Contract and several IG evaluations reviewed components of the MEPS portfolio. The evaluations found that there were significant time lags between the survey and the time data were released for public use, as well as inefficiencies in program design. Because of these evaluations, AHRQ conducted an extensive management and program restructuring of MEPS that improved the structure of the survey as well as the time it takes to release the data. Other evaluations of the new MEPS and HCUP also occur.

Evidence: 1) 1987 Report on NMES Planning Contract. 2) Office of the Inspector General: Evaluation of the 1987 NMES. 3) HHS Evaluations of the Design of the 1987 NMES. 4) Reports on components of the 1996 MEPS. 5) Evaluation of HCUPnet and Central Distributor 2002.

Is the program budget aligned with the program goals in such a way that the impact of funding, policy, and legislative changes on performance is readily known?

Explanation: AHRQ's OMB budget justification and Congressional justification display the AHRQ budget. However, when AHRQ submits its budget request to the Department for review, the annual targets are adjusted according to the funding level requested and/or the final funding level provided by the Department. Budget requests and funding level decisions are not made based on achieving the established long-term and annual performance goals. In addition, AHRQ does not have in place a model/mechanism that allows it to determine per unit cost of service to help in adjusting its budget or program targets accordingly.

Evidence: 1) OMB Budget Justification submitted each Fall. 2) Congressional Justification submitted each February with the President's Budget.

Has the program taken meaningful steps to address its strategic planning deficiencies?

Explanation: AHRQ has acknowledged the multiple difficulties of tracking budgetary expenditures along with tying these expenditures to actual program performance. AHRQ plans, using budgeted FY 2003 resources, to begin to deploy a reporting module (phase I) to the activity areas allowing them to view and track their own budgets. Phase II will allow the activity areas to interconnect appropriate areas of the Agency's planning system with the budget system through a set of common fields, and finally, the GPRA program goals. The ultimate goal of this project will be targeted integration of the existing Agency planning database with the budget database system, allowing Agency leadership to easily identify, and flag for action those program areas that are not meeting their GPRA goals.

Evidence:  

Is evaluation of the program's continuing relevance to mission, fields of science, and other "customer" needs conducted on a regular basis?

Explanation: In the mid-1990s, attempts to encourage the private sector to build multi-state databases were not successful and lead to internal reviews of program/activity mission and relevance. MEPS was overhauled and regular evaluations of these programs/activities are being conducted.

Evidence:  

Has the program identified clear priorities?

Explanation: Overall, the priority for these activities is to collect and disseminate timely data on cost and utilization of health care services, as well as to make available feedback on customers' perception of the care they received and their health plans. Furthermore, through communication with users, workshops, meetings, and planned customer surveys MEPS assesses/will assess community needs. HCUP routinely solicits outside feedback and guidance through the annual meeting with the 29 HCUP partner states and stakeholder meetings. AHRQ program staff also review performance goals on an annual basis and prioritize these goals in accordance with AHRQ's mission. The AHRQ reauthorization also states the purpose of the agency and thus the intent of these activities.

Evidence: 1) Reauthorized 2000-2005 (P.L. 106-129) under the Healthcare Research and Quality Act. 2) www.ahrq.gov/hrqa99a.htm. 3) Congressional Justification. 4) Annual GPRA Plan.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: AHRQ regularly collects data on the annual performance goals established in the GPRA plan and grantees and internal efforts to meet these goals. CAHPS work plans include statement of tasks and sub-tasks required to achieve specific goals, identification of staff with responsibility for that activity, and dates by which tasks and sub-tasks must be completed. Project Officers also use these documents to measure progress toward completion of activities as they perform their annual site visits with each grantee. If progress is insufficient, the cooperative agreement may be terminated. Similar mechanisms are in place for the other programs.

Evidence: 1) Work plan tasks and subtasks. 2) Grantee progress reports. 3) Grantee financial status reports.

Are Federal managers and program partners (grantees, subgrantees, contractors, etc.) held accountable for cost, schedule and performance results?

Explanation: The Agency's strategic plan guides the overall management of the agency. Each Office and Center has an individual strategic plan and annual operating plan. Cost, schedule and performance are part of the performance plans of the AHRQ management, including Division, Center, and Agency Directors. The annual operating plan identifies those things that contribute to AHRQ achieving its performance goals and internal management goals. These factors are incorporated into each employee's annual performance plan/review. At the end of each year, the Office and Center Directors review accomplishments in relation to the annual operating plans in preparation for drafting the next year's plans. The results of these reviews contribute significantly to Office and Center performance reports. Some managers performance plans also take into consideration their staffs performance in managing program operation. In addition, contracts are performance-based.

Evidence: Program managers' performance contract.

Are all funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: All appropriated funds are obligated in accordance with the annual operating plans, formulated for obligation and outlay on a quarterly basis.

Evidence: 1) Estimated obligations by quarter in apportionments for FYs 1999-2002. 2) Actual obligations by quarter for FYs 1999-2002.

Does the program have incentives and procedures (e.g., competitive sourcing/cost comparisons, IT improvements) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The programs' operating plans do not include efficiency and cost effectiveness measures and targets that address such things as per unit cost or some other measures directly linked to the activities of the program.

Evidence: 2002 Operations Plan Goals.

Does the agency estimate and budget for the full annual costs of operating the program (including all administrative costs and allocated overhead) so that program performance changes are identified with changes in funding levels?

Explanation: Although AHRQ is able to provide the cost of unit service for the MEPS activities, this PART also addresses HCUP and CAHPS. AHRQ does not have in place a model/mechanism that allows it to determine per unit cost of service for CAHPS and HCUP. Therefore, AHRQ does not adjust its budget or program targets accordingly. Furthermore, although AHRQ's OMB budget justification and Congressional justification display the AHRQ budget, when AHRQ submits its budget request to the Department for review, the annual targets are adjusted according to the funding level requested and/or the final funding level provided by the Department. Budget requests and funding level decisions are not made based on achieving the established long-term and annual performance goals.

Evidence:  

Does the program use strong financial management practices?

Explanation: Because the Department prepares audited financial statements for its largest components only, AHRQ's financial statements are not audited. In 2002, AHRQ engaged Clifton Gunderson LLP for technical support consultation and analysis for certain financial management practices.

Evidence:  

Has the program taken meaningful steps to address its management deficiencies?

Explanation: Programs are adopting performance-based contracts which require superior performance by the contractor to receive the full project fee. Other contracts are awarded on a competitive basis or sole sourced to capable entities with proven results.

Evidence:  

Does the program allocate funds through a competitive, merit-based process, or, if not, does it justify funding methods and document how quality is maintained?

Explanation: AHRQ announces research grant opportunities through program announcements (PA) and requests for applications (RFA). Contract opportunities are announced through a similar process. Grant applications are reviewed for scientific and technical merit by a peer review group with appropriate expertise. Funding decisions are based on the quality of the proposed project, availability of funds, and program balance among research areas. Contracts are awarded using a similar process.

Evidence:  

Does competition encourage the participation of new/first-time performers through a fair and open application process?

Explanation: HHS' policies create a fair and open competition including making project documents and products available for review by new bidders. Also, the PAs and RFAs encourage the development of new ideas and research questions that will benefit the field.

Evidence: 1) Requests for Proposals. 2) Requests for Information. 3) Statements of Work.

Does the program adequately define appropriate termination points and other decision points?

Explanation: Major tasks and expansion plans have interim steps that allow for review and evaluation to permit appropriate termination or progression. Contracts contain option years so that the program can extend its activities for defined periods of time. Each year, as part of the AHRQ work plan development, activities are assessed for their continuing utility.

Evidence: Operation Plan.

If the program includes technology development or construction or operation of a facility, does the program clearly define deliverables and required capability/performance characteristics and appropriate, credible cost and schedule goals?

Explanation: HCUP and MEPS involve certain forms of technology development. HCUP developed a series of interactive databases and MEPS uses a computerized data collection process. Contracts are performance-based. Project Officers also use these documents to measure progress toward completion of activities as they perform their annual site visits with each grantee. If progress is judged as insufficient, the cooperative agreement may be terminated. Similar mechanisms are in place for the other programs.

Evidence: Contractor Progress Reports.

Has the program demonstrated adequate progress in achieving its long-term outcome goal(s)?

Explanation: AHRQ has contributed to the overall availability of longitudinal national and state level data. AHRQ has already begun focusing its efforts toward improving the availability of timely data through the redesign of its MEPS program, as a result of findings about deficiencies in the program. More outcome-oriented goals need to be developed regarding HCUP and CAHPS activities.

Evidence: The time it takes to have MEPS data available for use and analysis have improved from 1997 to date. AHRQ continues to strive for improved performance overtime.

Does the program (including program partners) achieve its annual performance goals?

Explanation: AHRQ has maintained the timeframe of 12 months to have point-in-time data available. AHRQ has also improved the time between completing data collection efforts to data dissemination. More annual goals need to be developed for HCUP and CAHPS activities.

Evidence: The time it takes to have MEPS point-in-time data available for use and analysis has been maintained at 12 months. AHRQ continues to strive for improved performance overtime for Household Survey data.

Does the program demonstrate improved efficiencies and cost effectiveness in achieving program goals each year?

Explanation: The average cost of these research collection and dissemination tools has decreased as AHRQ has realized cost efficiencies.

Evidence: HCUP average costs of database development is $43,500; the estimate projects $46,000. MEPS costs range from $3,300 per case for household data to $9,351 for medical provider/pharmacies data.

Does the performance of this program compare favorably to other programs with similar purpose and goals?

Explanation:  

Evidence:  

Do independent and quality evaluations of this program indicate that the program is effective and achieving results?

Explanation: The HCUP evaluation of quality and how representative the National Inpatient database (1995-2000 data) indicated that HCUP is effective in both areas. An evaluation of HCUPnet and its Central Distributor released in 2002 also drew the same conclusions. The series of MEPS evaluations found that the program needed to be redesigned and thus a massive reform effort was conducted. A customer satisfaction survey is currently undergoing final signoff.

Evidence:  

If the program includes construction of a facility, were program goals achieved within budgeted costs and established schedules?

Explanation:  

Evidence: