 |
Office of the Assistant Secretary for Planning and
Evaluation |
|
Policy Information Center |
The eight Agencies of the
Public Health Service (PHS) and the Office of the Assistant Secretary for
Health (OASH) maintain their own evaluation programs--including the functions
of evaluation planning and policy review; quality assurance through technical
review; project coordination and management; dissemination of reports; and
utilization of results. This chapter provides an agency-by-agency summary of
the evaluation activities. Each section begins with an overview of the Agency's
evaluation program, describing its philosophy, policies, and procedures. Next
are summaries of the major evaluations completed in fiscal year (FY) 1994 and
the evaluations currently in progress. Finally, there is a discussion of new
directions for the Agency's evaluations, including priorities for future
program and policy evaluation projects. Abstracts and contact persons for all
evaluations completed in FY 1994 are presented at appendix A, organized by
Agency. Appendix B lists all evaluations in progress, by Agency. Review
criteria used by the special panel of senior editorial advisors for assessing
evaluations can be found in appendix C.
Agency for Health Care Policy and Research
MISSION: To generate and disseminate information that improves
the health care system.
AHCPR Evaluation Program The Agency for
Health Care Policy and Research (AHCPR) has designed a portfolio that responds
to three types of evaluation needs:
To address these evaluation needs, an evaluation component
is built into each AHCPR project. The evaluation mechanisms, which vary
depending on the project, include special evaluation studies; peer review of
grant applications and subsequent review by the National Advisory Council for
Health Care Policy, Research, and Evaluation; the User Liaison Program (which
provides information on the value of our research to State policymakers); and
other evaluation efforts such as focus groups to provide feedback on AHCPR
products and to provide baseline information to assist in the design of future
evaluation projects.
AHCPR received delegated authority to review evaluation
projects in 1992 and, as part of delegated review, established a formal
structure to review proposed evaluation projects that coincides with the review
of concepts for all other Agency projects. Thus, review of proposed evaluation
contracts is integrated into the Agency's formal planning and budget
process.
AHCPR has implemented a two-tier process for reviewing
evaluation projects eligible for 1 percent set-aside funds. The process begins
with an executive evaluation review to assess the policy relevance and relative
priority of proposals submitted. This review is conducted by the Administrator
and senior staff. The second, or technical, review assesses policy-relevant
project proposals for feasibility, soundness of design, costs, potential
importance of the findings, and relation to ongoing evaluation activity. This
second review is conducted by the Task Force on AHCPR Evaluation Projects, a
group consisting of one individual with evaluation expertise from each Office
and Center.
Summary of FY 1994 Evaluations During
fiscal year (FY) 1994, AHCPR completed six evaluations on two major topics:
examining the process of developing clinical practice guidelines, and
monitoring and evaluating health care delivery.
One of the statutory responsibilities of AHCPR is to
arrange for the development, review, and revision of clinical practice
guidelines. AHCPR-supported guidelines may be used by physicians, educators,
and health care practitioners to help determine how diseases, disorders, and
other health conditions can most effectively and appropriately be prevented,
diagnosed, treated, and managed. The development of guidelines relates to
AHCPR's strategic plan goal to determine what works best in clinical practice
and to translate that knowledge into clinical decisionmaking. The first three
AHCPR-supported evaluation studies completed in FY 1994 focused on
strengthening the methodology underlying the guidelines by examining the
strengths and limitations of current methods, and by identifying improved
approaches and how they might be implemented.
The first project examined the methods used to conduct
peer and pilot reviews for six of the guidelines. Peer and pilot review methods
are used before a guideline is released to ensure its methodological soundness
and scientific accuracy as well as to obtain information on its clarity,
organization, and format. The study culminated in several recommendations for
improving the quality of peer and pilot review.
Two other studies focused on methods for incorporating
treatment cost analyses into the guideline development process. One examined
the methodologies and data sources used in cost studies for eight of the
guidelines and examined some of the difficulties in making cost estimates about
various guideline treatment options. The other focused on identifying databases
that could be used to provide information to generate cost estimates. This
study found that no single database can be used to estimate the costs of
recommendations in clinical practice guidelines. It is necessary to assemble
data from several sources to estimate costs because claims databases sometimes
lack information on certain services, and there are significant questions about
the accuracy of data in various databases.
Information from these three studies has been used to
inform two AHCPR reports to Congress on methods for developing guidelines and
methods for including cost information in the guidelines. The findings have
also been used to formulate a more uniform approach to guideline development
based on what works best from the various approaches used to date.
The second area of emphasis among the AHCPR evaluations
completed in FY 1994 relates to the Agency's strategic plan goal to monitor and
evaluate health care delivery. Critical to fulfilling this goal is research
that can be useful to policymakers by providing a greater understanding of the
effects of emerging delivery systems and changes in the health care market
structure. One such project focused on methods and issues related to the
implementation and integration of advanced clinical information systems. The
report presented a design for studying integrated health care information
systems, including issues such as study feasibility, site selection, research
design, and a data collection plan.
Another study examined inappropriate extension of
hospitalization and the role of discharge planning, and identified research
needs and potential existing data sources. Major findings include the
following: 5 percent of extended hospital stays are not medically necessary;
the cost of these stays was conservatively estimated at $420 million in 1987
dollars. Reasons for extended hospital stay include lack of sufficient nursing
home beds and other alternate placement settings, delays in qualifying patients
for posthospital benefits, and delays in appointing conservators for
incompetent patients. Discharge planning appears to have only a small impact on
extended hospital stays.
The final evaluation completed in FY 1994 assessed the
availability of research on the cost-effectiveness of managed care health
plans. The study found managed care plans had lower hospital utilization,
greater use of less costly alternatives to expensive procedures and tests,
greater use of preventive measures, comparable quality of care, and somewhat
lower enrollee satisfaction generally, but greater satisfaction with cost. The
study recommended additional research on managed care performance, including
more comprehensive research projects to determine what works in managed care,
rather than just whether managed care works.
Evaluations in Progress AHCPR currently
supports seven evaluations that focus on its mission to generate and
disseminate information that improves the health care system. AHCPR's ongoing
projects provide information that will facilitate improvements in health care
quality by advancing the science of quality measurement. In addition, AHCPR
continues its commitment to improving the quality of AHCPR-supported clinical
practice guidelines.
Two prominent FY 1995 projects relate to improving the
science of quality measurement and improvement. The first, the Measurement
Typology Project, was designed to collect and summarize information on clinical
quality measures. These are technical quality measures used to estimate the
extent to which health care providers deliver services that are appropriate for
each patient's condition; whether services are provided safely, competently,
and in an appropriate timeframe; and what the outcomes are from the services
provided. The project summarizes 40 clinical performance measure sets--which
contain a total of 1,287 performance measures--and provides a framework for
evaluating the usefulness of various measures. The next phase of this project
will expand and refine the typology, which ultimately will provide the
foundation for an ongoing national resource of validated clinical quality
measures that will assist in quality measurement and improvement.
The second project, to be completed in FY 1995, was
designed to focus on consumer information needs. It developed a model consumer
survey to examine how consumers perceive the quality of health care they
receive, including their attitudes toward their access to care, use of specific
health services, and health outcomes. This project and the Measurement Typology
Project are complementary efforts to facilitate evaluation of quality from the
perspectives of clinical quality assessment and consumer satisfaction.
AHCPR is also continuing its commitment to improving its
clinical practice guidelines. Three projects focus on methods for improving the
efficiency and effectiveness of guideline development, and three large-scale
targeted evaluation projects will be completed during FY 1995. These projects
will provide information on the usefulness of the guidelines for quality
measurement and improvement.
An evaluation design study conducted by the George
Washington University Center for Health Policy is examining methods for
studying the process of guideline development. A follow-on study being
conducted by the American Institutes for Research (AIR) is applying these
evaluation design principles to examine factors that influence providers' views
of the quality of various guidelines. Some key findings of the George
Washington University study compare AHCPR's guidelines to those sponsored by
other organizations. AHCPR's guidelines are found to address a broader audience
(including a wider range of clinicians and clinical settings and a wider range
of nonclinicians, including patients, families, payers, regulators, and
purchasers); to have very detailed and comprehensive written policies and
procedures for their development; to have more specialized expert resources on
which to draw; and to use multiple methods for external review.
These differences have important implications for
evaluating the credibility, usability, and potential impact of the
AHCPR-supported guidelines on provider practice. Recommendations from this
study, as well as those from the AIR follow-on project, are expected to be used
to improve guideline development. AHCPR is also undertaking a project with the
Institute of Medicine to examine optimal methods for selecting guideline
topics.
In addition to projects to improve the guideline
development process, AHCPR has sponsored a number of targeted evaluation
projects. Through the use of evaluation contracts, cooperative agreements, and
intramural research projects, AHCPR has supported evaluative efforts for
guidelines on acute postoperative pain; urinary incontinence; benign prostatic
hyperplasia; cataracts in adults; prediction and prevention of pressure ulcers;
and depression in primary care.
Two targeted evaluation projects, to be completed in FY
1995, will provide important information on the feasibility of using the
guidelines for quality measurement and improvement, as well as for insight into
the difficulties of implementing guidelines. The first project develops,
implements, and evaluates quality and utilization review criteria and
educational outreach based on AHCPR-supported clinical practice guidelines. The
project uses five Medicare Peer Review Organizations (PROs) to develop criteria
based on three guidelines (urinary incontinence, acute postoperative pain, and
benign prostatic hyperplasia [BPH]); develop and test training materials to use
the criteria for case review; pilot test the criteria to assess intra- and
inter-rater reliability; apply guideline-based review criteria; and compare
guideline-based review with the review systems currently used by PROs. The PROs
are playing an integral role in developing, implementing, and evaluating
alternative educational outreach strategies based on the BPH guideline.
The second project is building on AHCPR's existing
efforts to evaluate clinical practice guidelines by translating AHCPR-supported
guidelines into medical review criteria for use in a variety of care settings
including hospitals, health maintenance organizations (HMOs), ambulatory
clinics, and physicians' offices. This project, like the project described in
the previous paragraph, will assess the usefulness of alternative educational
interventions for disseminating the guidelines and review criteria and for
changing provider practice behavior. One component of the project focuses on
cataract in adults, and the other on pressure ulcers in adults. The criteria
for this project will be developed with the Department of Veterans Affairs (VA)
and will be tested in a number of VA facilities. In a follow-on project, AHCPR
and the VA will test alternative methods for disseminating the guideline and
using the criteria as tools to assess the effect of guidelines on practice
patterns.
New Directions for Evaluation AHCPR is
in the process of developing a new strategic plan that reexamines current
activities in relation to a rapidly changing health care marketplace. This new
direction recognizes that AHCPR's programs must generate the information and
tools needed to improve health care delivery and its outcomes. In addition to
generating new knowledge, AHCPR's portfolio will focus on translating research
findings into forms of information that actively assist consumers,
practitioners, payers, and others in making effective health care
decisions.
AHCPR is also responding to new challenges to government
to become more efficient, flexible, creative, and effective. The Agency
recognizes the need for evaluation activities to be linked to the planning
process and to yield information that can inform policymaking, budget planning,
and program management.
In the coming years AHCPR will continue to support
research in the area of quality measurement and improvement. A Request for
Applications, titled Consumer Assessments of Health Plans Study (CAHPS), was
announced in the NIH Guide in May 1995. The proposed study will build
on the consumer survey project described above. The project will demonstrate
and evaluate the use of consumer surveys to assess health plans. AHCPR will
also build on work in the area of clinical performance measurement by expanding
the Measurement Typology Project to incorporate guideline-based clinical
performance measures into the measures inventory.
Centers for Disease Control and Prevention
MISSION: To promote health and quality of life by preventing
and controlling disease, injury, and disability.
and
Agency for Toxic Substances and Disease Registry
MISSION: To prevent exposure and adverse human health
effects and diminished quality of life associated with exposure to hazardous
substances from waste sites, unplanned releases, and other sources of pollution
in the environment.
CDC Evaluation Program The Centers
for Disease Control and Prevention (CDC) place a high priority on evaluations
seeking to answer policy, program, and strategic planning questions. Evaluation
studies are developed and selected on the basis of eight strategies to achieve
its mission. These strategies are to--
- Monitor health
- Detect and investigate health problems
- Conduct research to enhance prevention
- Develop and advocate sound public health policies
- Implement prevention methods
- Promote healthy behaviors
- Foster safe and healthful environments
- Provide leadership and training
The CDC Director provides annual guidance to the various
Center, Institute, and Office (CIO) Directors on 1 percent set-aside evaluation
activities. This memorandum generally includes information about the types of
studies to be carried out with 1 percent evaluation funds. Each proposal
undergoes multiple levels of review. Initial review is conducted by the Office
of Program Planning and Evaluation. Subsequent reviews are completed by CDC
analysts in the Office of the Assistant Secretary for Health (OASH) and the
Office of the Assistant Secretary for Planning and Evaluation (OASPE). Study
authors are provided with comments, questions, and recommendations made by
reviewers. In addition to providing their responses, authors are given the
opportunity to revise their proposals at this time.
A panel of CDC evaluators, scientists, and program
managers are convened to review and rank proposals. Review criteria include (1)
relevance to prevention effectiveness; (2) relative importance of the public
health problem being addressed; (3) probability that the proposed project will
accomplish its objectives; and (4) extent to which other CDC programs will
benefit from the project. Results from this panel review are converted into a
comprehensive ranking that is provided to the Director of CDC. Final funding
decisions are made at this time.
Finally, staff in the Office of Program Planning and
Evaluation work closely with program staff to ensure development of a clear
statement of work for selected projects. Before initiation of procurements, a
final ad hoc review of the project statement of work is completed.
ATSDR Evaluation Program Agency for
Toxic Substances and Disease Registry (ATSDR) receives its funds from
Environmental Protection Agency/Superfund appropriations rather than Public
Health appropriations; therefore, ATSDR does not receive a 1 percent
evalu-ation set-aside. Nevertheless, the Agency is responding to the changes
mandated in its program planning and evaluation efforts by the National
Performance Review and the Government Performance and Results Act (GPRA) of
1993. To meet those requirements, ATSDR staff members modified the Agency's
planning process, incorporating implementation strategies and
outcome/performance measures.
Prominent issues addressed in the new planning system
emphasize ATSDR's commitment to improve the health of people affected by
hazardous substances polluting the environment. Improvements include using
exposure assessments and demographic data to identify people at risk and, more
directly, assessing/addressing the concerns of customers. The new planning
system provides the basis for measuring ATSDR performance and making systematic
improvements as part of its internal evaluation activities.
Summary of FY 1994 CDC Evaluations CDC
completed 12 evaluations in fiscal year (FY) 1994. These evaluations covered
training and information dissemination, surveillance, program effectiveness,
prevention, and costs of disease.
Training and information dissemination was the focus of
several evaluations, two of which were highlighted in chapter II. The first was
an evaluation of CDC and ATSDR training activities that assessed the training
needs of State and local health departments and inventoried CDC's current
training activities. It provided an example of using an evaluation to document
current practice to help generate a new agenda for program action. The other
highlighted evaluation was a survey of readers of the Morbidity and
Mortality Weekly Report, a CDC publication of interest to public health
professionals around the Nation. The survey found that this publication
generally met the needs of its readers and was valued for its accuracy,
relevance, and concise reporting format. Study recommendations are expected to
help fine-tune this publication in response to reader suggestions.
Information dissemination activities also were addressed
in an evaluation sponsored by the Office on Smoking and Health (OSH).
Recommendations for key management and operational aspects of information
dissemination in OSH were made based on interviews with key officials in CDC
and outside the organization. The evaluation also made suggestions for
strategic planning in view of OSH's evolving leadership role in the tobacco
control community.
With respect to surveillance, several evaluations
focused on gathering statistics for policy analysis and decisionmaking. One
evaluation addressed the resiliency of the Model State Vital Statistics Act and
Regulations to accommodate changes in social customs and technology of
registering vital events and statistics. The results are being used to advise
States about the need for revisions in the collection of vital statistics. Two
evaluations are concerned with medical records in relation to national survey
data. Both studies are intended to determine whether medical records confirm
survey respondents' reports about selected conditions and impairments. In the
first study, selected data elements of the 1988 National Maternal and Infant
Health Survey (NMIHS) were compared to records maintained by the States. This
evaluation assessed the quality and completeness of the information reported,
identified discrepancies, and examined the nature and frequency of
discrepancies. The second study evaluated the accuracy of diagnostic reporting
in the National Health Interview Survey. Information collected from users of
health care services was compared with infor-mation collected from the source
of the health care services.
A third category of studies sought to evaluate health
programs to determine their effectiveness. One study assessed the effect of
burgeoning patient loads on sexually transmitted disease (STD) clinics, the
impact of changing funding levels, and the increase in more elaborate patient
testing. The study examined the effectiveness of services and identified
factors that contributed to overburdening the delivery system. A similar type
of study assessed the effectiveness and identified exemplary practices of
State-based diabetes control programs (DCPs) in providing services with the
potential for reducing diabetes-related mortality and morbidity. The outcomes
examined were (1) the number of people reached by the programs; (2) the
improved coverage provided by clinics; (3) the level of integration into
ongoing medical service delivery; and (4) the effect of leveraging resources
for diabetes programs. Findings showed that DCPs have had a measurable effect
on diabetes services. Recommendations for program expansion were provided as
part of this evaluation.
"Assessing Prevention Effectiveness: A Collaborative
Effort With Selected Health Maintenance Organizations" is the first part of a
two-phase study. A key component of this study includes the development of a
framework and process to assess prevention effectiveness in health maintenance
organizations (HMOs), types of services provided, and their potential to work
with CDC. Data collected in Phase I provide the information required to move to
Phase II of the study.
The purpose of the evaluation study by the Division of
Cancer Prevention and Control is to collect data pertinent to program-related
decisions. The evaluation focuses on components of a comprehensive breast and
cervical cancer early detection and control program. In addition to these
components (public and provider education, quality assurance, surveillance,
screening, and followup), the combined effect of components is
assessed.
The final category of evaluations sponsored by CDC
developed methods to estimate direct medical costs for various diseases. One
study estimated direct medical costs of chronic hepatitis B, concentrating on
acute care costs. Costs from both Medicaid and private sector data were
collected. Another study estimates the direct medical costs of congenital
syphilis, using 1990 figures. The estimate included medical care costs for the
first year, special education costs required by children with congenital
syphilis, and lifetime custodial care costs, all of which were categorized by
severity.
CDC Evaluations in Progress CDC has a
total of 32 evaluations in progress. They fall into four general categories:
surveillance/data collection studies; program evaluations;
community/intervention effectiveness studies; and evaluation methodology
studies. Performance improvement is a major focus of each of these
studies.
Surveillance/data collection is the focus of the largest
number of evaluations. For example, a study of the effectiveness of CDC
surveillance for drug-resistant pneumococcal infections addresses drug
resistance, which was identified as a major challenge in CDC's Emerging
Infections Plan. This project will evaluate the validity of antimicrobial
resistance data collected from sentinel hospitals, using CDC's sentinel
hospital surveillance program. The hospital surveillance program, located in 13
hospitals in 12 States, is designed to determine the magnitude of
drug-resistant pneumococcal disease and to provide clinicians with the ability
to select optimal regimens of empiric therapy. By using population-based
surveillance for invasive pneumococcal disease in two geographically distinct
areas, this project will evaluate the extent to which CDC's sentinel
surveillance program is capturing drug-resistant pneumococcal disease.
Another study that falls within the category of
surveillance/data collection entails evaluation of STDs in the United States.
The objectives of this study are to (1) determine the accuracy of CDC's
surveillance data on STDs by comparing them with independently collected data
from a survey of providers; and (2) to determine, also from a survey of
providers, the extent of adherence to CDC's diagnostic and treatment guidelines
for STDs and to identify ways of increasing compliance with those
guidelines.
Program evaluations, also an important focus of numerous
current evaluations, are being undertaken for grant programs, including the
Lead Poisoning Prevention Program and the Injury Prevention and Control
Program. Other studies in this group involve evaluations of the National
Laboratory Training Network, the San Juan Laboratory's Dengue Hemorrhagic Fever
Program, and the Fatality Assessment and Control Program.
Community-based interventions are the subject of several
other evaluations. Four studies address the prevention of violence. These
include projects focused on youth violence prevention, suicide in Native
American communities, domestic violence medical education programs, and support
systems for battered women.
Evaluation methodology is the focus of several ongoing
projects. For example, one project is developing a comprehensive evaluation
strategy that can be incorporated into planning, budget, and legislation for
the National Center for Chronic Disease Prevention and Health
Promotion.
New Directions for CDC Evaluation
Evaluation studies focusing on program performance and effectiveness will
continue to be of primary importance to CDC. As CDC moves toward a
comprehensive performance monitoring system, focused studies in this area will
be of utmost importance. Evaluations will be conducted to provide data for
decisionmaking regarding the need for broader program implementation.
Similarly, as programs develop and implement performance indicators, projects
designed to provide data for performance measurement and to assess the
effectiveness and efficacy of such indicators will be initiated.
Food and Drug Administration
MISSION: To protect and promote public health through food, drug, medical
device, and cosmetic regulation.
FDA Evaluation Program Systemic
changes in the government management environment are strongly influencing the
setting, conduct, and use of evaluation activities in the Food and Drug
Administration (FDA). Three forces--all related to the mandates of the
Government Performance and Results Act (GPRA) of 1993--are reshaping the
evaluation function in the FDA.
1. Performance management. The reorientation of all
government managers toward performance management has shifted the
responsibility for program evaluation from specialized staff offices and
contractor studies to day-to-day line managers. Picking appropriate program
goals, establishing valid measures toward those goals, and collecting
management information to record measured progress are now integral parts of
the government manager's responsibility. Relearning the role of management with
regard to these shifted responsibilities is a key priority. The old
order--evaluation as a proprietary domain of staff offices--is a bygone luxury
to FDA programs in the era of streamlined government.
2. Customer participation. Most of FDA's management
measures of its own performance are also measures of the performance of the
regulated industries. Thus, while the beneficiary of FDA's performance is
ultimately the general public, the Agency operates in such a manner that it
supplies industry with an essential component of commercial success. FDA's
approval of a new drug, for example, not only satisfies a legal requirement but
also assures the public of the safety and efficacy of the drug. Collaboration
between the FDA and its regulated customers regarding the design and
coordination of the joint responsibilities to ensure effective, high-quality
products has been a revolutionary concept, but it is becoming the norm under
the customer-conscious GPRA directives.
3. Increasing rigor of the rulemaking process. The FDA
establishes standards of safety and efficacy through rules published in the
Federal Register. Many safety and efficacy standards are also
performance standards that industry is obliged to meet. Examples are the Good
Manufacturing Practices regulations. Today, virtually every FDA final rule of
significant magnitude includes elements found in classic program evaluations: a
critique of the existing system, alternatives for better performance,
performance and cost tradeoffs, reactions and suggestions of customers, and
conclusions, with an action timetable for implementation. New directives from
the Administration as well as possible legislative action by Congress will make
the rulemaking process of the future even more like the classic evaluation
process.
In sum, FDA's evaluation efforts are driven by the
mandates of GPRA and its corollaries, are carried out by line managers rather
than specialized evaluation staffs, and are focused in the areas of performance
management, customer participation, and more rigorous rulemaking. Because it
receives its funds from Agriculture appropriations rather than Public Health
appropriations, the FDA does not manage a 1 percent evaluation
set-aside.
Summary of FY 1994 Evaluations Fiscal
year 1994 was an important test period for the new evaluation paradigm. The
following examples show how evaluation has been integrated into the line
manager responsibilities and interlocked with the affected customer.
Implementation of the Prescription Drug User Fee Act
(PDUFA). In FY 1994, the more than 1,000 full-time employees involved in FDA's
drug and biologic review process completed their first full year using the
performance management goals jointly established by the FDA and industry. Line
managers developed measures for evaluating progress toward these performance
measures, reported the results to Congress and the several hundred industry
customers of this program, and used the resulting management information to
self-manage the program toward higher FY 1995 performance goals. Line managers
have also focused on streamlining, achieving other review efficiencies, and
reducing the regulatory burden while maintaining the high quality and standards
associated with FDA decisionmaking.
The negotiation process toward medical device user fees.
During FY 1994, the FDA and representatives of the medical device industry
engaged in a series of meetings that constituted a joint evaluation of each
other's performance in the device development process. Current performance of
both parties was assessed quantitatively; new alternatives were modeled,
critiqued, and cost-audited by independent accountants; and the specifications
for a new program based on shared performance expectations were translated into
a framework for multiyear performance goals. While the culminating legislative
step was not realized in 1994, the joint effort stands as a model of a
customer-oriented, performance-based program evaluation.
Harmonization of international regulatory requirements.
While benchmarking has long been an important evaluation technique in the
private sector, it has not been widely used in the Federal sector--and
especially in regulatory agencies--because of the perceived uniqueness in
function and process of the Government Agencies. At the international level, of
course, benchmarking--especially for regulators--is not only possible but
highly desirable. Global forums such as the International Committee on
Harmonization invite such benchmarking. Their efforts are maturing and reaching
the point at which specific components of the regulatory process in different
countries have been compared and assessed by the managers of regulatory
programs. The initial successes suggest that regulatory benchmarking in the
context of international forums will be an increasingly important way for
regulators to evaluate and improve their performance.
Evaluations in Progress The following
evaluation objectives will influence line managers during FY 1995:
Performance management. Managers of all FDA programs
are evaluating their performance measures in light of GPRA standards.
Broad-based training of managers to enable them to evaluate the performance
design of their responsibilities is under way. The first phase of this
multiyear process toward a new standard of performance measure will end with
the formulation of FDA's FY 1997 budget.
Additional user fee programs. FDA managers will
finalize measures for two performance--oriented user fee programs identified in
the President's budget. One is a culmination of efforts by FDA managers and
industry representatives for a medical device user fee act; the other involves
developing with import brokers shared expectations and performance goals for a
user fee-supported electronic processing system for imports.
Commencement of negotiated rulemaking. In FY 1995, the
FDA will add a new customer-sensitive dimension to its increasingly rigorous
rulemaking function by beginning to implement the President's directive to
promote negotiated, consensual rulemaking. This marriage of customer
participation and rigorous rulemaking will receive serious attention from FDA
managers.
Two major projects are currently under way:
PDUFA management. During FY 1995--the middle year of
the 5-year user fee program--the FDA will use the performance data from this
GPRA-style program to generate the resource alignment needed to achieve the
high performance goals for FY 1997--the final year of the current user fee
legislation.
Mammography Quality Standards Act of 1992 (MQSA)
assessment. The FDA's second major user fee program, MQSA, is also a
performance-oriented, GPRA-style program. Although it is similar in design to
PDUFA, much of the data on its performance are more external to the agency. The
FDA is therefore using a contract study to gather data from affected facility
sites to gauge the effects of the standards on the availability of mammography
facilities.
New Directions for Evaluation Changes
in government management are creating systemic changes in the FDA's evaluation
function. A new paradigm driven by line managers' performance responsibilities,
by the imperatives for involving customers, and by directives for analytically
rigorous rulemaking has replaced the traditional practice of evaluation studies
guided by specialized evaluation staffs or third party consultants.
Health Resources and Services Administration
MISSION: To improve health by assuring quality health care to
underserved, vulnerable populations and by promoting a primary care and public
health workforce.
HRSA Evaluation Program The objectives
of the Health Resources and Services Administration (HRSA) evaluation program
are to improve program management and policy development and to provide
information that will enhance strategic planning, budget decisions, and
legislative planning. Consequently, a high priority is given to studies that
will (1) assess program effectiveness or outcomes, or (2) enhance the Agency's
capacity to measure performance by identifying meaningful performance
indicators, creating or refining databases, and developing study designs or
approaches. Over the past several years, HRSA has made considerable progress in
expanding such capacity.
The evaluation program in HRSA is designed to ensure
that the four Bureaus and the Office of Rural Health Policy (ORHP) identify
their particular information needs within a framework established by the
Administrator each spring through a memorandum on evaluation planning guidance.
Other staff offices may also propose studies. After study proposals are
reviewed for clarity by staff of the Office of Planning, Evaluation, and
Legislation (OPEL), proposals are reviewed for technical merit by a committee
of senior evaluators representing each Bureau, ORHP, the Agency for Health Care
Policy and Research, the National Center for Health Statistics (NCHS), the
Office of the Assistant Secretary for Health, and the Office of the Assistant
Secretary for Planning and Evaluation. Authors receive written comments on
methodological or other design issues. Proposals are then reviewed by a
committee of the Bureau Directors and other senior HRSA executives, whose role
is to consider the proposals for relevance to important policy, budgetary, or
legislative issues; potential to answer questions about program effectiveness
or impact; and degree of attention to crosscutting topics. Proposals and
committee recommendations are submitted to the Administrator for decision. OPEL
staff work closely with program staff to develop a strong scope of work and
other required documentation.
HRSA places major emphasis on the dissemination of
information about findings and utilization of completed studies. Developing
articles for publication in the professional literature and presentation at
professional meetings is strongly encouraged, and draft manuscripts are
typically called for as a contractor product. Briefings on completed studies of
broad interest are scheduled periodically for the Administrator and other
senior officials. An annual report on completed evaluation studies describes
study purposes and findings, and tells how study results have been disseminated
and used. Another annual publication provides brief summaries that describe the
purpose and proposed methodology for studies initiated in the preceding fiscal
year.
Summary of FY 1994 Evaluations Two
significant studies were completed that examined aspects of the community and
migrant health centers. "Evaluation Design: Evaluation of the Effectiveness of
Community Health Centers" developed a study methodology to examine the efficacy
and effectiveness of the centers. This work has provided a way to assess the
impact these providers have on the health status of their users. "Linkage
Programs: Survey of Mental Health Services" examined a joint National Institute
on Drug Abuse/HRSA demonstration program designed to test community-based
approaches for linking substance abuse and primary care services. The study
found that the projects had been successful in creating supportive environments
for the assessment, referral, and delivery of mental health services. Findings
from this study will help other providers of primary care services establish
the necessary alliances and treatment networks to reduce delivery system
fragmentation and develop a continuum of mental health services.
"HIV Service Networks in Four Rural Areas," highlighted
in chapter II, is a series of case studies undertaken to help States identify
ways to improve the quality, availability, and organization of services for
rural residents with HIV. The case studies described two distinct approaches to
providing medical care to rural residents with HIV: (1) cooperative working
relationships between HIV clinics and physicians in private practice, and (2)
linkages between physicians and a medical school or rural-based regional care
facility for training and backup consultation. Findings are being used to
formulate HRSA policies on HIV care in rural areas and to provide technical
assistance to States with large rural populations. The findings are also being
used to design a rural component, funded by HRSA, for a large national study of
AIDS services sponsored by the Agency for Health Care Policy and
Research.
Three other AIDS-related studies were completed by HRSA
in FY 1994. Two of these address the priority area of community infrastructure
building. "Implementation of Title I of the Ryan White CARE Act of 1990"
established a preimplementation baseline in Baltimore, Maryland, and Oakland,
California, covering the organization and funding of HIV-related services, the
functioning of planning councils, and the perceptions of HIV patients regarding
service delivery. The Henry J. Kaiser Family Foundation subsequently funded
followup surveys on these issues. "The Participation of People With HIV in
Title I HIV Health Services Planning Councils" studied planning councils in
Atlanta, Philadelphia, San Diego, and Seattle, and identified factors
supporting the initial and sustained involvement of HIV-positive individuals in
these councils. It also identified alternative methods employed by the councils
to gain input from and provide feedback to HIV-positive populations in the
absence of their direct participation on the councils. "Evaluation of Methods
for Estimating Unit Costs of HIV Health and Support Services" developed common
definitions for service, cost, and expenditure categories, and identified
potential indicators of cost savings attributable to coordinated,
comprehensive, community-based care systems.
"The Demographic and Treatment Characteristics of the
Hill-Burton Population" obtained data on a sample of patients receiving
services under the Hill-Burton Uncompensated Services Program and compared
these with data on the general population. The study found that, compared with
the total U.S. population, the Hill-Burton population was more likely to be
female, young adult, and unmarried, with a very low household income.
Three studies dealt with health professions programs.
"Survey of Beneficiaries of Nursing Education Projects" concluded that Federal
support of these projects has enhanced the supply of advanced practice nurses.
"The Training and Practice of Preventive Medicine Specialists: An Underutilized
Resource" studied the area and, based on its findings, recommended
incorporating preventive medicine into the medical school curriculum. In light
of the increased attention being given to improved management practices,
"Evaluation of Bureau of Health Professions Strategic Directions--Phase I"
developed a set of outcome indicators and identified potential data sources and
gaps relating to programs under Titles VII and VIII of the Public Health
Service Act.
Evaluations in Progress The 41 studies
that were ongoing as of September 1994 include four types: (1) measurement of
program outcomes, effectiveness, or impact; (2) development of performance
indicators, and/or data systems or databases; (3) design and/or testing of
study approaches; and (4) provision of other types of information for program
management or policy development.
The largest study of program performance and outcomes is
the National Evaluation of the Healthy Start initiative. Begun in September
1993, this 5-year prospective study includes both process and outcomes
components. The process evaluation is assessing the development and
implementation of comprehensive, coordinated systems of perinatal care in the
15 initially funded Healthy Start sites. The outcomes component concerns
changes in the health status of pregnant women and infants across the sites.
Client-specific data from a standard data set will be used to examine the
relationship among clients' sociodemographic characteristics, project-specific
service use, and maternal and infant health outcomes. Secondary data, including
linked birth and death records and Medicaid claims data, will be used to
compare outcomes of clients residing in Healthy Start project areas with those
of women living in matched comparison areas. The study also will employ a
series of more qualitative methodologies to gain information about the unique
features of each of the 15 projects. Four questions are being addressed. Did
the Healthy Start initiative succeed? If so, why? If not, why not? And, what
would be required for a similar intervention to succeed in another
setting?
Two other studies are assessing the effectiveness of the
community and migrant health centers programs, using national samples of
grantees and including both quantitative and qualitative techniques. Additional
studies concern effectiveness and outcomes of the National Practitioner Data
Bank, the Healthy Schools/Healthy Communities program, and the Injury
Prevention Implementation Incentive Grants program. One further study examines
the effects of Ryan White CARE Act Title I funding on services for active and
recovering HIV-infected drug users.
Among studies of the second type are projects to assess
data systems for the Ryan White CARE Act Titles I and II programs, and to
develop national integrated requirements estimates for physician assistants,
nurse practitioners, certified nurse-midwives, and physicians.
The Agency also has several ongoing studies to design
and test study approaches. One effort is intended to design a method to
evaluate the validity of current estimates of the reported mortality rate among
infants of Mexican descent residing in the United States and examine factors
contributing to the reported low rate. The study includes nontraditional
sources of data concerning births that occur outside the confines of the infant
tracking systems. Another study is developing a self-assessment tool for use by
HIV health planning councils in Title I cities and by care consortia in Title
II states. Grantees and their planning councils and care consortia have
expressed interest in methodologies and resources that will help them know what
they are accomplishing. The tool will be sensitive to the limited resources
available for planning and evaluation, providing a cost-effective method for
evaluating performance. An additional study is developing a design for
assessing variations in the procurement rates of Organ Procurement
Organizations (OPOs). Identification of the factors associated with low
procurement rates should help increase these rates.
Finally, several studies are intended to provide other
types of information for program management or policy development. One
exploratory project on the nature of telemedicine systems serving rural
populations is providing baseline information in connection with the new HRSA
Rural Telemedicine Grant program. An additional study is examining the impact
of certain Medicaid waivers on Federally Qualified Health Centers. Another
study is examining the relationship of medical school characteristics to
graduates who choose primary care and ultimately provide care to the
underserved. Still another study is exploring the impact of the Ryan White CARE
Act on strategies for serving African Americans. One of the study products will
be a report providing detailed examples of effective and ineffective processes
and strategies for enhancing access for African Americans. The report will also
identify performance indicators being used and approaches for gathering data to
measure performance. Finally, a 2-year study by the Institute of Medicine is
defining the place of primary care in the health care environment of the future
and developing normative goals for primary care.
New Directions for Evaluation The
major priority of HRSA evaluations in FY 1995 and 1996 will be studies to
continue the development of meaningful measures and data systems to facilitate
performance assessment. During FY 1995, HRSA is conducting a project to review
the status of performance indicators for all line organizations and major
programs and determine the degree to which data sources are currently available
for applying the appropriate indicators. The impetus for this effort is the
implementation of the GPRA, which will begin with the FY 1997 budget.
In FY 1995 and 1996, HRSA's other evaluation studies
will continue to reflect the Agency's mission of expanding access to care for
the underserved and such crosscutting priority areas as academic/community
partnerships in health professions education, community infrastructure
building, managed care, expanded roles for States, information technology, and
HIV/AIDS. HRSA also expects to develop multiyear evaluation goals for selected
program areas in connection with new authorizing legislation and to initiate or
continue studies of effectiveness, outcomes, and/or impact in selected program
areas.
Indian Health Service MISSION: To
provide a comprehensive health services delivery system for American Indians
and Alaska Natives, allowing for maximum tribal involvement in developing and
managing programs to meet their health needs.
IHS Evaluation Program The evaluation
program of the Indian Health Service (IHS) serves the Agency's programmatic and
policy objectives. These objectives, developed in consultation with tribal
communities, were most recently articulated in a "vision statement" of the IHS
Director in November 1994. The statement calls for the IHS to continue to be
the best primary care rural health system in the world. This goal is
predicated, in part, on additional program emphasis in areas in which
demographic and disease indicators point to a strong need for service delivery:
elderly care, child abuse, women's health care, and substance abuse. Substance
abuse, for example, is considered to be one of the most pressing problems
facing American Indians and Alaska Natives[1]
because of its toll in morbidity, mortality, and social turmoil.
The IHS evaluation program provides valid and reliable
information to help the Director promote this vision for the Agency, and the
evaluation program is also expected to help guide its implementation. The
program traditionally has addressed three general areas--service delivery,
health status, and management. The areas are interrelated, especially as the
IHS faces times of fiscal austerity. If the IHS is to improve service delivery
and maintain quality care, it must increase efficiency, effectiveness, and
accountability. Management practices are continually being reviewed to ensure
that IHS programs operate efficiently to ensure quality patient care and
preventive care--the cornerstones of the IHS.
The IHS conducts technical and policy reviews of
proposed evaluations--much like other PHS evaluation programs--but the program
also has several distinguishing features. The most noteworthy feature is the
active participation of tribes in the development and execution of evaluation
studies. American Indians are recognized as primary stakeholders whose needs
and concerns must be addressed, especially if the IHS is to help tribes develop
and provide health care to their communities. For example, the IHS sponsors
roundtable discussions with tribal leaders and Indian health program officials
to elicit advice and consultation about directions for evaluations and
policies. This approach to evaluation--a responsiveness to stakeholders and an
orientation toward a pluralism of cultural values--is referred to by
professional evaluators as a "fourth-generation" evaluation. Earlier types of
evaluations, which began after World War II, were more technical, descriptive,
and marked by the judgments and values of the evaluator.
Another distinguishing feature is that most evaluations
are performed by contracting firms owned and operated by American Indians. This
policy is an outgrowth of regulations and guidelines intended to increase the
participation of American Indians and tribes in program management and service
delivery.
Summary of FY 1994 Evaluations The IHS
completed nine evaluations in FY 1994, focusing on service delivery, health
status, and management. These areas are pivotal to the Director's vision of
ensuring the delivery of high-quality primary care. Three of these evaluations
deserve special notice because they address areas that are slated for renewed
programmatic emphasis in the coming years.
One evaluation assessed the availability, utilization,
and quality of data sets describing the health status of women and children in
the Navajo area. Maternal and child health (MCH) data sets play a vital role in
monitoring the health status of American Indian populations, shaping programs
and assessing their effectiveness. The first phase of the project sought,
through interviews and site visits, to determine whether IHS service unit staff
were able to fulfill existing data requirements use the data that were
generated. The project found staff to be inundated with data requests that did
not completely meet their needs for the care of patients and for the promotion
of maternal and child health. The second phase examined the quality of a
specific automated data set on childbirth outcomes by comparing it with
original chart entries. The comparisons were favorable, thus reinforcing the
quality of the automated data set. The findings and recommendations of the
first phase of the report are leading to a better understanding of MCH data
needs and to refinements in the types of MCH data required of service area
staff, along with improved vehicles for obtaining them.
Another evaluation, highlighted in chapter II, analyzed
the health status of American Indians in California. It documented the health
status of, and access to health care services for, those in both federally
recognized and non-federally recognized tribes. These and other comparisons
showed that the overall health status of both groups was poorer than that of
the California population as a whole. The findings underscore the significance
of maintaining and expanding coverage to the entire American Indian population
of California.
A third evaluation developed long-term health care
projections for alcohol-related hospitalizations. The projections--which were
based on a new method called "long-term projection methodology"--sought to
determine the impact of changes in hospitalizations as a result of advances and
innovations in treatment, tech-nology, and behavior. The technique forecast a
20 percent decline in alcohol-related hospitalizations. While these projections
were encouraging, they were insufficient to meet Healthy People 2000
objectives. This finding is one of many that have led to a renewed IHS emphasis
on prevention of substance abuse. This study will be used specifically to guide
evaluations such as that planned on regional adolescent alcoholism treatment
centers (described below under "New Directions for Evaluation") and will
provide tribes with information relevant to alcoholism program planning, policy
development, and program evaluation. For example, the study informs IHS
providers that the only age-gender group showing increases in alcohol-related
hospitalizations is American Indian women over the age of 45.
Evaluations in Progress The IHS
currently is sponsoring 21 evaluations covering service delivery, management,
and prevention. Two of these evaluations underscore the IHS commitment to the
development of new programs to prevent child abuse and family violence.
Violence and abuse are serious problems that erode the quality of family life
and can result in death, serious injury, or long-term health and behavioral
problems. Family violence and child abuse are difficult to study because of
underreporting and shifting definitions of what is acceptable behavior. The
diversity of cultures, languages, and customs of American Indian tribes also
heightens the difficulty of determining the extent of these problems and
evaluating what works to prevent them.
One report, which is expected to be completed in FY
1995, is an assessment of the extent of child abuse and neglect among American
Indian tribes and the ways in which the IHS responds to these problems. The
evaluation also develops a model program to combat child abuse and neglect.
Preliminary results from survey data indicate about 34 percent of American
Indian children are at risk of becoming victims, and substance abuse commonly
plays a role. The intervention program under development is being modeled after
a successful program developed in 1985 by the State of Hawaii. The program
provides home visitor services to new mothers, ensures continuity of medical
care for children, and links families with essential services.
The other evaluation report is a case study of family
violence in four distinct American Indian communities. Researchers conducted
more than 100 unstructured interviews at the four sites to probe the nature and
extent of family violence, which was defined as spousal abuse, child abuse and
neglect, child sexual abuse, or elder abuse. They found little consensus across
the four study sites about which type of family violence represents the gravest
problem. These and other study findings are being incorporated into a model to
guide the development of prevention programs. Eventually, this model will be
available for use by an individual, family, group, or tribe to develop programs
to prevent or reduce family violence in American Indian communities.
New Directions for Evaluation The IHS
foresees the need for evaluations in the following areas: mental health
services for urban Indians, regional treatment centers (RTCs) for substance
abuse disorders, and health services for elderly American Indians. Urban
American Indians, who account for approximately 56 percent of the American
Indian population according to the 1990 Census, have important mental health
needs. The urban American Indian usually lives in poverty and has little or no
support system, in contrast to the American Indian living on a reservation
among others who live at the same economic level and rely on the security of
extended family and housing assistance. Indians who come to the cities
encounter a physical environment, social organization, interpersonal behavior,
attitudes, values, and sometimes even language that are foreign to their
experience. They may suffer from feelings of isolation, depression,
desperation, and anxiety, and may have problems with self-esteem. The
traditional support of the extended family may be totally lacking. In 1976,
Congress passed the Indian Health Care Improvement Act (P.L. 94-437), which was
landmark legislation for all Indian health concerns but particularly for urban
populations. Title V of the legislation specifically authorized health outreach
and referral and the delivery of services to American Indian people in urban
areas. Before this, health care delivery was not permitted off the reservation.
The urban health program was expanded in 1988 in the Indian Health Care
Amendments (P.L. 100-713). These amendments clarified the types of primary care
and outreach services that could be provided, paving the way for the provision
of mental health services. Mental health services are considered ripe for
evaluation because the IHS supports extensive services in some urban areas and
few, if any, services in other urban areas. This disparity needs to be
evaluated.
Nine RTCs offer residential substance abuse treatment to
Indian youth, most of them aged 12 to 19, in recognition of the fact that young
people are especially vulnerable to long-term patterns of alcohol and drug
abuse. Although alcoholism is decreasing among older American Indians, it
remains stable in adolescents. Alcohol abuse among American Indian youth
usually begins during adolescence or earlier. The RTC programs were legislated
into existence by Congress and are an important part of the continuum of care
for American Indian youth. Programs that began in the late 1980s and were
accredited by the Joint Commission on Accreditation of Health Care
Organizations are now deemed ready for evaluation because they have been in
operation for several years and they appear to play an important role in
reducing substance abuse. Evaluations are needed to document the impact of
these programs, including outcome measures; to determine the level of agreement
between referring and discharge diagnoses of individuals admitted to RTCs; to
examine the characteristics and histories of youth and staff at the RTCs; and
to determine the impact these centers have had in the local areas. The results
can be used to redefine the operation of existing RTCs and guide the design of
future centers.
Longer life expectancy among American Indians is posing
new challenges for IHS service delivery. Demographics showing that life
expectancy has increased from 61 years in 1972-74 to 71.6 years in 1986-88 have
led to the creation and expansion of special services geared to the elderly.
Evaluation of health services for elderly persons who are ambulatory,
homebound, or institutionalized is considered critical if the IHS is to
effectively meet the needs of this growing population. Evaluations should
address the increasing need for long-term care. The emerging nature of the
problem gives the IHS an opportunity to define and develop the most appropriate
services for American Indian elders before the need overwhelms American Indian
communities and leaves many without services. Improving the quality of life for
elderly American Indians is thought to be a critical need in the next few
decades.
National Institutes of Health
MISSION: To discover and disseminate new knowledge leading to improved
health for all Americans.
NIH Evaluation Program Evaluation is
an integral part of the role of the National Institutes of Health (NIH) in the
support of biomedical research, training, and public education. Evaluation
studies are undertaken to ensure that NIH meets its specific goals to--
- support biomedical and behavioral research of the
highest quality;
- inform health researchers, health care providers, industry, and the
public of advances and opportunities to improve health;
- support the training and continued availability of biomedical and
behavioral scientists;
- support the facilities and equipment needed to sustain scientific
progress; and
- manage its resources effectively and efficiently.
A distinguishing feature of the NIH Evaluation Program is
the variety of evaluation instruments it employs. The most familiar instrument
is a formal evaluation study that examines whether a program has successfully
met its objectives. But NIH supports a host of evaluation strategies that go
beyond traditional program evaluations. The NIH peer review system is one type
of evaluation strategy: research proposals from scientists around the Nation
are subjected to a rigorous assessment by fellow scientists, and only the most
meritorious proposals receive funding. Other evaluation strategies include
national advisory councils, boards of scientific counselors, consensus
development conferences, and committees. These groups are charged with
assessing a body of research to establish priorities, developing long-range
goals and strategies, addressing emerging issues, identifying significant
opportunities, assessing needs for new programs and activities, and
recommending expansion, realignment, or continuation of ongoing
programs.
The reason for the diversity of evaluation instruments
lies in the nature of research. Research--especially basic research--depends on
pursuing the unknown. The results of a research program and the generation of
new knowledge usually cannot be anticipated. Consequently, research does not
readily lend itself to the most common type of evaluation, an outcomes
evaluation. Programs that provide services or promulgate regulations are most
suited to outcomes evaluation because they are intended to achieve explicit,
preconceived objectives. NIH attempts to evaluate its research programs with
methods suited to the serendipitous nature of the research enterprise.
Another distinguishing feature of the NIH Evaluation
Program is its use of the 1 percent set-aside evaluation fund strictly for
programs that transcend individual NIH Institutes. The focus on NIH-wide
evaluations is a self-imposed policy. NIH relies on its component Institutes,
Centers, and Divisions (ICDs) to generate requests for funding of NIH-wide
projects from the 1 percent set-aside, in addition to those that are centrally
directed. The ICDs also conduct individual evaluations supported by their own
program funds.
In June 1991, the Office of the Assistant Secretary for
Health (OASH) authorized NIH to approve all set-aside funded evaluations,
whatever the budget, while it maintained an ex officio presence in the review
process. A two-tiered system is used to review project requests for 1 percent
set-aside funding. One tier is the Evaluation Policy Oversight Committee
(EPOC), and the other is the Technical Merit Review Committee (TMRC). The EPOC
includes representatives from the Office of the Director, NIH, and ICD
representatives at the level of Director, Deputy Director of an ICD, or
Associate Director of the Office of the Director. EPOC conducts policy- level
concept reviews of proposals for NIH-wide evaluation studies that use set-aside
funds, establishes the overall NIH set-aside budget, and oversees the process.
EPOC recommendations are approved by the Director, NIH, or designee before the
initiation of any study. The TMRC is responsible for the technical review of
the submissions and for recommending to the EPOC whether a project fits within
departmental guidelines for the set-aside fund.
Evaluations and evaluation priorities pertaining to
individual ICDs are shaped by ICD Directors and Deputy Directors. The results
help ICDs and the Director, NIH, establish priorities, develop long-range goals
and strategies, and review programs in terms of scientific excellence,
relevance, cost, and uniqueness.
Summary of FY 1994 Evaluations The
eight evaluations completed in FY 1994 addressed almost all elements of NIH's
mission, from research to public education. Four of the eight evaluations are
highlighted in chapter II. The evaluations summarized below illustrate the
diversity of the NIH Evaluation Program through a commitment to evaluating a
body of research that informs public policy, evaluating the Nation's need for
scientific manpower, and evaluating public health information.
One of these four evaluations addressed the safety of
selected childhood vaccines. This study was mandated by Congress under Section
313 of Public Law 99-660 to yield essential information that would help the
Public Health Service draft recommendations about the use of, and compensation
for adverse reactions to, vaccines against tetanus, diphtheria, measles, mumps,
polio, Haemophilus influenzae type b, and hepatitis B. The study
entailed an expert review by a committee of the Institute of Medicine. The
committee examined all relevant medical and scientific literature on the
potentially serious risks associated with currently licensed childhood
vaccines. Its findings about each vaccine have been incorporated into brochures
given to parents before children are vaccinated and into proposals to revise
the list of compensable injuries presumed to have been caused by certain
vaccines.
Another evaluation was on national needs for biomedical
and behavioral research personnel. This evaluation addressed the Nation's
future need for biomedical and behavioral research scientists and the
contribution of NIH training grants called National Research Service Awards
(NRSAs). The study was the 10th in a continuing series of reports to NIH and
the U.S. Congress on this topic.
The National Research Service Award Act of 1974
consolidated all previous training authorities into the NRSA program. The Act
authorized both predoctoral and postdoctoral support to individuals and to
institutions. To implement the Act, NIH set up individual fellowships and
grants to institutions for training predoctoral and postdoctoral students.
Close to $400 million is spent annually on these training grants. A National
Research Council expert committee, under contract to NIH, found that although
the NRSA program is relatively small in terms of the total number of trainees,
it is enormously powerful in its ability to change research emphasis and to
attract the highest quality individuals to research careers. It is viewed as a
prestigious, highly competitive program, and it is clear that initiatives
introduced through the NRSA program can have a powerful impact on intended new
research directions or constituencies. A final evaluation was performed to
determine the feasibility of assessing NIH-supported research to increase
condom use. This project was a feasibility study, or evaluability assessment,
to determine if an outcomes evaluation could be performed in a second phase and
if it would be useful to do so. The evaluation was motivated by public health
efforts to prevent the spread of AIDS through the use of condoms. Its
objectives were to assess the findings of condom use research efforts, guide
the development of future program areas, and suggest methodological guidelines
to facilitate the evaluation of future condom research programs.
The evaluation identified and inventoried the universe
of condom use research studies supported by NIH grants. A reproducible
methodology that combined automated database searching of NIH grants with human
judgment was established. The methodology identified more than 500 relevant
studies. A sample of 76 studies was examined in detail to identify how well
demographic characteristics were defined, which sampling methodologies were
employed, whether and what type of comparison group was used, and so on. The
final report was widely disseminated. The evaluation was valuable because it
generated useful methodological tools for meta-analysis and because it led to a
decision not to pursue a larger scale evaluation. What emerged was a research
agenda that the ICDs (seven had participated in the technical advisory group
for the evaluation) could use to address the vital questions related to condom
use most closely related to their institutional missions.
Evaluations in Progress NIH has 24
evaluations in progress. They range from small- to large-scale assessments,
from evaluability studies to full-blown evaluations. One study builds on a
longstanding role in ensuring the training and continued availability of
superior biomedical and behavioral scientists through the NRSAs. The first
study objective is to conduct an evaluation design study, that is, develop a
detailed plan for a comprehensive evaluation of career outcomes of the
predoctoral and postdoctoral trainees and fellows and the NRSA programs in
which they have participated. The second objective is to develop an approach to
characterize the nature and quality of the training actually experienced by
present and former trainees and fellows and to differentiate between a good
training program and simply good trainees. No baseline data are available on
trainees, and program versus selection effects have not been studied. The third
objective is to develop an approach to tap the perceptions of NIH staff,
present and former NRSA trainees and fellows, and university administrators
about the nature and impact of the training program. This study is expected to
be completed in 1995.
A second example is a study of the Physician Data Query
(PDQ), a comprehensive cancer database intended primarily for cancer health
professionals. The database contains state-of-the-art treatment summaries and
information on supportive care, screening, prevention, and experimental drug
therapies. The objectives of the study are to survey PDQ database users to
determine who is using the database and how the information is used, and to
assess user satisfaction with the information and the method of retrieval, for
example, CD-ROM, on-line, or hard copy.
The National Cancer Institute is directing a study to
identify ways to increase target audiences. The study will produce a written
summary of the activities and analyses of the project, data tapes and
documentation of all questionnaire responses, and a computerized system of data
collection, tabulation, and analysis. It will also include suggestions for
improvements to the PDQ database to ensure that it meets user needs.
The third evaluation examines research resources
available to primate researchers throughout the Nation. Through special grants,
NIH supports seven Regional Primate Research Centers, a unique national network
of nonhuman primate research and resource laboratories established in the early
1960s. The centers are geographically located throughout the United States, and
each is closely affiliated with an academic institution. Center activities
include the conduct of biomedical and behavioral research; research resource
support to investigators funded by other sources; the maintenance of more than
18,000 nonhuman primates; the establishment of breeding programs to meet the
centers' research requirements; conservation and preservation programs; expert
professional and technical support to investigators; and training of pre- and
postdoctoral professionals in primatology research. When the nonhuman primate
is the most appropriate species to study, the centers provide a cost-effective
response to the need for national repositories of nonhuman primates, scientific
expertise, and specialized facilities and equipment.
The evaluation is assessing all aspects of the Regional
Primate Research Centers program, such as its effectiveness in meeting current
program objectives and guidelines, scientific distribution and emphasis of
research programs, future planning decisions, and collaborations with and
access by non-center investigators. Other issues to be addressed are compliance
with policies and guidelines, financial management, host relationship, grant
award process, peer review, status in the scientific community, and reporting
and dissemination of program results. The centers are being compared with
several selected non-center institutions with significant NIH-supported
nonhuman primate research. The results of the study will be used to improve and
refine the centers program.
New Directions for Evaluation In peer
review, for example, NIH has been examining streamlining review of grant
applications to make more effective use of reviewers' time. Streamlining has
been tried on an experimental basis and is now being implemented fully. The
results will be closely watched. The Government Performance Results Act (GPRA)
of 1993 requires NIH to develop a strategic plan, an annual performance plan,
and performance indicators by September 1997. This effort is currently a
central focus of NIH evaluation activities. NIH is exploring a variety of
indicators for potential areas in the strategic plan such as information
dissemination, technology transfer, and its reinvention program. In addition,
priority will be given to funding 1 percent set-aside project proposals
submitted by the ICDs and the NIH Office of the Director that relate to the
GPRA. Also, the EPOC will reexamine and redefine NIH evaluation priorities for
the future.
Substance Abuse and Mental Health Services Administration
MISSION: To improve the quality and availability of
prevention, treatment, and rehabilitation services for substance abuse and
mental illness.
SAMHSA Evaluation Program The
Substance Abuse and Mental Health Services Administration (SAMHSA) is committed
to evaluating its overall programs and individual grant projects to
assess--
To the greatest extent appropriate and feasible, SAMHSA
will encourage the use of comparable data elements and instruments across its
evaluations in order to work toward a comprehensive evaluation system and to
minimize respondent burden.
SAMHSA conducts grant programs under a variety of
legislative authorities. These authorities can generally be grouped into two
types: services and demonstrations. The evaluation required for a particular
grant program depends on the type and purpose of the program. SAMHSA evaluates
its service programs to provide information to program managers about the
accountability of Federal funds. Its evaluations of demonstration programs
generate new knowledge to lead the field in developing policies that improve
services. Evaluation results from the two types of programs are used to improve
the performance of SAMHSA and the field of substance abuse and mental health
services. Program and evaluation staff work together to identify the questions
or goals each grant program should address and to propose appropriate
evaluation strategies.
The two types of grant programs represent two facets of
SAMHSA's mission: service delivery and knowledge development. SAMHSA's
leadership in the field depends on the successful interaction of these two
facets of its mission. SAMHSA uses evaluation to identify effective approaches
to prevention and treatment and uses service delivery funds to provide
incentives to the field to implement effective approaches. Major emphases of
SAMHSA's mission are the development, identification, and dissemination of
effective strategies and systems for treatment and prevention.
SAMHSA is now implementing a new, integrated model of
evaluation and planning. Strategic planning will identify priorities, such as
managed care, that drive the development of grant programs and evaluations. In
compliance with the Government Performance and Results Act (GPRA), SAMHSA is
attempting to improve performance by identifying performance goals and
indicators as part of its strategic planning process. The formulation of
programmatic and evaluation priorities will include consultation with the
SAMHSA and center advisory councils and with other experts in the fields of
evaluation and service delivery. Early and continuous coordination of program
planning and evaluation design will result in the articulation of evaluable
programmatic objectives. Evaluations will show how well the overall grant
programs have achieved their objectives, and SAMHSA will translate these
results into information that can be used for program and policy development.
The strategic planning and policy development processes will then use these
results to refine SAMHSA's priorities and objectives.
This evaluation policy will help SAMHSA achieve its goal
of continually informing policy and program development with knowledge culled
from past performance. In this way SAMHSA can best serve its customers by
enhancing the quality of public substance abuse and mental health
services.
In compliance with the Public Health Service (PHS)
guidelines for the technical review of evaluations, SAMHSA has a standing
review committee of PHS staff who are evaluation specialists from Agency for
Health Care Policy and Research, Centers for Disease Control and Prevention,
Health Resources and Services Administration, National Institutes of Health,
and SAMHSA. Representatives of the Office of the Assistant Secretary for Health
and the Office of the Assistant Secretary for Planning and Evaluation serve as
ex officio members, and the SAMHSA evaluation coordinator serves as committee
chair.
Evaluation project proposals are generally prepared by
SAMHSA program staff in the various centers. The standing committee reviews
each proposal on the following criteria: clarity of evaluation objectives;
appropriateness and feasibility of the specifications for evaluation design and
methods; appropriateness of the plans for dissemination of results; and use of
previous relevant evaluations and existing program data systems.
Each proposal must clearly state the relationship of the
evaluation to SAMHSA's overall policy priorities and evaluation program. The
committee generally reviews proposals for broad, comprehensive evaluations of
SAMHSA program activities, such as the cross-site evaluations of demonstration
grant programs. It does not review the evaluation proposals of individual
grantees.
Summary of FY 1994 Evaluations During
FY 1994, SAMHSA completed six evaluations. The six studies reflect some of the
population groups SAMHSA has identified as being in greatest need of substance
abuse and mental health services. These include pregnant and postpartum women
and their infants, children with serious emotional disturbance, high-risk
youth, and the homeless mentally ill.
The evaluation of a demonstration program serving
pregnant and postpartum women and their infants examined the results of
improved coordination, availability, and accessibility of health and substance
abuse-related services. A substantial number of the women served by these
programs reduced their substance use, and their babies were generally healthy.
SAMHSA will use these findings to encourage improvements in the performance of
service programs for substance-abusing pregnant and postpartum women and their
infants.
SAMHSA completed the design for an evaluation of a
comprehensive community services program for children with serious emotional
disturbance. The purpose of the evaluation is to collect information on client
and systems outcomes. The findings should help both policymakers and service
providers improve the performance and cost-effectiveness of mental health
services available to children.
Another evaluation conducted this year examined the
cost-effectiveness of a drug treatment enrichment program in curbing drug use
among students receiving training at Job Corps Centers. Job Corps Centers are
residential employment and training programs for high-risk youth. Students
participating in the enrichment program showed a significant decrease in drug
use compared with a control group. The findings of this study are useful in
verifying the effectiveness of substance abuse treatment and in informing the
field about appropriate substance abuse services for high-risk youth.
An evaluation of a demonstration program for homeless
mentally ill adults is highlighted in chapter II of this report. This study
examined the results of improved linkage, integration, and availability of
housing and social and medical services. The program was successful in reducing
homelessness and improving the mental health of study participants. These
results are encouraging communities to better integrate services for the
homeless mentally ill.
An evaluation of demonstration projects for community
partnerships to prevent substance abuse has produced information on the
successful implementation of substance abuse prevention approaches that employ
a comprehensive, coordinated, communitywide strategy. Evaluation results of the
impact of the partnership program are forthcoming. The Second Report to
Congress on Alcohol and Drug Abuse Prevention presents preliminary results of
the National Structured Evaluation (NSE). The NSE is a systematic assessment of
substance abuse prevention approaches implemented across the country. Final
results of this study, to be presented in a third and final report to Congress,
will have important implications for improving the performance of substance
abuse prevention programs.
Evaluations in Progress SAMHSA
currently has 10 major evaluations under way in the following general areas:
program accountability, evaluation of demonstrations, reinforcing behavioral
health, and commitment to customer service. Each type is described below, with
some examples.
Program accountability evaluations are undertaken to
inform program management and help managers refine program operations. This is
the primary type of evaluation conducted on SAMHSA's service grant programs.
For example, the children's mental health service program has an ongoing
evaluation. The evaluation will yield continuous information on program
implementation and on outcomes for children and families served. This
information will be used for reports to Congress, feedback to grantees, program
development, and performance improvement.
Evaluations of demonstrations are designed to generate
new knowledge for policy development. The primary purpose of SAMHSA's
demonstration programs is to generate new knowledge to lead the field in the
development of policies that improve services. For example, evaluation results
on substance abuse prevention for high-risk youth will allow policymakers to
draw inferences about the effectiveness of certain interventions for this
population. The Job Corps evaluation will continue to assess an enriched
substance abuse treatment program for adolescents. An evaluation of the program
for access to community care and effective services and supports (ACCESS) will
provide information for the design of ongoing service programs at the Federal,
State, and local levels.
Other SAMHSA evaluations strive to reinforce behavioral
health as a cornerstone of public health. In the context of managed care and
health care reform, SAMHSA's interagency collaborations promote the integration
of services and reinforce the role of alcohol, drug, and mental health services
as a critical component of general health care. Illustrations of such
activities include collaboration with the Administration on Children and
Families to evaluate a substance abuse prevention program for pregnant and
postpartum women and infants.
The National Treatment Improvement Evaluation Study
(NTIES) was designed as one of a family of studies of substance abuse
treatment, including two studies conducted by the National Institutes of Health
(NIH). This collaboration will enhance SAMHSA's ability to draw inferences
about treatment effectiveness. SAMHSA will continue working with the Department
of Housing and Urban Development to design a method by which therapeutic
communities can be incorporated into public housing areas. An HIV/AIDS mental
health services demonstration program is jointly funded by NIH and
HRSA.
Because of its commitment to the recipients of alcohol,
drug abuse, and mental health services, SAMHSA uses evaluation to guide the
field in improving the integration of services and their responsiveness to the
needs of clients. For example, the evaluation of the community partnership
program will determine the part-nership's ability to achieve integration and
responsiveness at the user level. Primary purposes of the community coalitions
demonstration program are to increase the efficiency of services at the
community level and to enhance the service delivery system's responsiveness to
its users. The evaluation of the program will assess the effectiveness of this
approach.
SAMHSA has promulgated a comprehensive treatment model
that includes a broad array of treatment and ancillary services targeted for
individuals whose substance abuse has severely affected many domains of their
lives. The services include housing, child care, transportation, social
services assistance, legal aid, and mental and physical health screenings and
treatment, as appropriate. Every SAMHSA grant announcement for substance abuse
treatment will require that grantees implement the comprehensive treatment
model. Evaluations will assess the effectiveness of the model.
New Directions for Evaluation SAMHSA
is designing its evaluation activities to guide programmatic and policy
decisions. The evaluation activities will complement the SAMHSA and PHS
strategic plans and will respond to emerging trends such as managed care and
health care reform. For example, in FY 1995 SAMHSA will initiate a program of
State contracts to evaluate managed care and treatment outcome. The program
will provide support to States to conduct short- and long-term studies at the
State and provider levels of the effects of managed care on substance abuse
treatment access, cost, and outcomes. SAMHSA will support States' use of
evaluation methodologies including State financial, provider, and client
databases.
These evaluation activities will reinforce the critical
role of behavioral health in general health care. Results of the evaluations
will ensure accountability and generate new knowledge. SAMHSA will use these
results to improve integration of services and responsiveness to the needs of
individuals suffering from or at risk for alcohol, drug abuse, and mental
health problems.
Office of the Assistant Secretary for Health
MISSION: To provide public health advice to the Secretary of
Health and Human Services and executive direction to the PHS Agencies through
coordination and oversight of their programs.
OASH Evaluation Program The primary
role of the Office of the Assistant Secretary for Health (OASH) is to
coordinate and develop evaluations across the entire Public Health Service. The
Assistant Secretary for Health (ASH) issues guidance each fiscal year to the
PHS Agencies for preparation of their plans outlining evaluation priorities and
projects proposed for implementation in that fiscal year (as mentioned in
chapter I). OASH reviews the Agency plans to identify gaps in evaluation data
needed for program or policy development. OASH also identifies potential
program areas that could benefit from a collaborative evaluation.
In addition to its PHS-wide coordination function, OASH
uses the 1 percent set-aside funds to initiate its own evaluations or policy
studies, usually in consultation with relevant Agency program offices. The
projects conducted by OASH are generally short, quick-turnaround policy studies
to support initiatives of the ASH. Occasionally, OASH will initiate or
participate in large-scale evaluations, particularly those that cut across PHS
program areas.
The OASH review and approval of evaluation projects is
performed by the Evaluation Policy Review Committee (EPRC), which makes
recommendations to the ASH on OASH evaluation priorities for each fiscal year;
reviews project proposals for relevance to PHS strategic planning objectives;
and provides advice to the ASH in determining the OASH use of 1-percent
evaluation funds and allocation of those funds to approved projects. Among the
criteria used by the EPRC to review proposed projects are relevance to ASH
priorities for policy, legislative, budget, program development, or
crosscutting PHS issues; relevance to departmental and PHS strategic planning
objectives and related evaluation strategies; coordination with other PHS and
departmental offices; and relevance to the Healthy People 2000
objectives. OASH evaluation staff review all EPRC-approved projects for
technical quality and feasibility, providing feedback to OASH program offices
on project proposals and statement of work for contract procurements. In
addition, the staff serves as a resource group for individual technical
assistance and consultation to project officers in developing evaluation
proposals.
Summary of FY 1994 Evaluations In FY
1994, OASH completed 16 evaluations in diverse areas of public health: health
care reform, immunizations, adolescent and school health, nutrition, primary
care, emergency preparedness, international health, and other smaller regional
studies of health services delivery. Most of these evaluations focused on
policy analysis to support initiatives in health care reform. Despite the
failure of health care reform legislation, these studies are relevant because
health care reform proposals are still being considered by Congress and because
market-based reforms are occurring in the absence of legislation.
One study on the returns on investment in public health
examined the contributions of public health programs in reducing the incidence
of preventable illnesses and injuries. This work summarized the effectiveness
of public health strategies from a cost-benefit analysis perspective to
underscore the importance of the public health infrastructure in realizing the
two primary goals of health care reform: to expand access to care and to
control costs.
Two related studies examined the implications of health
care reform on the health care workforce. The study on a proposed Graduate
Nursing Education Account, which is highlighted in chapter II, documented the
projected scarcity of advance practice nurses such as nurse practitioners,
certified nurse-midwives, certified registered anesthetists, and clinical nurse
specialists. It analyzed several options for channeling funds to support
advance practice nursing education to meet future needs. Another study examined
the supply, training, and distribution of primary care providers as part of
health care reform efforts. It examined the rationale for new proposals to
control the supply of new physicians, proposals for an all-payer pool for
graduate medical education, and estimates of the number of advanced practice
nurses that would be needed as substitutes in the case of a reduction in the
number of physicians.
In recent years, OASH has given priority to evaluations
of PHS immunization policies and programs. One study, also highlighted in
chapter II, examined the economic underpinnings of the vaccine supply. A major
finding was that States that supply vaccines at low prices to physician offices
and encourage parents to have their children vaccinated have higher rates of
immunization. In another study, OASH developed a design to assess the
functional capabilities of the Vaccine Adverse Event Reporting System (VAERS).
This design project will help PHS officials evaluate VAERS to determine whether
it is providing reliable and accurate information, and how health care
providers, manufacturers, and consumers perceive the effectiveness of the
system.
Support for nutrition policy studies has been another
OASH evaluation priority. In FY 1994, OASH completed a project with the
American Institute of Nutrition to examine state-of-the-art techniques for
measuring nutritional status and to make recommendations for a core set of
nutritional status indicators for low-income populations. Development of these
indicators will be important for the development of national and State
nutrition monitoring systems, particularly in expanding the survey coverage of
low-income populations.
Adolescent health, including school heath education
programs, is another OASH program evaluation priority area. Two projects in
this area, both described in chapter II, were completed in FY 1994. The first
study sought ways to evaluate the academic benefits of school health programs.
It developed a general framework, with alternative research designs, for
assessing the effects of school health interventions on students' school
performance. The second study was a focus group study of the knowledge,
attitudes, behaviors, beliefs, and environments of youth engaged in high-risk
behaviors. The results of this study will be used to develop appropriate health
messages to youth about high-risk behavior.
Evaluations in Progress OASH has 14
program and policy evaluations under way in two major areas: health care reform
and population-based services. In addition, OASH is supporting several smaller
projects investigating crosscutting PHS issues and several evaluations specific
to OASH program offices. Some of the evaluations that are likely to be
completed in FY 1995 are described below.
With respect to health care reform, OASH is continuing
to examine two major issues: the role of Federal, State, and local health
information systems in achieving public health objectives, and the impact of
health care reform on the health care workforce and the role of academic health
centers (AHCs).
Health information systems at all levels of government
are the focus of three projects. One is developing sets of indicators that
communities can use to assess performance and to foster closer collaboration
between public health and personal health care systems and employers in
achieving specific public health goals. Another is reviewing current State
laws, policies, practices, and environments governing public health reporting
systems, health care information systems, health data confidentiality, and
access and linkage policies. This project will identify aspects of existing
laws, policies, or practices that create barriers to standardization,
integration, data linkage, and access relating to health information systems
within and across States. Third, OASH is supporting the National Committee on
Vital and Health Statistics in a project to conduct analyses for the
development of uniform health data sets for enrollment and encounter
information from health care providers in reformed health care systems. Data
derived from the uniform data sets will provide a foundation for monitoring and
evaluating public health programs as well as systems of care in the framework
of national health information networks.
Health care reform's impact on the health care workforce
and on the AHCs, where many health professionals are trained, is the subject of
two projects. First, OASH initiated a project to study how the growth of
competitive health care financing and service delivery systems based on managed
care affects the financial support available to AHCs and graduate medical
education. The project focuses on AHCs in three cities: San Diego,
Minneapolis-St. Paul, and Washington, D.C. Another project is examining the
extent to which teaching hospitals associated with the AHCs are reorganizing
and/or reducing services and other activities in areas in which competitive
health plans hold a substantial share of the market. The project will assess
the impact of such changes on hospital training programs for
physicians.
In addition to these analytic studies, OASH will support
a series of roundtable discussions among leading national experts in
government, academic, clinical, and research settings regarding the role of
AHCs in an era of health care reform. The purpose of these discussions is to
exchange views and reach a national consensus on questions of the potential
impact of health care reform on AHCs. Many questions are being raised about
their mission, financial capability, and contribution to patient care and
public health; how they can best participate in a national shift from training
specialist physicians to training primary care practitioners; how health care
reform affects clinical research; and what strategies may best protect the
quality of clinical research and its contributions to public health.
The OASH evaluation program has also given priority to
evaluations in the area of clinical preventive services. The Office of Disease
Prevention and Health Promotion (ODPHP) will soon complete the design for the
"Put Prevention Into Practice" initiative, a package of resource materials for
health personnel in clinical settings to promote more effective use of clinical
preventive services for their patients. The design will be implemented in late
FY 1995. The results of this evaluation will help national primary care
provider organizations determine the optimal use of these materials in their
respective clinical settings.
ODPHP is also supporting a major effort to improve
methodologies for evaluating the cost-effectiveness of clinical preventive
services. A group of specialists in cost-effectiveness analysis, both inside
and outside government, is working to develop guidelines to apply
cost-effectiveness methods to clinical preventive services. The group will
address current controversies about these methods and propose alternatives for
improving the comparability of cost-effectiveness analyses.
As mentioned above, OASH often initiates evaluations of
programs or policy issues that cut across the PHS Agencies. Several OASH
projects in progress illustrate collaborative evaluations. Since 1993, OASH has
worked with the Food and Drug Administration (FDA), the National Institutes of
Health (NIH), and the Substance Abuse and Mental Health Services Administration
to evaluate the departmental methadone regulations, a project expected to be
completed in FY 1995. The study is examining the impact of Federal regulations
on the provision of methadone treatment services and on the development of new
anti-addictive medications. The final report will contain recommendations for
improving the current methadone regulations.
In another project, OASH is working with the Centers for
Disease Control and Prevention, the FDA, and NIH to study HIV transmission to
hemophiliacs through blood products. In the early period of the AIDS epidemic,
before the invention of rapid laboratory tests for identifying HIV infection,
blood transfusion services for hemophiliacs operated without effective HIV
screening or production methods to eliminate the virus. The Secretary of the
Department of Health and Human Services, Donna Shalala, launched an
investigation in 1993 to get a more complete understanding of the events that
occurred in those early years. The results, expected in FY 1995, will be
helpful for ensuring the safety of the Nation's blood supply against future
challenges.
OASH also uses evaluation funds to support projects that
measure the effectiveness of program activities managed by the various OASH
Offices. For example, the Office of Minority Health presently is evaluating its
HIV/AIDS Education/Prevention Grant Program. The grants program is providing
assistance to community-based organizations in developing innovative approaches
for reaching persons living in minority communities who are at risk for HIV
infection. The evaluation is now analyzing data on disease prevention
intervention models that have a likelihood of successful replication in other
minority communities.
Another example of OASH evaluation activity is the
efforts of the OASH Office of Research Integrity (ORI) to examine the
consequences of being accused of research misconduct, as well as the
consequences for the whistleblower. The results of these two projects will be
useful to ORI for monitoring institutional compliance with PHS regulations on
research misconduct, planning educational activities, and making improvements
in PHS policies and procedures for responding to allegations of research
misconduct.
New Directions for Evaluation The PHS
FY 1995 Strategic Plan has a goal to strengthen the public health
infrastructure. In recognition of the danger faced by health departments as a
result of years of shrinking budgets, the PHS Agencies and OASH are striving to
augment the capability of the Federal Government, tribal governments, States,
and communities to identify and address high-priority health problems for their
populations. One approach is to expand the public health knowledge base by
supporting an appropriate balance of focused and multidisciplinary research in
the areas of basic biological science, clinical medicine, public health
practice, behavioral and social sciences, epidemiology, health systems and
services, nutrition, and occupational and environmental health. In coming
years, OASH will support the development and implementation of measures that
will assess the capacity of State and local public health agencies to perform
population-based functions and the extent to which States meet PHS program
goals.
Managed care is changing the way health care is paid for
and delivered and shifting responsibilities for care. This has serious
implications not only for vulnerable populations but also for public health
"safety net" providers, academic health centers, and public health agencies.
This year, OASH will participate with the PHS Agencies and the Health Care
Financing Administration to examine the impact of managed care on public health
systems, primarily through demonstrations of the Medicaid Waiver (Section 1115)
in five States. The Medicaid Waiver allows States to experiment with
alternative approaches to improving the access and quality of health care
services for the uninsured and reducing the costs of care. The analysis is
expected to focus on the number of eligible participants, the costs to
providers, the quality and types of care, and the access for special
populations--such as American Indians and Native Alaskans, children with
special health care needs, people in underserved areas, substance abusers, and
the chronically mentally ill.