GAO-04-850, Medicare: CMS Needs Additional Authority to Adequately Oversee Patient Safety in Hospitals


This is the accessible text file for GAO report number GAO-04-850 
entitled 'Medicare: CMS Needs Additional Authority to Adequately 
Oversee Patient Safety in Hospitals' which was released on July 20, 
2004.

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov.

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately.

Report to Congressional Requesters:

United States Government Accountability Office:

GAO:

July 2004:

Medicare:

CMS Needs Additional Authority to Adequately Oversee Patient Safety in 
Hospitals:

GAO-04-850:

GAO Highlights:

Highlights of GAO-04-850, a report to congressional requesters:  

Why GAO Did This Study:

Hospitals accredited by the Joint Commission on Accreditation of 
Healthcare Organizations (JCAHO) are considered in compliance with 
Medicare participation requirements. GAO examined the extent to which 
JCAHO’s pre-2004 hospital accreditation process identified hospitals 
not complying with Medicare requirements, the potential of JCAHO’s new 
process for improving the detection of deficiencies in Medicare 
requirements, and the effectiveness of CMS’s oversight of JCAHO’s 
hospital accreditation program. GAO analyzed CMS data on hospitals 
state surveyors found to have deficiencies in Medicare requirements 
that JCAHO surveyors did not detect, analyzed CMS’s measure of JCAHO’s 
ability to detect noncompliance with Medicare requirements, and 
interviewed JCAHO officials. 

What GAO Found:

JCAHO’s pre-2004 hospital accreditation process did not identify most 
of the hospitals found by state survey agencies in CMS’s annual 
validation survey sample to have deficiencies in Medicare requirements. 
In comparing the results of the two surveys, CMS considered whether it 
was reasonable to conclude that the deficiencies found by state survey 
agencies existed at the time JCAHO surveyed the hospital. In a sample 
of 500 JCAHO-accredited hospitals, state agency validation surveys 
conducted in fiscal years 2000 through 2002 identified 31 percent (157 
hospitals) with deficiencies in Medicare requirements. Of these 157 
hospitals, JCAHO did not identify 78 percent (123 hospitals) as having 
deficiencies in Medicare requirements. For the same validation survey 
sample, JCAHO also did not identify the majority--about 69 percent--of 
deficiencies in Medicare requirements found by state agencies. 
Importantly, the number of deficiencies found by validation surveys 
represents 2 percent of the 11,000 Medicare requirements surveyed by 
state agencies in the sample during this time period. At the same 
time, a single deficiency in a Medicare requirement can limit the 
hospital’s capability to provide adequate care and ensure patient 
safety and health. Inadequacies in nursing practices or deficiencies in 
a hospital’s physical environment, which includes fire safety, are 
examples of deficiencies in Medicare requirements that could endanger 
multiple patients. 

The potential of JCAHO’s new hospital accreditation process to improve 
the detection of deficiencies in Medicare requirements is unknown 
because the process was just implemented in January 2004. JCAHO plans 
to move from using announced to unannounced surveys in 2006, which 
would afford JCAHO the opportunity to observe hospitals’ operations 
when the hospitals have not prepared in advance to be surveyed. In 
addition, the pilot test of the new accreditation process was of 
limited value in predicting whether it will be an improvement over the 
pre-2004 process in detecting deficiencies. Limitations in the pilot 
test included that hospitals were not randomly selected to participate; 
that observers from JCAHO accompanied each surveyor, thus possibly 
affecting surveyors’ actions; and that JCAHO evaluated the results 
instead of an independent entity. 

CMS has limited oversight authority over JCAHO’s hospital 
accreditation program because the program’s unique legal status 
effectively prevents CMS from taking actions that it has the authority 
to take with other health care accreditation programs to ensure 
satisfactory performance. For example, requiring JCAHO’s hospital 
accreditation program to submit to a direct review process or placing 
the program on probation while monitoring its performance. Further, 
CMS relies on a measure to evaluate how well JCAHO’s hospital 
accreditation program detects deficiencies in Medicare requirements 
that provides limited information and can mask problems with program 
performance, uses statistical methods that are insufficient to assess 
JCAHO’s performance, and has reduced the number of validation surveys 
it conducts.

What GAO Recommends:

GAO believes that Congress should consider giving CMS the authority 
over JCAHO’s hospital accreditation program that it has over other 
accreditation programs and recommends that CMS modify its methods for 
assessing JCAHO’s performance. CMS agreed with GAO’s recommendations. 
JCAHO stated that GAO’s methodology was incomplete and did not 
comprehensively assess its overall performance. GAO emphasized that its 
engagement was limited to one aspect of deficiency detection and was 
not intended to reflect JCAHO’s overall performance. 

www.gao.gov/cgi-bin/getrpt?GAO-04-850.

To view the full product, including the scope and methodology, click on 
the link above. For more information, contact Janet Heinrich at (202) 
512-7119.

[End of section]

Contents:

Letter:

Results in Brief:

Background:

JCAHO's Pre-2004 Hospital Accreditation Process Often Did Not Detect 
Serious Deficiencies Found by State Survey Agencies:

Potential of JCAHO's New Hospital Accreditation Process Is Unknown, and 
Testing Was Limited:

CMS Oversight Authority of JCAHO's Hospital Accreditation Program Is 
Limited and Needs Improvement:

Conclusions:

Matter for Congressional Consideration:

Recommendations for Executive Action:

Agency and Other External Comments and Our Evaluation:

Appendix I: Scope and Methodology:

Appendix II: Medicare Conditions of Participation:

Appendix III: Features of JCAHO's New Accreditation Process:

Appendix IV: Comments from the Centers for Medicare & Medicaid 
Services:

Appendix V: Comments from the Joint Commission on Accreditation of 
Healthcare Organizations:

Appendix VI: GAO Contact and Staff Acknowledgments:

GAO Contact:

Acknowledgments:

Related GAO Products:

Tables:

Table 1: Hospitals in CMS's Validation Survey Sample with Serious 
Deficiencies that State Survey Agencies Identified but JCAHO Surveyors 
Did Not, Fiscal Years 2000-2002:

Table 2: Percentage of Serious Deficiencies Identified by State Survey 
Agencies but Not by JCAHO Surveyors in CMS's Validation Survey Sample, 
Fiscal Years 2000-2002:

Table 3: Number of Serious Deficiencies, by COP, Identified by State 
Survey Agencies but Not by JCAHO Surveyors in CMS's Validation Survey 
Sample, Fiscal Years 2000-2002:

Table 4: Accreditation Decisions for Hospitals Surveyed Under JCAHO's 
New Survey Process Pilot Test as Compared to Results from JCAHO's Pre-
2004 Survey Process:

Table 5: Hypothetical Examples of the Effect on the Rate of Disparity 
of a Decrease in the Number of Hospitals with Serious Deficiencies in a 
Sample of 200 Hospitals:

Table 6: Number of Hospitals Targeted for Validation Surveys Compared 
with Usable Traditional Validation Surveys Completed:

Table 7: Medicare Conditions of Participation:

Table 8: JCAHO's Description of Features of Its New Hospital 
Accreditation Process:

Abbreviations:

AOA: American Osteopathic Association: 
CMS: Centers for Medicare & Medicaid Services: 
COP: condition of participation: 
HHS: Department of Health and Human Services: 
JCAHO: Joint Commission on Accreditation of Healthcare Organizations: 
OIG: Office of Inspector General: 
PFP: priority focus process: 
PPR: periodic performance review:

United States Government Accountability Office:

Washington, DC 20548:

July 20, 2004:

The Honorable Charles E. Grassley: 
Chairman: 
Committee on Finance: 
United States Senate:

The Honorable Pete Stark: 
Ranking Minority Member: 
Subcommittee on Health: 
Committee on Ways and Means: 
House of Representatives:

In fiscal year 2002, nearly 7.4 million Medicare beneficiaries received 
inpatient health care at hospitals that participated in Medicare. 
Federal law establishes criteria for hospitals for purposes of 
Medicare. The Centers for Medicare and Medicaid Services (CMS), the 
agency responsible for administering Medicare, has established quality 
and patient safety requirements called conditions of participation 
(COP) that hospitals must meet in order to be eligible for Medicare 
payment. Hospitals that are accredited by the Joint Commission on 
Accreditation of Healthcare Organizations (JCAHO) are generally deemed 
under federal law to be compliant with Medicare requirements for 
patient safety and health and become eligible for payments from 
Medicare.[Footnote 1],[Footnote 2] No other health care accreditation 
program has this same statutory authority.

JCAHO is a private, not-for-profit organization that accredits most of 
the hospitals that participate in Medicare. JCAHO sets standards that 
accredited hospitals must meet and reports that these standards are 
more comprehensive than the Medicare COPs.[Footnote 3] In January 2004, 
JCAHO implemented a new hospital accreditation process with goals that 
included further enhancing health care quality and safety.

CMS oversight of JCAHO's hospital accreditation program is limited 
because it cannot restrict or remove JCAHO's accreditation authority if 
the agency detects problems. To oversee the program, CMS conducts on-
site validation surveys of a sample of JCAHO-accredited hospitals and 
reports annually to Congress on the results of these surveys. The 
validation surveys, which are performed by agencies that CMS has 
agreements with in each state, help CMS determine whether Medicare 
quality and safety requirements are being met. CMS compares the results 
of these state surveys against survey results obtained through JCAHO's 
hospital accreditation program. CMS uses a measure called the rate of 
disparity that summarizes the extent to which an accreditation program 
has failed to cite deficiencies identified by state agency validation 
surveys. We are using the term serious deficiency in this report to 
indicate a deficiency in one or more Medicare COPs. Examples of serious 
deficiencies include a hospital's inability to provide adequate nursing 
services or failure to implement and enforce infection control 
policies. According to CMS, serious deficiencies substantially limit a 
hospital's capability to render adequate care and adversely affect the 
safety and health of patients.

Questions have been raised by the Department of Health and Human 
Services' (HHS) Office of Inspector General (OIG) and others regarding 
whether accreditation by JCAHO ensures that hospitals provide adequate 
care. Specifically, experts have questioned how well JCAHO's hospital 
accreditation process identifies deficiencies in hospitals that could 
jeopardize patient safety and health. A comprehensive study by the HHS 
OIG found that JCAHO's surveys were not likely to identify patterns of 
deficient care.[Footnote 4]

You asked that we examine the effectiveness of JCAHO's hospital 
accreditation process in ensuring that hospitals comply with Medicare 
COPs to ensure the safety and health of Medicare beneficiaries. 
Specifically, we (1) examined the extent to which JCAHO's pre-2004 
hospital accreditation process identified deficiencies in Medicare COPs 
that were identified by state survey agencies, (2) determined whether 
JCAHO's new hospital accreditation process has potential for improving 
the detection of deficiencies in Medicare COPs and whether the process 
was adequately tested, and (3) examined the effectiveness of CMS's 
oversight of JCAHO's hospital accreditation program.

To determine the extent to which JCAHO's pre-2004 hospital 
accreditation process identified deficiencies in Medicare COPs that 
were identified by state survey agencies, we used data from a CMS 
comparison of state validation survey findings with findings of JCAHO's 
hospital accreditation surveys, which indicated whether JCAHO found 
deficiencies in its standards. Of the four possible outcomes to this 
comparison of survey findings--(1) JCAHO and state agencies both 
identify no deficiencies, (2) JCAHO identifies deficiencies not found 
by state agencies, (3) JCAHO and state agencies both identify the same 
deficiencies, and (4) state agencies identify deficiencies that JCAHO 
does not--we focused on the fourth because it highlights the need for 
CMS oversight of the hospital accreditation program. For the second 
outcome, there could be two reasons for the disparity between JCAHO's 
and state survey agencies' findings: hospitals corrected deficiencies 
identified by JCAHO prior to the state agency survey or the state 
survey agency did not identify a deficiency that existed. In addition, 
not all JCAHO findings are equivalent to noncompliance with a Medicare 
COP. To determine whether JCAHO's findings on deficiencies in its 
standards were comparable to the state agencies' findings, CMS staff 
compared the two surveys and considered whether it was reasonable to 
conclude that the deficiencies found by state survey agencies existed 
at the time JCAHO surveyed the hospital. For deficiencies that CMS 
determined that JCAHO failed to identify, CMS met with JCAHO to address 
disputed findings and consider additional evidence on comparability 
offered by JCAHO. CMS provided results for a sample of 500 JCAHO-
accredited hospitals from fiscal years 2000 through 2002. We determined 
that the data CMS provided on serious deficiencies were adequate for 
addressing the issues in this report. On the basis of this sample of 
500 JCAHO-accredited hospitals, we determined, using CMS's data, both 
the percentage of serious deficiencies and the percentage of hospitals 
with serious deficiencies identified by the state survey agencies where 
JCAHO surveyors did not find comparable deficiencies. The analysis we 
performed on the results of the validation surveys was limited to the 
hospitals included in the validation survey sample and cannot be 
generalized to all JCAHO-accredited hospitals.

To determine whether JCAHO's new hospital accreditation process has 
potential for improving the detection of serious deficiencies, we 
identified changes in the accreditation process and analyzed 
significant new features. To determine whether JCAHO's new hospital 
accreditation process was adequately tested, we reviewed the testing 
procedures and results that JCAHO used to determine the effectiveness 
of its new survey process in identifying quality and safety 
deficiencies. Because the new accreditation process was implemented 
recently, we did not have information to compare JCAHO survey 
performance in detecting serious deficiencies with state agency survey 
performance.

To determine the adequacy of CMS's oversight of JCAHO's hospital 
accreditation program, we reviewed relevant statutory and regulatory 
provisions regarding oversight of health care accreditation programs 
and how CMS had implemented this authority in order to provide 
oversight. To supplement our review, we conducted interviews with 
officials from CMS, state survey agencies, and JCAHO; representatives 
from other organizations active in health care accreditation and the 
hospital industry; and experts in quality of care. We conducted our 
work from June 2003 through July 2004 in accordance with generally 
accepted government auditing standards. (For a complete description of 
our scope and methodology, see app. I.)

Results in Brief:

JCAHO's pre-2004 hospital accreditation process did not identify most 
of the hospitals found by state survey agencies in CMS's annual 
validation survey sample to have serious deficiencies in Medicare COPs. 
In a sample of 500 JCAHO-accredited hospitals, state agency validation 
surveys conducted in fiscal years 2000 through 2002 identified 31 
percent (157 hospitals) with serious deficiencies; of these, JCAHO did 
not identify 78 percent (123 hospitals) as having serious deficiencies. 
For the same validation survey sample, JCAHO also did not identify the 
majority-about 69 percent-of serious deficiencies found by state 
agencies. Importantly, the number of deficiencies found by validation 
surveys represents 2 percent of the 11,000 Medicare COPs surveyed by 
state agencies in the sample during this time period. At the same time, 
a single serious deficiency can limit a hospital's capability to 
provide adequate care and ensure patient safety and health. 
Inadequacies in nursing practices or deficiencies in a hospital's 
physical environment, which includes fire safety, are examples of 
serious deficiencies that could endanger multiple patients.

The potential of JCAHO's new hospital accreditation process to improve 
the detection of serious deficiencies over the pre-2004 process is 
unknown because the process was just implemented in January 2004. JCAHO 
plans to move from announced to unannounced surveys in 2006, which 
would afford JCAHO the opportunity to observe hospitals' operations 
when the hospitals have not prepared in advance to be surveyed. In 
addition, the pilot test of the new accreditation process was of 
limited value in predicting whether it will be an improvement over the 
pre-2004 process in detecting deficiencies. Limitations in the pilot 
test included that hospitals participating in the pilot were not 
randomly selected and that JCAHO evaluated the results instead of an 
independent entity.

CMS has limited oversight authority over JCAHO's hospital accreditation 
program because the program's unique legal status effectively prevents 
CMS from taking actions, such as requiring JCAHO's hospital 
accreditation program to submit to a direct review process or placing 
the program on probation while monitoring its performance, that it has 
the authority to take with other health care accreditation programs to 
ensure satisfactory performance. Furthermore, CMS's existing oversight 
of JCAHO's hospital accreditation program needs improvement. Although 
CMS officials said that validation surveys are conducted to assure 
Congress that JCAHO's accreditation process provides a reasonable 
assurance that hospitals comply with Medicare requirements, there are 
limitations to the agency's validation survey program. CMS has no 
formal written protocol for selecting the hospitals to include in the 
state agency validation survey sample; relies on a measure--the rate of 
disparity--that provides limited information and could mask problems 
with an accreditation program's performance in detecting serious 
deficiencies; uses statistical methods that are insufficient to 
accurately portray JCAHO's performance; and has reduced the percentage 
of validation surveys from 5 percent to approximately 1 percent of 
JCAHO-accredited hospitals, which provides less reliable information on 
the performance of JCAHO's hospital accreditation program.

We suggest that Congress consider giving CMS the same oversight 
authority over JCAHO's hospital accreditation program that CMS has for 
all other health care accreditation programs. To improve CMS's 
assessment of JCAHO's hospital accreditation process, we recommend that 
CMS modify the measure it uses to indicate how well an accreditation 
program detects serious deficiencies in Medicare COPs; maximize the 
extent to which validation survey findings can be generalized to all 
JCAHO-accredited hospitals and include its survey protocol in its 
annual reports to Congress; and annually conduct validation surveys on 
a sample of JCAHO-accredited hospitals that is equal to at least 5 
percent of all JCAHO-accredited hospitals.

CMS and JCAHO commented on a draft of this report. In its comments, CMS 
concurred with our findings and recommendations. JCAHO stated that it 
did not object to our matter for congressional consideration that CMS 
be given the same oversight authority over JCAHO's hospital 
accreditation program that it has over other health care accreditation 
programs. JCAHO took issue with our methodology, which it said was 
incomplete and did not comprehensively assess the performance of 
JCAHO's hospital accreditation program. Our review was not intended to 
be a comprehensive evaluation of JCAHO's hospital accreditation 
program. Rather, we focused on the ability of JCAHO's hospital 
accreditation program to ensure that hospitals that accept Medicare 
patients comply with Medicare COPs. In the same vein, JCAHO stated that 
the report does not sufficiently recognize JCAHO's identification of 
deficiencies in its surveys that may be corrected before state 
surveyors arrive. We added language to the report to emphasize that our 
focus was on the serious deficiencies state survey agencies found that 
JCAHO did not because these serious deficiencies demonstrate the 
importance of CMS oversight of the hospital accreditation process. 
JCAHO also stated that we misrepresented the potential of its new 
accreditation process to detect deficiencies in Medicare COPs and 
provided new data for the first quarter of 2004 that indicate that 2004 
JCAHO surveys may have detected a greater percentage of deficiencies 
related to patient care compared with the pre-2004 accreditation 
process. However, we maintain that until CMS validation surveys for 
2004 are completed, there is no basis on which to determine whether the 
new process improves the detection of noncompliance with Medicare COPs. 
CMS and JCAHO also provided technical comments on the report, which we 
incorporated as appropriate.

Background:

To participate in Medicare, hospitals must maintain standards of 
patient safety and health that comply with Medicare COPs. For example, 
the COP related to nursing services includes such requirements for 
hospitals as providing a 24-hour nursing service that is supervised or 
furnished by a registered nurse. There are currently 23 Medicare 
COPs.[Footnote 5] (See app. II for a description of the 23 Medicare 
COPs.) CMS proposed revisions to all of the COPs in 1997, but it did 
not finalize them. Since then, CMS has revised several of the COPs, 
including those concerning the life safety code; quality assessment and 
performance improvement; organ, tissue, and eye donations; and nurse 
anesthetist supervision.

Health care accreditation programs other than JCAHO's hospital 
accreditation program may generally adopt their own requirements if CMS 
determines that an accreditation program's requirements are at least 
equivalent to Medicare COPs.[Footnote 6] If CMS also determines, among 
other things, that the accreditation program's survey process is likely 
to identify any serious deficiencies in COPs, it must generally grant 
"deeming authority" to the accreditation program and treat entities 
accredited by these organizations as meeting Medicare COPs. CMS has the 
authority to review these programs, and it can impose a probationary 
period while monitoring performance and remove deeming authority if 
warranted.

JCAHO:

Most hospitals demonstrate compliance with standards equivalent to 
Medicare COPs through accreditation by JCAHO.[Footnote 7] In 2002, 
JCAHO accredited 4,211, or 82 percent, of Medicare-participating 
hospitals.[Footnote 8] Hospitals accredited by JCAHO received payments 
for Medicare-covered inpatient services of approximately $98 billion, 
or 90 percent, of the $109 billion that was spent on hospital care in 
2002. JCAHO, as part of its accreditation-related activities, also 
develops survey procedures, trains its surveyors, and formulates 
performance measures. JCAHO is governed by a 29-member board of 
commissioners and has a staff of over 1,000.[Footnote 9]

JCAHO's deeming authority for hospitals is established in statute and 
therefore can only be changed by Congress. As a result of this unique 
statutory authority, hospitals accredited by JCAHO-because they meet 
JCAHO standards-are deemed to meet Medicare COPs as well.[Footnote 10] 
In contrast, the American Osteopathic Association (AOA)--a private, 
not-for-profit professional organization that offers accreditation 
services for hospitals and other health care organizations--holds 
deeming authority that is subject to CMS's direct review and 
approval.[Footnote 11] While hospital accreditation is its largest 
program, JCAHO also has accreditation authority under Medicare for 
certain other health care providers, including clinical laboratories, 
hospices, ambulatory surgical centers, and home health care agencies. 
All of these other JCAHO accreditation programs are subject to CMS's 
direct review and approval.

To be accredited by JCAHO, a hospital must meet eligibility 
requirements, satisfactorily complete a triennial on-site survey 
process, and continue to maintain JCAHO's standards between surveys. 
The accreditation surveys that JCAHO conducts every 3 years are 
particularly important. For most hospitals, the triennial survey is the 
only time that JCAHO conducts an on-site review of the hospital's 
compliance with all quality standards and issues decisions on how well 
the hospital has complied with JCAHO's standards. In 2004, JCAHO 
implemented a new hospital accreditation survey process, which, 
according to JCAHO, is intended to reduce the cost of accreditation to 
health care organizations and JCAHO, enhance public confidence that 
health care organizations are in continuous compliance with standards, 
increase the real and perceived value of accredited organizations, meet 
the requirements of deeming authorities and purchasers, and improve 
satisfaction for hospitals participating in the accreditation program.

CMS Oversight of JCAHO:

CMS exercises oversight of JCAHO's hospital accreditation program 
primarily through its validation surveys and annual reports to 
Congress. Under federal law, CMS must continually study the operation 
and administration of Medicare, including validating the JCAHO hospital 
accreditation process, and submit annual reports to Congress.[Footnote 
12] CMS has agreements with state agencies to conduct validation 
surveys. There are different kinds of validation surveys, including 
traditional validation surveys-surveys conducted on a sample of 
hospitals within 60 days of their triennial JCAHO survey. [Footnote 13] 
Traditional validation surveys provide the basis for assessing the 
effectiveness of JCAHO's hospital accreditation process in detecting 
deficiencies in Medicare COPs, which JCAHO-accredited hospitals are 
treated as meeting. Validation surveys also include 18-month surveys, 
which monitor how well JCAHO-accredited hospitals are complying with 
Medicare COPs midway between their 3-year JCAHO surveys, and allegation 
surveys, which are triggered by complaints or other reports of 
situations that pose potential threats to patient health and safety in 
JCAHO-accredited hospitals. CMS has the authority to remove the deemed 
status of a JCAHO-accredited hospital where a state agency's validation 
survey results in a finding that the hospital is out of compliance with 
one or more Medicare COP.

CMS uses a rate of disparity measure to summarize the extent to which 
an accreditation program, such as JCAHO's hospital accreditation 
program, has not found serious deficiencies identified by CMS through 
state agency validation surveys. For a hospital accreditation program, 
using the results from validation surveys, the rate of disparity for 
hospitals surveyed by the state survey agencies is calculated as the 
difference between the number of hospitals found with serious 
deficiencies by state agencies and the number of hospitals found with 
comparable deficiencies by the accreditation program, divided by the 
number of hospitals sampled. CMS regulations provide that if the 
validation survey results for an accreditation organization with 
deeming authority indicate a rate of disparity that reaches the 
threshold level of 20 percent disparity or greater, CMS will notify the 
organization that its deeming authority may be in jeopardy and that the 
agency is initiating a deeming authority review.[Footnote 14] With 
respect to JCAHO, CMS includes the rate of disparity in its annual 
reports to Congress in which it reports the results of its validation 
program for JCAHO's hospital accreditation program.

JCAHO's Pre-2004 Hospital Accreditation Process Often Did Not Detect 
Serious Deficiencies Found by State Survey Agencies:

JCAHO's pre-2004 hospital accreditation process often did not identify 
either hospitals with serious deficiencies or the individual serious 
deficiencies found by state survey agencies through CMS's validation 
program. In a sample of 500 JCAHO-accredited hospitals, state agency 
validation surveys conducted in fiscal years 2000 through 2002 
identified 31 percent (157 hospitals) with serious deficiencies; of 
these, JCAHO did not identify 78 percent (123 hospitals) as having 
serious deficiencies. For the same validation survey sample, the 
majority of the serious deficiencies state survey agencies identified 
but JCAHO did not were in the physical environment COP category, which 
covers fire safety and prevention.

JCAHO Did Not Identify Three-Quarters of the Hospitals That State 
Survey Agencies Found to Have Serious Deficiencies:

From fiscal years 2000 through 2002, JCAHO did not identify 123 of the 
157 hospitals (78 percent) with serious deficiencies that CMS's 
validation program identified out of a sample of 500 JCAHO-accredited 
hospitals. Table 1 shows the hospitals with serious deficiencies that 
state survey agencies identified and JCAHO did not during fiscal years 
2000 through 2002. In 343 of the 500 hospital validation surveys, state 
agency surveyors did not find serious deficiencies. Both state agency 
surveyors and JCAHO surveyors identified 34 hospitals as having a 
serious deficiency.

Table 1: Hospitals in CMS's Validation Survey Sample with Serious 
Deficiencies that State Survey Agencies Identified but JCAHO Surveyors 
Did Not, Fiscal Years 2000-2002:

Fiscal year: 2000; 
Number of hospitals in CMS's validation sample: 184; 
Hospitals state survey agencies found to have serious deficiencies: 
Number: 61; 
Hospitals state survey agencies found to have serious deficiencies: 
Percent: 33; 
Hospitals with serious deficiencies identified by state survey agencies 
but not identified by JCAHO[A]: Number: 49; 
Hospitals with serious deficiencies identified by state survey agencies 
but not identified by JCAHO[A]: Percent: 80.

Fiscal year: 2001; 
Number of hospitals in CMS's validation sample: 204; 
Hospitals state survey agencies found to have serious deficiencies: 
Number: 61; 
Hospitals state survey agencies found to have serious deficiencies: 
Percent: 30; 
Hospitals with serious deficiencies identified by state survey agencies 
but not identified by JCAHO[A]: Number: 49; 
Hospitals with serious deficiencies identified by state survey agencies 
but not identified by JCAHO[A]: Percent: 80.

Fiscal year: 2002; 
Number of hospitals in CMS's validation sample: 112; 
Hospitals state survey agencies found to have serious deficiencies: 
Number: 35; 
Hospitals state survey agencies found to have serious deficiencies: 
Percent: 31; 
Hospitals with serious deficiencies identified by state survey agencies 
but not identified by JCAHO[A]: Number: 25; 
Hospitals with serious deficiencies identified by state survey agencies 
but not identified by JCAHO[A]: Percent: 71.

Total; 
Number of hospitals in CMS's validation sample: 500; 
Hospitals state survey agencies found to have serious deficiencies: 
Number: 157; 
Hospitals state survey agencies found to have serious deficiencies: 
Percent: 31; 
Hospitals with serious deficiencies identified by state survey agencies 
but not identified by JCAHO[A]: Number: 123; 
Hospitals with serious deficiencies identified by state survey agencies 
but not identified by JCAHO[A]: Percent: 78. 

Source: GAO analysis of CMS data.

Note: Hospitals with serious deficiencies are defined as those not 
meeting one or more of the Medicare COPs. From fiscal year 2000 through 
2002, JCAHO surveyed 4,666 hospitals for accreditation.

[A] Determined by CMS through its matching of deficient COPs found by 
state agency surveyors to JCAHO surveyors' findings of JCAHO standards 
out of compliance.

[End of table]

According to JCAHO, disparity between state agency and JCAHO findings 
in the 123 hospitals in part may be attributed to the timing of the two 
surveys, JCAHO's phasing in of new requirements, different 
interpretations of the COPs by state surveyors, and inherent surveyor 
bias. However, in its comparison to determine disparity between the two 
surveys, CMS does consider whether it is reasonable to conclude that 
the deficiencies found by state survey agencies existed at the time 
JCAHO surveyed the hospital.

JCAHO Did Not Detect Two-Thirds of the Serious Deficiencies Identified 
by State Survey Agencies:

From fiscal year 2000 through 2002, JCAHO did not detect 167 of the 241 
serious deficiencies (69 percent) identified through CMS's validation 
program from a sample of 500 JCAHO-accredited hospitals. The number of 
serious deficiencies found by CMS's validation program represents 2 
percent of the 11,000 Medicare COPs surveyed by state agencies in the 
sample and were found in 157 hospitals. However, one serious deficiency 
in any one of these hospitals could limit its ability to provide 
adequate care to its patients. For example, a serious deficiency in the 
nursing services COP at a hospital in Texas found by a state agency but 
missed by JCAHO in 2000 included such problems as failure to prepare 
and administer drugs in accordance with federal and state laws, 
inadequate supervision and evaluation of the clinical activities of 
nonemployee nursing personnel, and nursing care and procedures provided 
to patients that were not within the scope of accepted standards of 
practice. Among hospitals with serious deficiencies identified by CMS's 
validation program but not by JCAHO, there were on average 1.1 serious 
deficiencies per hospital, with a range from 1 to 6. Table 2 shows the 
percentage of serious deficiencies identified by CMS's validation 
program but not by JCAHO for fiscal years 2000 through 2002.

Table 2: Percentage of Serious Deficiencies Identified by State Survey 
Agencies but Not by JCAHO Surveyors in CMS's Validation Survey Sample, 
Fiscal Years 2000-2002:

Fiscal year: 2000; 
Number of serious deficiencies identified by state survey agencies: 82; 
Number of serious deficiencies identified by JCAHO: 12; 
Serious deficiencies identified by state survey agencies but not by 
JCAHO[A]: Number: 70; 
Serious deficiencies identified by state survey agencies but not by 
JCAHO[A]: Percent: 85%. 

Fiscal year: 2001; 
Number of serious deficiencies identified by state survey agencies: 
103; 
Number of serious deficiencies identified by JCAHO: 40; 
Serious deficiencies identified by state survey agencies but not by 
JCAHO[A]: Number: 63; 
Serious deficiencies identified by state survey agencies but not by 
JCAHO[A]: Percent: 61%. 

Fiscal year: 2002; 
Number of serious deficiencies identified by state survey agencies: 56; 
Number of serious deficiencies identified by JCAHO: 22; 
Serious deficiencies identified by state survey agencies but not by 
JCAHO[A]: Number: 34; 
Serious deficiencies identified by state survey agencies but not by 
JCAHO[A]: Percent: 61%. 

Fiscal year: Total; 
Number of serious deficiencies identified by state survey agencies: 
241; 
Number of serious deficiencies identified by JCAHO: 74; 
Serious deficiencies identified by state survey agencies but not by 
JCAHO[A]: Number: 167; 
Serious deficiencies identified by state survey agencies but not by 
JCAHO[A]: Percent: 69%. 

Source: GAO analysis of CMS data.

Note: Hospitals with serious deficiencies are defined as those not 
meeting one or more of the Medicare COPs.

[A] Determined by CMS through its matching of deficient COPs found by 
state agency surveyors to JCAHO surveyors' findings of JCAHO standards 
out of compliance.

[End of table]

Of the 167 serious deficiencies identified by CMS's validation program 
from fiscal year 2000 through 2002 but not detected by JCAHO, 87 were 
related to a hospital's physical environment, which includes life 
safety code standards on fire prevention and safety.[Footnote 15] For 
these 3 years, JCAHO did not detect 81 percent of the serious physical 
environment deficiencies identified by state agency surveyors. Table 3 
shows the number of serious deficiencies, by category, identified by 
state survey agencies in CMS's validation program but missed by JCAHO 
surveyors. The larger number of deficiencies in physical environment 
may be related to the difference in how state agencies generally survey 
separately a hospital's compliance with the life safety code portion of 
the physical environment COP. JCAHO surveys assess compliance with the 
life safety code using a combination of the hospital's self-assessment, 
a hospital building tour, and observations made by all surveyors during 
the survey process. Examples of deficiencies in physical environment 
that JCAHO did not identify but CMS's validation program found in a 
hospital in Alabama in 2000 included the following: several exterior 
exits lacked emergency exit lighting; several exterior exits were 
illuminated only by single light bulbs; fire alarm system and fire 
extinguishers had not been inspected annually as required; and an 
automatic sprinkler system had not been inspected annually and 
maintained by certified personnel as required. Serious deficiencies in 
the COP on physical environment compromise patient safety and health.

Table 3: Number of Serious Deficiencies, by COP, Identified by State 
Survey Agencies but Not by JCAHO Surveyors in CMS's Validation Survey 
Sample, Fiscal Years 2000-2002:

COP: Physical environment; 
Number of serious deficiencies identified by state survey agencies: 
107; 
Number of serious deficiencies identified by state survey agencies but 
not by JCAHO[A]: 87.

COP: Quality of care: Anesthesia services; 
Number of serious deficiencies identified by state survey agencies: 3; 
Number of serious deficiencies identified by state survey agencies but 
not by JCAHO[A]: 2.

COP: Quality of care: Discharge planning; 
Number of serious deficiencies identified by state survey agencies: 2; 
Number of serious deficiencies identified by state survey agencies but 
not by JCAHO[A]: 2.

COP: Quality of care: Emergency services; 
Number of serious deficiencies identified by state survey agencies: 2; 
Number of serious deficiencies identified by state survey agencies but 
not by JCAHO[A]: 2.

COP: Quality of care: Food and dietetic services; 
Number of serious deficiencies identified by state survey agencies: 5; 
Number of serious deficiencies identified by state survey agencies but 
not by JCAHO[A]: 4.

COP: Quality of care: Governing body; 
Number of serious deficiencies identified by state survey agencies: 16; 
Number of serious deficiencies identified by state survey agencies but 
not by JCAHO[A]: 7.

COP: Quality of care: Infection control; 
Number of serious deficiencies identified by state survey agencies: 15; 
Number of serious deficiencies identified by state survey agencies but 
not by JCAHO[A]: 9.

COP: Quality of care: Laboratory services; 
Number of serious deficiencies identified by state survey agencies: 1; 
Number of serious deficiencies identified by state survey agencies but 
not by JCAHO[A]: 1.

COP: Quality of care: Medical record services; 
Number of serious deficiencies identified by state survey agencies: 7; 
Number of serious deficiencies identified by state survey agencies but 
not by JCAHO[A]: 4.

COP: Quality of care: Medical staff; 
Number of serious deficiencies identified by state survey agencies: 10; 
Number of serious deficiencies identified by state survey agencies but 
not by JCAHO[A]: 1.

COP: Quality of care: Nursing services; 
Number of serious deficiencies identified by state survey agencies: 17; 
Number of serious deficiencies identified by state survey agencies but 
not by JCAHO[A]: 10.

COP: Quality of care: Organ, tissue, and eye procurement; 
Number of serious deficiencies identified by state survey agencies: 5; 
Number of serious deficiencies identified by state survey agencies but 
not by JCAHO[A]: 5.

COP: Quality of care: Outpatient services; 
Number of serious deficiencies identified by state survey agencies: 1; 
Number of serious deficiencies identified by state survey agencies but 
not by JCAHO[A]: 1.

COP: Quality of care: Patients' rights; 
Number of serious deficiencies identified by state survey agencies: 10; 
Number of serious deficiencies identified by state survey agencies but 
not by JCAHO[A]: 9.

COP: Quality of care: Pharmaceutical services; 
Number of serious deficiencies identified by state survey agencies: 14; 
Number of serious deficiencies identified by state survey agencies but 
not by JCAHO[A]: 9.

COP: Quality of care: Quality assurance; 
Number of serious deficiencies identified by state survey agencies: 18; 
Number of serious deficiencies identified by state survey agencies but 
not by JCAHO[A]: 8.

COP: Quality of care: Radiologic services; 
Number of serious deficiencies identified by state survey agencies: 1; 
Number of serious deficiencies identified by state survey agencies but 
not by JCAHO[A]: 0.

COP: Quality of care: Rehabilitation services; 
Number of serious deficiencies identified by state survey agencies: 1; 
Number of serious deficiencies identified by state survey agencies but 
not by JCAHO[A]: 1.

COP: Quality of care: Respiratory care services; 
Number of serious deficiencies identified by state survey agencies: 1; 
Number of serious deficiencies identified by state survey agencies but 
not by JCAHO[A]: 1.

COP: Quality of care: Surgical services; 
Number of serious deficiencies identified by state survey agencies: 5; 
Number of serious deficiencies identified by state survey agencies but 
not by JCAHO[A]: 4.

COP: Quality of care: Total quality-of-care COPs; 
Number of serious deficiencies identified by state survey agencies: 134; 
Number of serious deficiencies identified by state survey agencies but 
not by JCAHO[A]: 80. 

Source: GAO analysis of CMS data.

Note: Neither state survey agencies nor JCAHO identified serious 
deficiencies in two of the categories-compliance with laws and nuclear 
medicine services-which are not included in this table.

[A] Determined by CMS through its matching of deficient COPs found by 
state agency surveyors to JCAHO surveyors' findings of JCAHO standards 
out of compliance.

[End of table]

The total number of deficiencies not identified by JCAHO in the 
quality-of-care COP categories--those COPs that involve the oversight 
and delivery of patient care--is similar to the number not identified 
by JCAHO in the physical environment COP. While the number of serious 
deficiencies not found by JCAHO in individual quality-of-care COP 
categories is smaller than the number not found in physical 
environment, when these quality-of-care COPs are combined, the 
proportion of serious deficiencies JCAHO missed is almost 60 percent of 
the total number of serious deficiencies identified by state survey 
agencies. The following are examples of hospitals found to be out of 
compliance with multiple quality-of-care COPs:

* In 2000, CMS removed the deemed status as a Medicare provider of a 
JCAHO-accredited hospital in California for failure to comply with two 
COPs, one of which was infection control. The hospital failed to 
provide a sanitary environment to avoid sources and transmission of 
infections and communicable diseases and failed to develop a system for 
ensuring the sterilization of medical instruments.

* Also in 2000, CMS notified a hospital in Texas that if it did not 
implement a plan of correction the hospital's participation in the 
Medicare program would be terminated. Serious deficiencies at this 
hospital included lack of compliance with the pharmaceutical services 
and nursing services COPs because medications were administered without 
physician orders and a double dose of narcotics was given in the 
emergency room, with no explanation for the excessive dosage, to a 
patient who later died.

State surveyors in CMS's validation program also may miss serious 
deficiencies. In related work on skilled nursing facilities and home 
health agencies, we found that the number of serious deficiencies found 
by state agencies was highly variable among states and may be 
understated.[Footnote 16] State agencies' detection of serious 
deficiencies in hospitals also varied widely among states for the 3 
years we reviewed. For example, state survey agencies in California, 
Illinois, and Ohio found serious deficiencies in over 45 percent of the 
surveys they conducted between fiscal years 2000 through 2002. In 
contrast, Florida and New York found serious deficiencies in less than 
10 percent of the surveys they conducted, and Louisiana did not find 
serious deficiencies in any of the surveys it conducted.[Footnote 17]

Potential of JCAHO's New Hospital Accreditation Process Is Unknown, and 
Testing Was Limited:

The potential of JCAHO's new hospital accreditation process to improve 
the identification of serious deficiencies is unknown because it is too 
soon after its January 2004 implementation for a meaningful evaluation; 
in addition, JCAHO's testing of the new process was limited. CMS has 
not had the opportunity to complete its validation program for 2004 to 
determine whether JCAHO surveyors using the new process are missing 
serious deficiencies later identified by state agency validation 
surveys. While unannounced surveys, which are planned for 
implementation in 2006, have the potential to improve the detection of 
serious deficiencies, other features of the new process that JCAHO did 
not test before implementation may have limitations that could affect 
the potential of the new process to identify problems with patient 
care. JCAHO's pilot test of the new process had limitations, including 
using a sample of hospitals that volunteered for the pilot instead of 
using a random sample and self-evaluating the results instead of using 
an independent entity.

Potential of New Process Is Unknown:

Because JCAHO's new accreditation process was implemented in January 
2004, it is too soon to know whether the new process is better at 
detecting serious deficiencies in Medicare COPs than the pre-2004 
accreditation process. A JCAHO official told us the new process will 
aid in the detection of deficiencies, but we found that some of the 
features may have shortcomings that could limit their effectiveness. 
New features of the accreditation process include the hospital's self-
assessment of compliance with accreditation standards midway through 
the accreditation cycle, surveyor review of the care provided to 
specific patients to determine the adequacy of the hospital's health 
care delivery system, and performance of all accreditation surveys on 
an unannounced basis beginning in 2006. (See app. III for a description 
of selected new features of JCAHO's new hospital accreditation 
process.)

Periodic Performance Review:

Periodic performance reviews assess hospital compliance with applicable 
standards and are performed at the 18-month midpoint between 3-year on-
site accreditation surveys. According to JCAHO, the periodic 
performance review will have several benefits. These include providing 
hospitals with a process to assess their ongoing compliance and 
requiring them to correct or plan to correct all deficiencies 
identified. Periodic performance reviews must be conducted either by 
the hospital as a self-assessment or, if the hospital chooses, by JCAHO 
through an on-site review.

However, periodic performance reviews may not necessarily improve the 
detection of deficiencies. JCAHO did not pilot test these reviews for 
the potential to detect deficiencies and did not test whether hospitals 
that conducted reviews do a better job of continuing to comply with 
standards. In addition, for hospitals performing self-assessments, 
JCAHO will not check these self-assessments to determine whether 
hospitals fully and accurately identified quality problems and 
developed adequate corrective action plans to address the problems 
identified.

Priority Focus Process and Patient Tracer Methodology:

According to JCAHO, the priority focus process and patient tracer 
methodology together have the potential to enhance the ability of 
surveys to detect deficiencies by directing the attention of surveyors 
to key patient care areas. The priority focus process uses a data-based 
formula to identify a limited number of areas in each hospital that are 
particularly important to patient health and safety. Priority focus 
areas might include infection control, medication management, or 
patient safety. Surveyors use the priority focus process combined with 
the patient tracer methodology to focus their surveys to specific areas 
for review. The patient tracer methodology guides their choice of 
current patients to "trace" through the experience of care within an 
organization. For example, if the hospital's priority focus process 
data suggest that a patient with an orthopedic-related diagnosis such 
as a hip fracture should be traced, the JCAHO surveyor would review the 
patient's medical record, noting where the patient had entered into the 
hospital and any services and transfers that occurred. Then the 
surveyor would retrace the steps in the patient's care process by 
observing and talking to staff in some of the areas in which the 
patient received care. If the patient entered through the emergency 
department, was transferred to a medical/surgical unit, and then went 
to the operating room, the surveyor would go to these areas to 
interview staff about the care given to this specific patient. With 
information from patient tracers, the surveyor will assess whether any 
compliance issues exist with JCAHO standards. If the surveyor 
identifies a compliance issue while tracing one patient, the surveyor 
may review the records of similar patients to determine whether the 
problem is isolated or represents a pattern of care.

However, JCAHO did not test the extent to which the priority focus 
process and the patient tracer methodology could help surveyors detect 
deficiencies. A JCAHO official told us these new features of the 
accreditation process were intended to help surveyors trace patients in 
a consistent way and not necessarily to improve the detection of 
deficiencies.

Unannounced Surveys:

JCAHO plans to conduct all hospital accreditation surveys on an 
unannounced basis beginning in 2006.[Footnote 18] JCAHO stated that 
unannounced surveys will ensure that hospital performance is based on 
the observation of hospitals' routine operations rather than on how 
they operate after they have the opportunity to prepare to be surveyed. 
A JCAHO official also indicated that unannounced surveys will be more 
likely to detect deficiencies. The OIG and other organizations share 
JCAHO's position on the value of unannounced surveys of hospitals and 
other health care organizations. The value of unannounced surveys also 
has been recognized for nursing homes, which state agencies survey on 
an unannounced basis.

JCAHO's Pilot Test of New Process Was Limited:

JCAHO's pilot test of its new hospital accreditation process was 
limited and therefore unable to help determine the potential of the new 
process to detect deficiencies in Medicare COPs. According to JCAHO, 
the pilot test suggests that the new process was more likely than the 
former process to find quality problems. However, the pilot test sample 
included hospitals that volunteered or were selected by JCAHO and were 
not randomly selected, pilot test surveyors were accompanied by 
observers from JCAHO's central office, and pilot test results were not 
independently evaluated. In addition, CMS has not completed its fiscal 
year 2004 validation program, which will include hospitals surveyed by 
JCAHO using the new process and thus does not yet have sufficient data 
on which to base a meaningful evaluation.

According to JCAHO's analysis of the pilot test, the new hospital 
accreditation process is more likely to identify quality problems since 
proportionately more hospitals under the new process received 
unfavorable accreditation decisions. JCAHO based its conclusion on a 
comparison of survey outcomes, called accreditation decisions, between 
18 hospitals in the pilot test conducted in 2002 and 2003 and the 1,524 
hospitals that had been surveyed under the pre-2004 accreditation 
process during 2003. Table 4 presents the data JCAHO used to make the 
comparison. As shown, proportionately fewer hospitals under the new 
process were accredited without having to make corrections. Although 
JCAHO provided the accreditation decision outcomes for these 18 pilot 
tests, it stated it preferred to use the number of "requirements for 
improvement" as the basis for analysis.

Table 4: Accreditation Decisions for Hospitals Surveyed Under JCAHO's 
New Survey Process Pilot Test as Compared to Results from JCAHO's Pre-
2004 Survey Process:

Accreditation decision: Accreditation; 
Pilot test of new survey process: Number of hospitals surveyed: 0; 
Pilot test of new survey process: Percentage of hospitals surveyed: 0%; 
Pre-2004 survey process: Number of hospitals surveyed: 320; 
Pre-2004 survey process: Percentage of hospitals surveyed: 21%. 

Accreditation decision: Survey findings with requirements for 
improvement[A]; 
Pilot test of new survey process: Number of hospitals surveyed: 13; 
Pilot test of new survey process: Percentage of hospitals surveyed: 
72%; 
Pre-2004 survey process: Number of hospitals surveyed: 1,191; 
Pre-2004 survey process: Percentage of hospitals surveyed: 78%. 

Accreditation decision: Conditional accreditation; 
Pilot test of new survey process: Number of hospitals surveyed: 3; 
Pilot test of new survey process: Percentage of hospitals surveyed: 
17%; 
Pre-2004 survey process: Number of hospitals surveyed: 13; 
Pre-2004 survey process: Percentage of hospitals surveyed: 1%. 

Accreditation decision: Preliminary denial of accreditation; 
Pilot test of new survey process: Number of hospitals surveyed: 2; 
Pilot test of new survey process: Percentage of hospitals surveyed: 11; 
Pre- 2004 survey process: Number of hospitals surveyed: 0; 
Pre-2004 survey process: Percentage of hospitals surveyed: 0%. 

Accreditation decision: Total; 
Pilot test of new survey process: Number of hospitals surveyed: 18; 
Pilot test of new survey process: Percentage of hospitals surveyed: 
100%; 
Pre-2004 survey process: Number of hospitals surveyed: 1,524[B]; 
Pre-2004 survey process: Percentage of hospitals surveyed: 100%. 

Source: JCAHO.

Note: JCAHO reported that it conducted pilot tests of the new 
accreditation process in an additional 12 hospitals in 2001. However, 
JCAHO was unable to provide the accreditation decisions for these 12 
pilot site hospitals.

[A] Hospitals in the pilot test with deficiencies were accredited 
contingent upon evidence of correcting deficiencies. The hospitals in 
the comparison group with deficiencies received accreditation with 
requirements for improvement.

[B] These 1,524 hospitals represent all those surveyed for 
accreditation by JCAHO during 2003.

[End of table]

However, JCAHO's pilot test analysis was limited in three respects, 
which may have accounted for the smaller number of favorable 
accreditation decisions hospitals received under the new process.

* The hospitals participating in the pilot test were not randomly 
selected by JCAHO. As a result, these hospitals may not be 
representative of all JCAHO-accredited hospitals and therefore results 
cannot be generalized.

* During the pilot test, an observer from JCAHO's central office 
accompanied each surveyor, and the knowledge that they were being 
observed may have influenced the surveyors' actions.[Footnote 19] Under 
the pre-2004 process, observers only rarely accompanied JCAHO 
surveyors.

* JCAHO conducted its own evaluation of pilot test results. Evaluation 
of the pilot test by an entity independent of either JCAHO or the 
hospitals tested could help to ensure that survey outcomes were 
impartially interpreted. For example, CMS used an independent group to 
evaluate its redesign of the nursing home survey process.

CMS Oversight Authority of JCAHO's Hospital Accreditation Program Is 
Limited and Needs Improvement:

CMS has limited oversight authority over JCAHO's hospital accreditation 
program, and its existing oversight activities need improvement. The 
unique status of JCAHO's hospital accreditation program, which is 
specified in statute, does not permit CMS to take corrective action, 
such as restricting or removing its deeming authority. Additionally, 
CMS uses a measure that provides limited information to evaluate the 
performance of JCAHO's hospital accreditation program, has 
significantly reduced the number of surveys conducted as part of CMS's 
validation program, and does not use measures that are based on sound 
statistical methods to assess the performance of JCAHO's hospital 
accreditation program.

CMS Oversight Authority of JCAHO Is Limited:

Because of JCAHO's unique legal status, CMS's oversight of JCAHO's 
hospital accreditation program is limited in two major ways: Unlike 
other accreditation programs with deeming authority, JCAHO does not 
have to reapply to CMS to reauthorize its deeming authority, and CMS 
cannot take action to address performance problems with JCAHO's 
hospital accreditation program.

JCAHO's hospital accreditation program is the only Medicare 
accreditation program for which CMS does not have to conduct an 
evaluation of the accreditation standards and the processes used to 
conduct surveys. Without this evaluation, CMS is deprived of key 
oversight tools it is authorized to use with other accreditation 
programs: detailed information about any proposed changes to the 
accreditation process and public input. CMS cannot require JCAHO to 
provide information about proposed changes to its accreditation 
requirements and hospital survey processes. Also, because it is not 
required to reapply to CMS for deeming authority, JCAHO does not have 
to provide CMS information that other accreditation programs must 
provide, such as a detailed description of its survey processes, a 
comparison of its standards to Medicare requirements, and the 
qualifications of its surveyors, which CMS reviews to ensure that the 
programs comply with Medicare requirements. For example, when JCAHO's 
hospice accreditation program applied for deeming status in 1999, CMS 
required changes to JCAHO's hospice accreditation process, including 
requiring JCAHO to make unannounced surveys of Medicare-certified 
hospices. According to a CMS official, JCAHO's hospital accreditation 
program has provided much of the information required of other 
accreditation organizations; however, CMS has no authority to require 
JCAHO to make changes to the hospital accreditation program as it does 
with other health care accreditation programs. Statutory provisions 
regarding public notice and comment do not apply to JCAHO's hospital 
accreditation program as they do to other accreditation programs. The 
reapplication process for other accreditation programs requires 
affording the public an opportunity to provide input to CMS on an 
accreditation program's request for deeming authority. Because JCAHO 
does not have to reapply for deeming authority, the public does not 
have the opportunity to review and comment on JCAHO's hospital 
accreditation program.[Footnote 20]

A second limitation is CMS's inability to address any performance 
issues with JCAHO's hospital accreditation program. Although the rate 
of disparity for JCAHO's hospital accreditation program exceeded 20 
percent in fiscal years 2000, 2001, and 2002 -a rate that would have 
triggered a deeming authority review for any other Medicare 
accreditation program-CMS was unable to take enforcement action to 
address JCAHO's performance. When other Medicare accreditation programs 
have a rate of disparity of 20 percent or more, CMS can take steps such 
as imposing a year-long probationary period and removing deeming 
authority at the end of the probationary period if the rate of 
disparity remains at 20 percent or more. For JCAHO, however, CMS's 
actions toward correcting the program's deficiencies are limited to 
including recommendations for improvement in its annual reports to 
Congress and negotiating with JCAHO to voluntarily adopt CMS's 
recommendations.

In its annual report to Congress, CMS made recommendations in fiscal 
year 2002 aimed at improving JCAHO's ability to detect serious 
deficiencies in the life safety code, part of the COP on physical 
environment. CMS noted that JCAHO permits hospitals to self-assess 
compliance with life safety code requirements.[Footnote 21] While CMS 
stated that it did not object to the concept of hospital self-
assessment of life safety code requirements, it made five 
recommendations to JCAHO for improving implementation:

1. Require hospitals to use qualified personnel, such as fire marshals 
and architects, to conduct self-assessments of compliance with the life 
safety code requirements.

2. Set minimum standards for identifying and improving life safety code 
deficiencies identified by hospital self-assessments.

3. Require hospitals to submit their self-assessments on life safety 
code issues prior to JCAHO conducting accreditation surveys to provide 
surveyors and personnel in JCAHO's central office time to review the 
material prior to the accreditation surveys.

4. Increase the use of JCAHO experts in the life safety code 
requirements in its central office.

5. Address the issue of hospitals that do not make improvement within 
self-determined time frames.

JCAHO did not adopt all of these recommendations. It disagreed with the 
first recommendation. Its response indicated that its requirement to 
use qualified personnel to complete the self-assessment, while more 
general, was sufficient. It further indicated that policies were in 
place for CMS's second and fifth recommendations. CMS later agreed that 
JCAHO's policies do satisfactorily address the fifth recommendation. 
JCAHO planned to examine ways to adopt CMS's third and fourth 
recommendations. CMS however, had no authority to compel JCAHO to 
comply with the remaining recommendations. According to CMS, it 
continues to discuss implementation of its recommendations with JCAHO. 
JCAHO stated that while its initial response to CMS's recommendations 
in 2003 reflected then current JCAHO policies, subsequent policy 
evolutions are addressing CMS's recommendations. Specifically, JCAHO is 
working with the American Society of Hospital Engineers to develop a 
process for review by experts of hospital self-assessments on life 
safety code issues prior to JCAHO's conducting on-site accreditation 
surveys and to identify those hospitals for which engineering expertise 
should be added to on-site surveys.

CMS's Validation Program Needs Improvement:

CMS states that the goal of its validation program is to provide 
reasonable assurance to Congress that the JCAHO accreditation process 
ensures hospital compliance with Medicare COPs. However, the measure 
CMS uses to evaluate the performance of JCAHO's hospital accreditation 
program provides limited information and could mask problems with an 
accreditation program's performance in detecting serious deficiencies, 
and it is based on a target sample size of 1 percent of JCAHO-
accredited hospitals. In addition, CMS does not report the extent to 
which its sample reflects the performance of the larger population of 
JCAHO-accredited hospitals.

Rate of Disparity:

The rate of disparity between JCAHO's hospital accreditation survey 
findings and state survey agency findings, as currently calculated by 
CMS, does not fully explain the performance of JCAHO's hospital 
accreditation program in detecting serious deficiencies. CMS uses this 
measure in its reports to Congress to assess JCAHO's hospital 
accreditation program and as the basis for making recommendations for 
improvement. CMS calculates the rate of disparity as the difference 
between the number of hospitals found with serious deficiencies by 
state survey agencies and the number of hospitals found with serious 
deficiencies by the accreditation survey, divided by the number of 
hospitals in the sample. For example, if state survey agencies 
conducted 200 surveys as part of CMS's validation program and found 60 
hospitals out of compliance with at least one COP, but JCAHO's survey 
found that only 22 of the hospitals were out of compliance, the rate of 
disparity would be 19 percent ((60 - 22)/200).

CMS has established in regulation a rate of disparity of 20 percent or 
greater as the threshold for taking action against an accreditation 
program. According to a CMS official, the use of 20 percent as the 
threshold is not based on empirical evidence but rather on what CMS 
believed Congress would find acceptable. Consequently, the threshold 
may not be appropriately placed to indicate unacceptable performance by 
a hospital accreditation program. For example, if JCAHO failed to 
identify serious deficiencies in all 14 hospitals that the state 
agencies identified with serious deficiencies from a sample of 79 
hospitals, the rate of disparity would be a satisfactory 18 percent 
((14-0)/79).[Footnote 22]

CMS's rate of disparity measure used in isolation does not consistently 
reflect an accreditation program's ability to detect serious 
deficiencies. As the number of hospitals with serious deficiencies 
detected by state survey agencies decreases, regardless of JCAHO's 
performance in detecting them, it is more likely that the rate of 
disparity will be less than CMS's 20 percent threshold. As a result, 
the performance of JCAHO's hospital accreditation program is difficult 
to judge based on this measure alone. For example, if state survey 
agencies performed 200 validation surveys and found 100 hospitals or 50 
percent with serious deficiencies and JCAHO found 30 hospitals or 30 
percent of the hospitals found by state agencies, the rate of disparity 
would be 35 percent ((100-30)/200). However, if the state agencies 
found 50 hospitals, or 25 percent, of the 200 hospitals with serious 
deficiencies and JCAHO found 15 hospitals, or 30 percent of the 
hospitals that the state agencies identified, the rate of disparity 
would be almost 18 percent ((50-15)/200). The percentage of serious 
deficiencies found by state survey agencies and also by JCAHO remained 
the same in both examples, but the rate of disparity was improved 
significantly by the larger number of hospitals without serious 
deficiencies in the second example. This indicates that the rate of 
disparity does not consistently measure the accreditation program's 
ability to detect serious deficiencies found by state survey agencies. 
(See table 5.) In addition to the rate of disparity, other components, 
such as the proportion of hospitals with serious deficiencies and the 
total number of serious deficiencies found by state agencies but missed 
by the accreditation program, are important indicators of an 
accreditation program's overall performance.

Table 5: Hypothetical Examples of the Effect on the Rate of Disparity 
of a Decrease in the Number of Hospitals with Serious Deficiencies in a 
Sample of 200 Hospitals:

Number of hospitals with serious deficiencies; 
Example 1: State agencies: 100; 
Example 1: JCAHO: 30; 
Example 2: State agencies: 50; 
Example 2: JCAHO: 15.

Percentage of hospitals state agencies found with serious deficiencies 
that were also found by JCAHO; 
Example 1: JCAHO: 30%; 
Example 2: JCAHO: 30%.

Percentage of hospitals without serious deficiencies identified by 
state agencies; 
Example 1: JCAHO: 50%; 
Example 2: JCAHO: 75%.

Rate of disparity; 
Example 1: State agencies: 35% ((100-30)/200); 
Example 2: State agencies: 18% ((50-15)/200).

Performance level; 
Example 1: State agencies: Above threshold; 
Example 2: State agencies: Below threshold. 

Source: GAO.

Note: CMS's rate of disparity threshold is 20 percent.

[End of table]

Statistical Analysis of Validation Survey Sample:

CMS does not analyze the statistical results of its validation survey 
samples in ways that would allow it to better assess JCAHO's ability to 
detect serious deficiencies. CMS has not documented the methods it uses 
to select hospitals for validation surveys and did not supply us with 
clear technical justification for the methods used. Further, CMS's 
validation sample includes hospitals that, because of its sampling 
method, have varying chances of selection, but it does not take this 
into account when calculating statistics based on the sample. According 
to CMS's sampling method, the selection of hospitals is influenced by 
factors such as the month in the fiscal year that JCAHO performed the 
accreditation survey and how many hospitals were targeted for 
completion that year in the state in which the hospital was located. 
Thus, some hospitals have a greater chance of selection than others. 
CMS also does not take these different chances of selection into 
account when calculating statistics for its annual reports to Congress, 
which prevents CMS from accurately assessing JCAHO's performance. 
Moreover, CMS does not measure and report in its annual reports the 
extent to which its estimates based on the validation survey sample are 
likely to reflect how well JCAHO detects deficiencies in the larger 
population of hospitals it accredits.[Footnote 23]

In addition, the number of usable traditional validation surveys 
completed is smaller than the number of hospitals CMS samples for 
validation surveys. This difference may affect the accuracy of the data 
that CMS presents to Congress if the hospitals where the traditional 
surveys were completed produce different results than those where 
surveys are not completed or are not usable. During its sampling 
process, CMS selects a sample size close to the targeted number of 
hospitals each year. Some hospitals from this sample may be excluded 
because CMS chose to perform another type of survey for them that 
cannot be used to validate a JCAHO accreditation survey. In addition, 
state agencies are not always able to complete the requested 
traditional validation surveys within 60 days from the JCAHO 
accreditation survey, as required, or a hospital may be excluded 
because it lost its deemed status or closed. The size of the difference 
between the number of hospitals sampled and the number of usable 
traditional validation surveys completed therefore varies, as it did 
during the 3-year review period (see table 6).

Table 6: Number of Hospitals Targeted for Validation Surveys Compared 
with Usable Traditional Validation Surveys Completed:

Fiscal Year: 2000; 
Hospitals targeted for validation surveys[A]: 236; 
Hospitals sampled for validation surveys[B]: 236; 
Usable traditional validation surveys completed[C]: 184.

Fiscal Year: 2001; 
Hospitals targeted for validation surveys[A]: 227; 
Hospitals sampled for validation surveys[B]: 217; 
Usable traditional validation surveys completed[C]: 204.

Fiscal Year: 2002; 
Hospitals targeted for validation surveys[A]: 227; 
Hospitals sampled for validation surveys[B]: 235; 
Usable traditional validation surveys completed[C]: 112. 

Source: CMS.

[A] The targeted number is set at the beginning of the fiscal year and 
is used for planning and resource allocation by CMS and the state 
survey agencies.

[B] The sampled hospitals are the hospitals selected for validation 
surveys during the year.

[C] Usable surveys exclude those not completed, those completed after 
the required 60-day time frame, and other types of surveys that can not 
be used to validate a JCAHO accreditation survey.

[End of table]

Annual Number of Validation Surveys:

CMS reduced the number of validation surveys conducted by state 
agencies from a target of approximately 5 percent of the total number 
of hospitals that JCAHO accredits to a target of approximately 1 
percent, with at least one survey in each state. Reducing the target of 
validation surveys from 5 percent to 1 percent results in the number of 
validation surveys being reduced from 227 in fiscal year 2002 to a 
target of 75 validation surveys in fiscal year 2003 and 72 in fiscal 
year 2004.

Reducing the targeted number of validation surveys to 1 percent 
provides less reliable information on how well JCAHO's hospital 
accreditation program ensures compliance with Medicare COPs. For 
example, for a 5-percent target, the estimate of the proportion of 
JCAHO-accredited hospitals with a particular deficiency that is derived 
from the validation survey could be as much as 6.0 percentage points 
higher or lower, for a range of 12.0 percentage points. If the 5-
percent target produced an estimate that 50 percent of JCAHO-accredited 
hospitals had a particular deficiency, the percentage of JCAHO-
accredited hospitals not complying could range from 44.0 to 56.0 
percent. However, for a 1-percent target the estimate could be 11.4 
percentage points higher or lower, for a range of about 22.8 percentage 
points. For example, if the 1-percent target produced an estimate that 
50 percent of JCAHO-accredited hospitals had a particular deficiency, 
the percentage of JCAHO-accredited hospitals not complying with a 
Medicare COP could range from 38.6 to 61.4 percent.[Footnote 24]

This reduction in the number of validation surveys is of additional 
concern because it coincides with the implementation of JCAHO's new 
accreditation process, which has an unproven capacity to detect 
deficiencies. CMS's target sample size for traditional validation 
surveys for fiscal year 2004 will be further reduced because the sample 
also includes 18-month validation surveys. In 2004, CMS is planning to 
conduct 17 of these 18-month surveys as part of its overall validation 
survey target of 72. Thus, CMS could be using as few as 55 validation 
surveys to determine JCAHO's performance.

Conclusions:

For 3 consecutive years, JCAHO's hospital accreditation program, which 
accredits most of the hospitals participating in Medicare, exceeded 
CMS's threshold for unacceptable performance. CMS validation surveys 
during that time period confirmed that JCAHO missed the majority of 
serious deficiencies found by state survey agencies. Yet, CMS was 
unable to take action against JCAHO's hospital accreditation program as 
it can with other accreditation programs because it lacked the 
authority to do so. Although CMS has recommended in its annual reports 
to Congress that JCAHO make changes in its hospital accreditation 
program to improve its ability to detect serious deficiencies, some of 
these recommendations have not been implemented. Thus, it is vital for 
patient safety that JCAHO hospital accreditation surveys detect 
existing serious deficiencies and deny accreditation to hospitals that 
do not comply with Medicare COPs.

CMS is unable to present to Congress an adequate assessment of JCAHO's 
performance because of limitations in its process for selecting 
hospitals for validation surveys and analysis of the survey results. 
CMS does not consistently portray the extent to which serious 
deficiencies are missed and does not identify the limitations in 
reporting the estimates it makes from its survey sample. CMS cannot 
assure Congress that JCAHO-accredited hospitals meet Medicare COPs 
because the measure for the rate of disparity, which determines poor 
performance, allows JCAHO to miss the majority of serious deficiencies 
and still be in an acceptable range of performance. Further, CMS's 
reduction in the number of validation surveys it uses to determine the 
performance of JCAHO's hospital accreditation program will provide less 
reliable information at a time when JCAHO is implementing a new 
hospital accreditation process that is unproven in its ability to 
detect serious deficiencies. In light of these limitations in CMS's 
validation of JCAHO's hospital accreditation program, we believe that 
CMS must improve its oversight so it can provide Congress with more 
accurate information regarding JCAHO's performance.

Matter for Congressional Consideration:

Given the serious limitations in JCAHO's hospital accreditation program 
and that efforts to improve this program through informal action by CMS 
have not led to necessary improvements, Congress should consider giving 
CMS the same kind of authority over JCAHO's hospital accreditation 
program that it has over all other Medicare accreditation programs.

Recommendations for Executive Action:

To strengthen the ability of CMS to identify and report to Congress on 
JCAHO's ability to ensure that the hospitals it accredits protect the 
safety and health of patients through compliance with the Medicare 
COPs, we recommend that the Administrator of CMS take the following 
three actions:

* modify the method used to measure the rate of disparity between 
validation survey findings and accreditation program findings to 
provide a reasonable assurance that Medicare COPs are being met and 
consider whether additional measures are needed to accurately reflect 
an accreditation program's ability to detect deficiencies in Medicare 
COPs;

* provide in the annual report to Congress an estimate, based on the 
validation survey sample, of the performance of all JCAHO-accredited 
hospitals, including the limitations and protocols for these estimates 
based on generally accepted sampling and statistical methodologies; and 
develop a written protocol for these calculations; and:

* annually conduct traditional validation surveys on a sample of JCAHO-
accredited hospitals that is equal to at least 5 percent of all JCAHO-
accredited hospitals.

Agency and Other External Comments and Our Evaluation:

CMS and JCAHO commented on a draft of this report. In its comments, CMS 
concurred with our recommendations. JCAHO stated it had no objection to 
our suggestion that Congress give CMS the same authority over its 
hospital accreditation program as it does over other Medicare 
accreditation programs. However, JCAHO took issue with the methodology 
we used for evaluating the performance of its hospital accreditation 
program. CMS's and JCAHO's specific comments and our response follow. 
CMS's comments are reprinted in appendix IV and JCAHO's comments are 
reprinted in appendix V. CMS and JCAHO also provided technical 
comments, which we incorporated as appropriate.

CMS stated that it has begun to examine the need for additional or 
alternative measures for the rate of disparity calculation. CMS also 
stated it will seek additional resources to further develop and 
implement new sampling and statistical methodologies that may allow 
results to be projected to all JCAHO-accredited hospitals, and to 
increase the validation sample size. CMS specifically noted that it 
considers life-safety code compliance, on the part of all provider 
types, to be critically important. In the past 8 years, in its annual 
reports to Congress and its dialogues with JCAHO regarding its hospital 
accreditation program, it has identified physical environment as an 
important area where JCAHO needs to focus attention, and CMS noted that 
68 percent of facilities that had a deficiency finding not identified 
by JCAHO had them in the physical environment area.

JCAHO stated that our methodology for evaluating the performance of its 
hospital accreditation program was incomplete and did not provide a 
comprehensive assessment of its program's performance. We did not 
intend to do a comprehensive evaluation of JCAHO's overall hospital 
accreditation program. Rather, we focused our evaluation on how well 
JCAHO's hospital accreditation program ensures hospitals' compliance 
with Medicare participation requirements. There are four possible 
outcomes to a comparison between JCAHO's accreditation survey and a 
state validation survey: (1) both JCAHO and state agencies identify no 
deficiencies, (2) JCAHO identifies deficiencies not found by state 
agencies, (3) both JCAHO and state agencies identify the same 
deficiencies, and (4) state agencies identify deficiencies that JCAHO 
does not. We limited our evaluation to the fourth outcome because it 
illustrates the need for CMS oversight of the hospital accreditation 
process. We have clarified the scope of our evaluation to emphasize our 
focus on this outcome.

JCAHO raised a concern that our characterization of JCAHO's missed 
deficiencies that state survey agencies found misleads readers to 
believe that JCAHO misses hospitals with deficiencies 78 percent of the 
time. We have revised language in the report to further emphasize that 
the missed deficiency rate applies to hospitals in the validation 
survey sample in which the state survey agencies found deficiencies and 
cannot be generalized to all JCAHO-accredited hospitals. JCAHO further 
stated that our report does not take into account that JCAHO's hospital 
accreditation program detects deficiencies in hospitals that CMS does 
not find. However, it is to be expected that state survey agencies will 
not find all deficiencies found by JCAHO because hospitals may have 
corrected the deficiencies prior to the state agency surveys.

JCAHO stated that we misrepresented the potential of the new 
accreditation process in detecting deficiencies in Medicare COPs and 
provided new data regarding its first quarter 2004 performance that 
indicate that JCAHO surveys may have detected a greater percentage of 
deficiencies related to patient care compared with the pre-2004 
accreditation process. However, we maintain that until CMS validation 
surveys for 2004 are completed, there is no basis on which to determine 
whether the new process improves the detection of deficiencies in 
Medicare COPs. In addition, JCAHO stated and we agree that evaluating 
and improving the quality of care in hospitals is not about counting 
deficiencies, it is about finding those deficiencies that, if not 
fixed, will generate poor results for patients and making sure that 
these deficiencies are remedied in a timely fashion.

JCAHO stated that we mischaracterized its response to the five 
recommendations that CMS made in 2002 to improve JCAHO's ability to 
detect deficiencies in the life safety code and that it is involved in 
frequent and ongoing dialogue with CMS regarding the recommendations 
and other life safety code issues. We have clarified language in the 
report regarding JCAHO's response to CMS's recommendations.

As agreed with your offices, unless you publicly announce its contents 
earlier, we plan no further distribution of this report until 30 days 
after its date. We will then send copies of this report to the 
Secretary of Health and Human Services and other interested parties. We 
will also make copies available to others upon request. In addition, 
the report will be available at no charge at the GAO Web site at http:/
/www.gao.gov.

If you or your staffs have any questions about this report, please call 
me at (202) 512-7119. Another contact and key contributors are listed 
in appendix VI.

Signed by: 

Janet Heinrich: 
Director, Health Care-Public Health Issues:

[End of section]

Appendix I: Scope and Methodology:

We examined the extent to which JCAHO's pre-2004 survey process 
identified hospitals with deficiencies and individual deficiencies in 
Medicare COPs that were identified by state survey agencies. We chose 
these measures because they reflect performance in detecting and 
correcting serious deficiencies, which according to CMS, substantially 
limit a hospital's capability to render adequate care and adversely 
affect the health and safety of patients. We reviewed data, provided by 
CMS, on 500 traditional validation surveys conducted by state survey 
agencies during fiscal years 2000 through 2002. In these validation 
surveys, state survey agencies documented whether they found serious 
deficiencies in Medicare COPs. CMS compared state survey agency 
findings with JCAHO's accreditation surveys that identified 
deficiencies in JCAHO's standards. CMS then determined whether the 
state survey agencies' findings on serious deficiencies in the 22 
Medicare COPs that can be deemed were comparable to JCAHO's findings on 
deficiencies in JCAHO's standards in the following way. Two CMS experts 
such as nurses reviewed the comparability of serious deficiencies in 
the quality-of-care conditions identified in validation surveys to 
deficiencies in JCAHO's accreditation standards identified in JCAHO's 
hospital accreditation surveys. Two experts, such as building 
engineers, reviewed the comparability of serious deficiencies 
identified in the validation surveys on the condition on physical 
environment. Where there was disagreement, the two experts met to 
resolve their differences. CMS does not have written protocols for 
determining comparability. Experts are expected to use their best 
professional judgment. CMS experts also had to consider whether it is 
reasonable to conclude that the deficiencies existed at the time that 
JCAHO surveyed the hospital. For those deficiencies that CMS determines 
that JCAHO has failed to identify, it met with JCAHO to address 
disputed findings and to consider additional evidence on comparability 
offered by JCAHO. There are four possible outcomes to this comparison 
of survey findings--(1) JCAHO and state agencies both identify no 
deficiencies, (2) JCAHO identifies deficiencies not found by state 
agencies, (3) JCAHO and state agencies both identify the same 
deficiencies, and (4) state agencies identify deficiencies that JCAHO 
does not--we focused on the fourth because it highlights the need for 
CMS oversight of the hospital accreditation program. For the second 
outcome, there could be two reasons for the disparity between JCAHO's 
and state survey agencies' findings: hospitals corrected deficiencies 
identified by JCAHO prior to the state agency survey or the state 
survey agency did not identify a deficiency that existed. In addition, 
not all JCAHO findings are equivalent to noncompliance with a Medicare 
COP.

From these 500 surveys, we determined the number of hospitals with 
serious deficiencies and the total number of serious deficiencies 
identified by state agencies but that CMS determined were not 
identified by JCAHO. These data include 123 hospitals in which state 
survey agencies identified one or more serious deficiencies and JCAHO 
did not make comparable findings according to CMS. These data also 
include 167 serious deficiencies identified by state agencies but that 
CMS determined comparable findings were not identified by JCAHO.

For fiscal years 2001 and 2002, we obtained from CMS a comparison 
between the validation surveys conducted by the state survey agencies 
and the accreditation surveys conducted by JCAHO, which identified 
serious deficiencies identified by the state agencies but not by JCAHO 
as determined by CMS. For fiscal year 2000, CMS did not supply its 
determinations of the comparability of findings in validation and 
accreditation surveys for 31 of 82 serious deficiencies. We followed a 
protocol similar to the one used by CMS to determine the comparability 
of the remaining 31 serious deficiencies, which included 29 quality-of-
care serious deficiencies and 2 physical environment serious 
deficiencies. Two analysts with nursing backgrounds compared the 
findings and made determinations on their comparability based on their 
professional judgment. In cases of disagreement, a third analyst with a 
background in nursing made the determination.

We did not include 1998 and 1999 data in our analysis because CMS used 
a method that undercounted the number of deficiencies identified by 
state survey agencies but not identified by JCAHO. CMS did not count as 
deficient those cases in which state survey agencies determined that a 
hospital was not meeting the COP on physical environment but JCAHO 
determined that the hospital was in compliance because the hospital was 
following correction plans approved by JCAHO.

To determine the potential of JCAHO's new accreditation process in 
improving the detection of deficiencies in Medicare COPs, we reviewed 
material supplied by JCAHO on development and testing of its new 
process and interviewed JCAHO officials about the steps taken to test 
the new process and to analyze results. We also examined the features 
of the new accreditation process by reviewing descriptive material 
obtained from JCAHO and interviewing experts in health care quality. 
Because the new accreditation process was implemented in January 2004, 
we were limited in our ability to determine the effectiveness of the 
new accreditation process because we were not able to perform a 
comparative analysis of validation survey and JCAHO survey results 
under the new process.

To examine the effectiveness of CMS's oversight of JCAHO's 
accreditation process, we analyzed the laws and regulations that define 
CMS's authority and JCAHO's authority. We reviewed the annual reports 
submitted to Congress on JCAHO's performance in identifying serious 
deficiencies and reviewed correspondence between CMS and JCAHO and 
interviewed officials in both organizations. We analyzed the rate of 
disparity that CMS uses to determine the performance of JCAHO's 
hospital accreditation process in identifying deficiencies in Medicare 
COPs.

To evaluate CMS's statistical methodology for the validation surveys, 
we interviewed CMS officials about the sampling and statistical 
methods. In the absence of written methodological documentation, we 
relied on information provided by CMS officials to evaluate the 
methodology. They gave us the following information about their 
sampling method. At the beginning of each year, CMS determines a target 
for the number of hospitals that will be sampled for validation surveys 
in each state. Each month, CMS receives a list of hospitals scheduled 
for a JCAHO accreditation survey in that month. Prior to sampling, CMS 
removes from the list those hospitals that have received a validation 
survey in the last 3-year accreditation cycle and hospitals that do not 
participate in Medicare. In the first month of the year, CMS selects a 
random sample of hospitals to be surveyed from JCAHO's list. In 
subsequent months, CMS removes hospitals in states in which the state 
target has been met and then selects a random sample of hospitals. 
Prior to sending the list to state survey agencies, CMS determines 
which hospitals will receive traditional validation surveys and which 
will receive other types of surveys that cannot be used to assess the 
performance of JCAHO's hospital accreditation program. State survey 
agencies must then complete traditional validation surveys within 60 
days of the completion of JCAHO's accreditation survey for the results 
to be used by CMS to measure the performance of JCAHO's hospital 
accreditation program. According to CMS officials, the sampling 
procedures CMS uses are necessary because they are not informed more 
than 1 month in advance which hospitals JCAHO will survey for 
accreditation.

In reviewing the sampling procedures they described, we determined that 
CMS initially selects a probability sample of hospitals for its state 
agency validation surveys.[Footnote 25] However, hospitals have varying 
chances of selection in the sample depending on the month in the fiscal 
year that JCAHO performs the accreditation survey and the number of 
hospitals targeted for completion that year in the state in which the 
hospital was located. Additionally, the way that CMS determines which 
type of survey the sampled hospital receives is not random. Therefore, 
the analysis we performed is limited to those hospitals included in the 
validation survey sample and cannot be projected to all JCAHO-
accredited hospitals.

[End of section]

Appendix II: Medicare Conditions of Participation:

To participate in Medicare, hospitals must maintain standards of 
patient safety and health that comply with Medicare requirements. There 
are currently 23 Medicare COPs. Table 7 provides a description of each 
Medicare COP.

Table 7: Medicare Conditions of Participation:

Medicare COP: Anesthesia services[A]; 
Description: Anesthesia services must be well organized and directed by 
a qualified doctor of medicine or osteopathy. The service is 
responsible for all anesthesia administered.

Medicare COP: Compliance with federal, state, and local laws; 
Description: A hospital must comply with applicable federal laws on 
patient health and safety and state and local laws on hospital and 
personnel licensing.

Medicare COP: Discharge planning; 
Description: A hospital must have a discharge planning process 
applicable to all patients. Policies and procedures must be in writing.

Medicare COP: Emergency services[A]; 
Description: If emergency services are provided they must be organized 
under the direction of a qualified member of the medical staff and have 
adequate medical and nursing personnel qualified in emergency care to 
meet the needs anticipated by the facility.

Medicare COP: Food and dietetic services; 
Description: Dietary services must be organized, directed, and staffed 
by qualified personnel. Contracted services must meet certain 
requirements.

Medicare COP: Governing body; 
Description: A hospital must have a legally responsible governing body 
or persons charged with the responsibilities of a governing body.

Medicare COP: Infection control; 
Description: A hospital's sanitary environment must avoid sources and 
transmission of infections and communicable diseases. It must have an 
active program to prevent, control, and investigate infections and 
communicable diseases.

Medicare COP: Laboratory services; 
Description: The hospital must maintain, or have available, adequate 
laboratory services.

Medicare COP: Medical record services; 
Description: A hospital must have a medical record service that has 
administrative responsibility for medical records.

Medicare COP: Medical staff; 
Description: A hospital must have an organized medical staff that 
abides by bylaws approved by the governing body and is responsible for 
the quality of patient medical care.

Medicare COP: Nuclear medicine services[A]; 
Description: Nuclear medicine services must meet the needs of the 
patients in accordance with acceptable standards of practice.

Medicare COP: Nursing services; 
Description: An organized nursing service must provide 24-hour nursing 
services that are supervised or furnished by registered nurses.

Medicare COP: Organ, tissue, and eye procurement; 
Description: The hospital must have and implement written protocols on 
procurement and have adequate organ transplant policies.

Medicare COP: Outpatient services[A]; 
Description: Outpatient services must meet patient needs consistent 
with acceptable standards of practice.

Medicare COP: Patients' rights; 
Description: A hospital must protect and promote patients' rights.

Medicare COP: Pharmaceutical services; 
Description: The hospital must have pharmaceutical services that meet 
patient needs.

Medicare COP: Physical environment; 
Description: Hospital construction, arrangements, and maintenance must 
ensure patient safety and provide diagnostic and treatment facilities 
and special hospital services appropriate to community needs.

Medicare COP: Quality assessment and performance improvement; 
Description: A hospital must have an effective, hospitalwide quality 
assurance program.

Medicare COP: Radiologic services; 
Description: The hospital must maintain, or have available, diagnostic 
radiologic services. Therapeutic services provided must meet 
professionally approved standards for safety and personnel 
qualifications.

Medicare COP: Rehabilitation services[A]; 
Description: Rehabilitation, physical therapy, occupational therapy, 
audiology, or speech pathology services must be organized and staffed 
to ensure the health and safety of patients.

Medicare COP: Respiratory services[A]; 
Description: Respiratory services must meet patient needs in accordance 
with acceptable standards of practice.

Medicare COP: Surgical services[A]; 
Description: Surgical services must be well organized and provided in 
accordance with acceptable standards of practice. Outpatient services 
must be consistent with inpatient care quality in accordance with the 
complexity of services offered.

Medicare COP: Utilization review; 
Description: Utilization review plans must provide for review of the 
services that a hospital and its medical staff provide to Medicare and 
Medicaid patients.

Source: GAO summary of Medicare COPs.

[A] Optional services not required by Medicare.

[End of table]

[End of section]

Appendix III: Features of JCAHO's New Accreditation Process:

In January 2004, JCAHO introduced a new hospital accreditation process 
that includes several new features. Table 8 includes a description of 
selected new features of JCAHO's hospital accreditation process.

Table 8: JCAHO's Description of Features of Its New Hospital 
Accreditation Process:

Feature of the new accreditation process: Periodic performance review; 
Description: The periodic performance review (PPR) is a new form of 
evaluation that is conducted by the organization and focuses on 
patient safety and quality of care issues. The organization self-
evaluates its compliance with all standards that are applicable to the 
services that the organization provides, and develops a plan of action 
for all areas of performance identified as needing improvement. JCAHO 
will work with the organization to refine its plan of action to assure 
that its corrective efforts are on target. The organization will also 
identify measures of success for validating resolution of the 
identified problem areas when the organization undergoes its complete 
on-site survey 18 months later; 
Three options to the full PPR are available to organizations. The 
options and their requirements are: 
Option 1; 
The organization performs the mid-cycle self-assessment, develops the 
plan of action and measures of success but does not submit PPR data to 
JCAHO. The organization attests that it has completed the foregoing 
activities but has, for substantive reasons, been advised not to submit 
its self-assessment or plan of action to JCAHO; 
The organization may discuss standards-related issues with JCAHO staff 
without identifying its specific levels of standards compliance; 
At the time of the complete on-site survey, the organization provides 
its measures of success to JCAHO for assessment; 
Option 2; 
The organization need not conduct a mid-cycle self-assessment or 
develop a plan of action; 
The organization undergoes an on-site survey at the mid-point of its 
accreditation cycle. The survey will be approximately one-third the 
length of a typical full on-site survey and the organization will be 
charged a fee to cover survey costs; 
The organization develops and submits to JCAHO a plan of action to 
address any areas of non- compliance found during the on-site survey. 
JCAHO will work with the organization to refine its plan of action. At 
the time of the complete on-site survey, the organization provides its 
measures of success to JCAHO for assessment; 
Option 3; 
The mid-cycle survey would be performed, as in Option 2, but, if the 
organization chooses, no written documentation or report of the survey 
would be left with the organization. Findings would be conveyed orally. 
This would eliminate the availability of a survey report for possible 
discovery from the organization, and would permit the organization, as 
is the case with Option 1, to control the language and documentation of 
the mid-cycle assessment activity. At the subsequent full survey, 
surveyors would not discuss with the organization, unless asked to do 
so, the fact that any particular standard had been found out of 
compliance at the mid-cycle assessment. Rather, they would focus on 
compliance with those standards at the time of the full survey; 
If the plan of action is approved, the organization's accreditation 
decision will remain the same. However, if the plan of action is not 
approved, the organization's accreditation decision will be changed to 
reflect the appropriate status. At the triennial on-site survey, 
implementation of the plan of action will be validated.

Feature of the new accreditation process: Priority focus process; 
Description: The priority focus process (PFP) is a data-driven tool 
that focuses survey activity on issues most relevant to patient safety 
and quality of care at the specific health care organization being 
surveyed. The PFP uses automation to gather pre-survey data from 
multiple sources including JCAHO, the hospital and other public 
sources. The PFP then applies rules to 1) identify relevant standards 
and appropriate survey activities, and 2) guide the selection of 
patient tracers. As part of the priority focus process, surveyors will 
track patients through their experience of care within an 
organization, assessing the quality and safety of care provided. The 
PFP does not imply that priority areas are out of compliance or 
deficient in any way. Rather, it lends consistency to the surveyor's 
on-site sampling process. The PFP also helps to focus the surveyor's 
assessment on quality and safety issues specific to an individual 
health care organization; 
The output of the PFP process will include: the top four to five 
priority focus areas-the processes, systems, or structures within a 
health care organization known to significantly impact the safety and 
quality of care specific to the health care organization being 
surveyed.

Feature of the new accreditation process: Tracer methodology; 
Description: An evaluation method in which surveyors select a patient 
and use that individual's record as a roadmap or "tracer" to assess 
and evaluate an organization's compliance with selected standards and 
the organization's systems of providing care and services. Using 
tracers, JCAHO surveyors will look at the care provided by each 
department within an organization, and how departments work together. 
Surveyors retrace the specific care processes that the individual 
experienced by observing and talking to staff in areas that the 
individual received care. As the individual's case is examined, the 
surveyor may identify performance issues in one or more steps of the 
process-or the interfaces between steps-that affect the care of the 
patient. Surveyors will look for commonalities that might point to 
potential system-level issues in the organization. The tracer activity 
also provides several opportunities for surveyors to provide education 
to organization staff and leaders, as well as to share best practices 
from other similar health care organizations; 
Tracer patients will primarily be selected from an active patient list. 
Typically, individuals selected for the tracer activity are those who 
have received multiple or complex services. 

Source: JCAHO.

[End of table]

[End of section]

Appendix IV: Comments from the Centers for Medicare & Medicaid 
Services:

DEPARTMENT OF HEALTH & HUMAN SERVICES: 
Centers for Medicare & Medicaid Services:
Administrator: 
Washington, DC 20201:

DATE: JUL 14 2004:

TO: Janet Heinrich:
Director, Health Care-Public Health Issues: 
General Accounting Office:

FROM: Mark B. McClellan, M.D., Ph.D.:
Administrator:

SUBJECT: General Accounting Office Draft Report: MEDICARE: CMS Needs 
Additional Authority and Better Measures to Adequately Oversee the 
Hospital Accreditation Program (GAO-04-850)

Thank you for the opportunity to comment on the above-referenced draft 
report from the General Accounting Office (GAO). The Centers for 
Medicare & Medicaid Services (CMS), state survey agencies, and the 
Joint Commission on the Accreditation of Healthcare Organizations 
(JCAHO) conduct frequent reviews of hospital performance. In the three 
years under consideration in the GAO study (2000-2002), there were 
17,780 surveys in the approximately 4,500 accredited hospitals in the 
United States, including the following:

* 4,616 JCAHO full triennial surveys;

* 1,290 JCAHO complaint or "for cause" special investigations:

* 500 CMS-sponsored validation surveys to assess the adequacy of JCAHO 
surveys: 7,542 complaint investigations by CMS or state agencies; and

* 3,832 revisits or other investigations by CMS or states to follow-up 
on prior complaint investigations, assess whether hospitals took action 
to fix significant problems identified on earlier surveys, or similar 
purposes.

These 17,780 reviews reflect a considerable national investment in 
external quality assurance for hospitals. They are complemented by the 
external assistance provided by Quality Improvement Organizations 
(QIOs) and, of course, the internal quality efforts of hospitals 
themselves.

Each year CMS conducts or arranges with states for a full survey of a 
sample ofaccredited hospitals that have undergone a JCAHO accreditation 
survey. We call these "CMS validation surveys" because their purpose is 
to assess the extent to which any disparity exists between the JCAHO 
findings and those of CMS. The GAO study focuses on the disparity 
information made possible by the 500 CMS validation surveys.

In 343 out of the 500 hospitals in which full CMS validation surveys 
were conducted during the three-year time period of the GAO study, no 
serious deficiencies were identified in the hospital. In the 157 
hospitals in which a serious deficiency was found through the CMS 
validation surveys, the number of deficiencies averaged about 1.6 per 
hospital. "Physical environment" issues (principally fire-safety) 
represented the single most frequent issue, one we discuss in more 
detail later in this letter. The GAO report observes that serious 
deficiencies identified in hospitals represent only 2 percent of all 
the Medicare requirements surveyed during the three-year period under 
study.

While we regard all deficiencies as serious matters, the overall low 
rate of identified deficiencies relative to the total number of 
hospitals is an encouraging sign that suggests that the overall 
accreditation process has merit.

However, additional improvements can and should. be made, no matter how 
much progress has already been accomplished or how low the rate of 
deficiency might be. Our own on-going, two-year study indicates that 
there may be further actions we can take under existing law to 
strengthen both CMS oversight and JCAHO's efforts. Examples include the 
following:

CMS Validation Surveys: Within the President's 2005 budget request we 
are working to increase the number of CMS validation surveys beyond the 
current level.

More Sensitive Performance Indicators: We developed the conceptual 
framework for more sensitive indicators to improve our ability to 
discern the nature and extent of any JCAHO performance issues. We will 
soon initiate testing of specific options for such indicators.

Complaint Data: We will explore the extent to which the approximately 
2700 complaint investigations conducted each year by CMS or states may 
be used as a valuable database to assess JCAHO accreditation practice.

We are gratified by observations in the report that the CMS has 
properly executed its statutory responsibility to continually study the 
operation and administration of the JCAHO hospital accreditation 
process and to submit an annual report to Congress.

With regard to JCAHO's performance, it is worth noting that "physical 
environment" represents the area of greatest discrepancy between the 
JCAHO findings and the CMS-sponsored validation surveys. Compliance 
with life-safety codes is the most common issue in the area of 
"physical environment," typically involving fire-safety precautions.

Of all facilities in which JCAHO missed a deficiency finding, about 68 
percent were accounted for by "physical environment" issues. This 
compared with a facility discrepancy rate of about 29 percent for 
health care only, and 3 percent where there was a finding of deficiency 
for both health care and physical environment.

Facilities with Findings Missed by JCAHO FY 2000-2002:

[See PDF for image]

[End of figure]

In the past eight years, in its annual reports to Congress and in its 
dialogue with JCAHO, CMS has identified the issue of physical 
environment as an important area that needs further attention. For 
example, the 1996 report to Congress observed that "validation surveys 
show that the CoP of Physical Environment continues to be the most 
frequently cited condition based on noncompliance with LSC." More 
recently, CMS' 2002 report to Congress continued to emphasize that "we 
have identified inconsistencies in its [LSC] implementation that we 
believe contributes to the differences in the validation findings."

CMS has always considered life-safety code compliance, on the part of 
all provider types, to be of critical importance. For this reason the 
2002 CMS report to Congress summarized five specific remedial steps 
that we recommended to JCAHO, and the 2003 report reported on the 
extent of progress being made. While JCAHO implementation of these 
recommendations has not been as expeditious as we desired, we are 
pleased to see some significant progress. The most recent example of 
such progress is the JCAHO agreement with the American Society for 
Healthcare Engineering (ASHE) to construct an electronic assessment 
tool, train JCAHO staff, and to support JCAHO's efforts to recruit 
health care facility engineers.

We will continue to emphasize with JCAHO the need to improve both 
health care and life-safety code compliance. Improvement in the area of 
life-safety code compliance would, by itself, bring JCAHO accreditation 
considerably closer to the findings rendered in the CMS-sponsored 
validation surveys.

With respect to the role of CMS and the CMS validation program, the 
report contends that the "rate of disparity" measure, as codified in 
federal regulations, is not sufficiently sensitive. The "rate of 
disparity" measure is used to gauge an accreditation organization's 
performance. In addition, the GAO report contends that the methodology 
for sample selection is not conducted in a statistically sophisticated 
manner, nor are the results presented in a way that extends the 
findings to all JCAHO hospitals through the application of an 
algorithm. Finally, the report conveys concern about the previous CMS 
decision to reduce the validation sample size from 5 percent to a 
state-stratified 1 percent sample.

We concur with GAO's view that the sampling methodology, sample size, 
and the formula for calculating the rate of disparity should be 
reevaluated. As GAO staff are aware, CMS has been actively studying 
these issues in the past two years. We have reached conclusions similar 
to those in the GAO study. as articulated in the attached document.

Thank you again for the opportunity to review and comment on the draft 
report. We look forward to seeing the final report and to working 
together to improve hospital care for the nation's Medicare 
beneficiaries. Attached are CMS' specific comments to GAO's 
recommendations.

Attachment:

Centers for Medicare & Medicaid Services' Comments to the GAO Draft 
Report: MEDICARE. CMS Needs Additional Authority and Better Measures to 
Adequately Oversee the Hospital Accreditation Program (GAO-04-850).

GAO Recommendation:

Modify the method used to measure the rate of disparity between 
validation survey findings and accreditation program findings to 
provide a reasonable assurance that Medicare COPS are being met and 
consider whether additional measures are needed to accurately reflect 
an accreditation program's ability to detect deficiencies in Medicare 
COPS.

CMS Response:

The rate disparity calculation is specified in Federal regulations at 
42 CFR section 488.8. However, it is quite appropriate to reexamine the 
rule and to consider additional or alternative measures to assess the 
performance of the accreditation organizations. The CMS has already 
begun to examine this issue as part of the agency's hospital quality 
improvement activities. We are working to refine existing measures and 
develop new ones.

It will be necessary to undertake rulemaking to revise the formula for 
calculating the rate of disparity measure, as well as to validate the 
threshold for acceptable performance or reasonable assurance. We 
believe that the notice and comment procedures inherent in the 
rulemaking process will provide an appropriate forum for this 
discussion of this significant public policy and will allow all of the 
stakeholders to participate. It will also provide for exposure to new 
perspectives and may yield innovative approaches to these problems that 
may have eluded us in the past.

In addition, we will explore regulatory strategies to address the long-
standing JCAHO performance issues with respect to the Life Safety Code. 
We will propose that this initiative be added to the Department's 
Regulatory Plan for FY 2005. These approaches will require additional 
CMS resources in terms of FTEs and additional funding.

GAO Recommendation:

Provide in the annual report to Congress an estimate, based on the 
validation survey sample, of the performance of all JCAHO-accredited 
hospitals, including the limitations and protocols_for these estimates 
based on generally accepted sampling and statistical methodologies, and 
develop a written protocol for these methodologies.

CMS Response:

It is appropriate to explore the possibility of developing and 
implementing new sampling and statistical methodologies within 
generally accepted statistical practices. We will examine whether 
alternate measures can more appropriately be generalized to the 
universe of all JCAHO hospitals. We will attempt to secure the 
additional resources necessary to undertake a thorough 
examination of these issues, to propose alternative sampling 
methodologies and develop more robust statistical analyses.

GAO Recommendation:

Annually conduct traditional validation surveys on a sample of JCAHO-
accredited hospitals that is equal to at least 5% of all JCAHO-
accredited hospitals.

CMS Response:

We will seek to increase the validation sample size as we formulate 
future budget requests. However, rather than simply increasing the 
sample rate to 5 percent, there may be more cost-effective approaches 
to enhancing our survey activities.

We note that a return to the 5 percent validation sample would require 
additional survey and certification funding that ranges from about $2.6 
million annually to almost $4.8 million per year, depending on the 
sampling methodology. Thus, additional cost-effective methods to assess 
JCAHO performance that would offset the need for major additional 
investments in full, traditional CMS validation surveys are likely to 
be valuable.

One such approach may be to make use of the database represented by the 
approximately 2700 complaint investigations conducted in accredited 
hospitals by CMS and states. We will undertake an initiative to analyze 
the extent to which this database may be useful in assessing JCAHO 
accreditation practice, and then develop analytic tools to put relevant 
findings into an improvement plan with JCAHO. In addition, to the 
extent that we can increase surveyor time in accredited hospitals, we 
will explore risk-based approaches that valuable surveyor time on those 
areas of JCAHO accreditation in which problems are most likely.

Finally, we will also seek regulatory changes that would provide CMS 
with additional and more substantial information on the JCAHO processes 
and findings so as to improve both overall CMS oversight and the 
effectiveness of CMS validation surveys.

[End of section]

Appendix V: Comments from the Joint Commission on Accreditation of 
Healthcare Organizations:

Joint Commission on Accreditation of Healthcare Organizations:

July 12, 2004:

Mr. David Walker: 
Comptroller General: 
Government Accountability Office 
441 G Street, N.W.

Washington, DC:

Dear Mr. Walker:

We would like to thank the Government Accountability Office for the 
opportunity to review the draft report entitled Medicare: C MS needs 
Additional Authority and Better Measures to Adequately Oversee the 
Hospital Accreditation Program. Because evaluating the performance of 
any organization is a complex undertaking, solicitation of the views of 
the entity under scrutiny helps to improve the accuracy of the analysis 
and provides context for the assessment.

The GAO's key recommendation is that "Congress should consider giving 
CMS the same kind of authority over the Joint Commission's hospital 
accreditation program that it has over all the other Medicare 
accreditation programs." The Joint Commission interposes no objection 
to this suggested statutory change, but takes great exception to the 
fact that the GAO arrives at this conclusion based upon a flawed study 
methodology and erroneous, alarming statistics that seriously mislead 
the public and do a great disservice to the Joint Commission.

When the deeming provision respecting hospital oversight was 
incorporated into the Medicare statute in 1965, the Congress did so on 
its own cognizance. The Joint Commission never sought this deemed 
status relationship nor was it even aware of the framing of the 
statutory provision respecting hospital accreditation that, to this 
date, limits the Executive Branch's oversight of the Joint Commission.

Nevertheless, the Joint Commission has always worked with CMS as if CMS 
had the same oversight authority for hospitals that it exercises for 
the other newer federal deemed status relationships with the Joint 
Commission (e.g., for home health care, for ambulatory surgery 
centers). This long-standing, positive working relationship has 
provided the nation enormous benefits through assuring continuous 
access to state-of-the-art evaluation of health care quality and 
patient safety in hospitals that are unparalleled in the world, and are 
looked to internationally as the gold standard for assessing hospital 
services. Further, the fact that only 2% of Medicare Conditions of 
Participation were found out of compliance by the CMS in hospitals 
during the three years of this subject GAO study is testimonial to the 
positive impacts of the effective working relationship between the 
Joint Commission and the CMS (and its predecessor) over the past four 
decades. These efforts to continuously improve health care quality and 
patient safety not only give beneficiaries confidence in their 
providers, but also ensure that the government is getting value for its 
spending. The Joint Commission leadership role in this area is evident 
in the fact that many private insurers and employers insist that 
hospitals serving their plan members be accredited by the Joint 
Commission.

The Joint Commission launched a new accreditation process in January of 
this year after three years of careful design and field testing. Each 
of the evaluation techniques incorporated into the new evaluation 
approach had previously been validated by other established evaluators 
both in health care and in other venues. We undertook this set of 
sweeping changes because of our commitment to continuous quality 
improvement and because we believe that a patient-centered approach to 
evaluation provides the most meaningful assessment of hospital 
performance. It also provides a strengthened vehicle for assuring 
continuous hospital attention to our standards requirements. While 
technically not authorized to approve the new accreditation process, 
CMS staff was briefed on its design on several occasions. Were CMS to 
have had such authority, this would have clearly made the Joint 
Commission even more comfortable in implementing the new accreditation 
process.

Thus, the Joint Commission takes no exception to the GAO's 
recommendation that the Congress consider giving CMS the same 
authority, over the Joint Commission's hospital accreditation program 
that it has over the other deemed Joint Commission accreditation 
programs. However, such a chance would make sense irrespective of the 
performance of the Joint Commission-and certainly should not be colored 
by the inflammatory, grossly inaccurate portrayal of the Joint 
Commission that is set forth in this seriously flawed GAO study. On 
July 7, 2004, the Joint Commission submitted to the GAO a 26-page 
technical corrections document which details the serious errors in and 
omissions from the draft GAO study. These are summarized below:

Methods and Use of Statistics:

Evaluating performance is a complex task. Like so many issues involving 
assessment, if one asks the wrong question, one gets the wrong answer. 
The GAO methodology seeks to assess the ability of the Joint Commission 
to evaluate hospital compliance with the Medicare CoPs by conducting 
"missed deficiency rate" analyses. However, the calculations performed 
by the GAO are at best incomplete, for the GAO has not included in the 
calculus the number of deficiencies found by the Joint Commission but 
not found by the State Survey Agency (SSAs.) The GAO has continued to 
ignore this key point, and has been undeterred in its focus on how many 
times the Joint Commission agrees with the SSAs. Essentially, what the 
GAO is providing to the reader is an incomplete ratio of "non-
agreement" between the Joint Commission and the State Survey Agencies.

This ratio, especially when used in isolation from other information, 
is neither a true indicator of Joint Commission effectiveness, nor an 
adequate exploration of whether the Join Commission's hospital 
accreditation program ensures that Medicare beneficiaries receive high 
quality care in keeping with the Medicare program's expectations. 
Specifically, the GAO does not acknowledge that non-agreement between 
the SSA and the Joint Commission is influenced by a number of factors, 
including differences in interpretation of standards compliance; the 
disproportionate rigidity of the CoPs requirements and their related 
scoring mechanisms compared to the Joint Commission's accreditation 
process; variations in the timing of the Joint Commission and SSA 
surveys; and other artifacts inherent in the validation program. Even 
with these imponderables, it is extraordinary that accredited hospitals 
are found to be in compliance with 98 percent of the CoPs in SSA 
validation surveys. It is even more extraordinary that this significant 
finding is omitted from the highlights page and is buried in the text 
under a finding that misleads the Congress and policymakers into 
believing that the Joint Commission does not identify serious 
deficiencies.

We further note that while the GAO claims that the Joint Commission 
misses 69 percent of-out-compliance CoPs, the GAO has neither the 
complete files of survey information nor the expertise necessary to 
make such a calculation. This allegation is therefore no more than 
conjectural and cannot be defended.

A paradox inherent in the GAO's statistical applications is that the 
GAO has chosen an even more inappropriate measure of evaluating whether 
COPS are in compliance than the one that they criticize the CMS for 
using over the past few years. Using the GAO metric, the greater the 
degree of Joint Commission success in ensuring that hospitals are, or 
become, in compliance with the Cops following its on-site accreditation 
surveys, the greater the likelihood that any SSA findings later in time 
will be considered "new" deficiencies. Even if these new deficiencies 
are exquisitely small in number, they will then represent a 100 percent 
"missed deficiency rate," thereby further misleading and potentially 
alarming the Congress and the public.

Furthermore, to provide an adequate, more accurate assessment of the 
Joint Commission's performance in assuring that safe, high quality care 
is available to Medicare beneficiaries; the GAO should be advising the 
CMS to take into account: (1) the actual number of serious deficiencies 
that existed in hospitals to be surveyed by SSAs before the Joint 
Commission and SSA surveys and how many of these were identified by the 
Joint Commission and corrected before the validation surveys; (2) the 
percentage of allegation (complaint) surveys that resulted in a finding 
that the hospital had at least one serious deficiency (an astounding 
low number of hospitals during the GAO study period, as previously 
reported to the Congress); and (3) the multiple value-added 
requirements that the Joint Commission requires of accredited 
organizations, such as the public reporting of clinical performance 
data, special requirements related to national patient safety goals, 
and over 100 standards that do not have corresponding Medicare 
requirements and reflect expectations relating to the state-of-the-art 
provision of care in hospitals. While Medicare's CoPs have not been 
fully updated since 1986, the Joint Commission has annually updated its 
standards - with public sector input - to push hospitals to continually 
improve the quality and safety of the care they provide.

The New Survey Process:

The GAO misunderstands and misrepresents the Joint Commission's 
evolution to a new accreditation process and in so doing appears to 
lack a basic understanding of the tenets of quality improvement. The 
Joint Commission's goal is to leverage hospitals to become better at 
what they do and to give the public confidence that their care is 
meeting contemporary standards. The GAO has been unrelenting in its 
focus on whether the pilot test findings of the new accreditation 
process would have led to a different distribution of hospitals with 
full accreditation status versus lower levels of accreditation, such as 
conditional or provisional. Further, the GAO has ignored the compelling 
data provided by the Joint Commission which show that the new survey 
accreditation process results in better discernment of the types of 
deficiencies that are directly related to patient care than the old 
process. We cannot over-emphasize this important fact. Evaluating and 
improving the quality of care in hospitals is not about counting 
deficiencies, it is about finding those deficiencies which, if not 
fixed, will generate poor results for patients, and making sure that 
these deficiencies are remedied in a timely fashion.

Life Safety Code (LSC):

The GAO has mischaracterized the Joint Commission's response to the 
five 2002 CMS recommendations for improving the LSC disparity rate. The 
Joint Commission has taken significant steps to address each of the CMS 
recommendations, and that information was provided to the GAO. Evidence 
of our commitment to the CMS recommendations is reflected by the fact 
that there was an approximately 50 percent decline in the number of 
hospitals found to be out of compliance with the LSC in the validation 
surveys during the study period. While the "disparity rate" declined 
only slightly over the study period, the number and percentage of 
hospitals that were found to be out of compliance with the LSC by the 
SSAs during this time period declined from 43 hospitals (23 percent) in 
the 2000 Medicare validation report to the Congress to 25 hospitals (12 
percent) in the 2002 Congressional report. This point underscores the 
inadequacy of the "missed deficiency rate" metric suggested by the GAO. 
By its nature, this metric does not account for improvements in 
hospital standards compliance.

An additional important point not mentioned in the GAO report is that 
the Joint Commission lobbied strongly and eventually successfully to 
have the CMS adopt the 2000 version of the LSC rather than the 1985 
version in use throughout the GAO study period. This difference in 
requirements contributed significantly to the identified disparities 
between Joint Commission and SSA surveys.

Finally, the GAO report fails to put hospital physical safety issues 
into perspective for the Congress, thus leaving the reader with the 
impression that accredited hospitals are not safe. Hospitals arc in 
fact one of the safest health care occupancies in the nation, owing in 
large part to the attention that the Joint Commission has placed on the 
safety of the physical environment.

Conclusion:

In closing, the Joint Commission is deeply concerned that the GAO has 
provided the public with a report that neither uses credible metrics 
nor includes highly relevant information about the Joint Commission's 
performance. In our view, it is irresponsible to alarm the public using 
statistics that have little meaning, and that do not reflect the true 
oversight of America's hospitals through Medicare's public-private 
sector partnership with the Joint Commission.

Sincerely,

Signed by: 

Dennis S. O'Leary, M.D. 
President: 

[End of section]

Appendix VI: GAO Contact and Staff Acknowledgments:

GAO Contact:

Marcia A. Mann, (202) 512-9526:

Acknowledgments:

In addition to the contact named above, Elaine Swift, Linda Kohn, Behn 
Kelly, Elizabeth T. Morrison, Roseanne Price, and Marie Stetser made 
key contributions to this report.

[End of section]

Related GAO Products:

Medicare Home Health Agencies: Weaknesses in Federal and State 
Oversight Mask Potential Quality Issues. GAO-02-382. Washington, D.C.: 
July 19, 2002.

Medicare: HCFA's Approval and Oversight of Private Accreditation 
Organizations. GAO/ HEHS-99-197R. Washington, D.C.: September 30, 1999.

Home Health Care: HCFA Properly Evaluated JCAHO's Ability to Survey 
Home Health Agencies. GAO/HRD-93-33. Washington, D.C.: October 26, 
1992.

Health Care: Criteria Used to Evaluate Hospital Accreditation Process 
Need Reevaluation. GAO/HRD-90-89. Washington, D.C.: June 11, 1990.

FOOTNOTES

[1] See 42 U.S.C. § 1395bb(a) (2000). 

[2] JCAHO is referred to in statute under its former name, the Joint 
Commission on Accreditation of Hospitals. 

[3] JCAHO develops its standards with a committee of experts and 
stakeholders, such as the government, hospitals, and consumers. 

[4] HHS OIG, The External Review of Hospital Quality: A Call for 
Greater Accountability, OEI-01-97-00050 (Washington, D.C.: July 1999). 

[5] One of the 23 COPs cannot be deemed by an accreditation 
organization. CMS relies on organizations other than the accreditation 
organizations to certify that hospitals comply with the COP that 
requires hospitals to establish a utilization review plan for services 
provided to Medicare beneficiaries. 

[6] Specifically, the agency's regulations require the accreditation 
organization's standards to be at least as stringent as the Medicare 
COPs, when taken as a whole. See 42 C.F.R. § 488.6(a) (2003).

[7] Forty-nine states allow JCAHO hospital accreditation as a full or 
partial substitute for meeting health care quality standards and other 
requirements for state licensure. 

[8] The remaining 18 percent of hospitals choose to be accredited by 
the American Osteopathic Association (AOA) or to be certified by state 
survey and certification agencies. 

[9] The board includes seven members chosen by the American Hospital 
Association, seven chosen by the American Medical Association, three 
chosen by the American College of Physicians--American Society of 
Internal Medicine, three chosen by the American College of Surgeons, 
and one chosen by the American Dental Association. In addition, the 
board consists of a nurse-at-large and six public members. The 
president of JCAHO is an ex officio member of the board. 

[10] When Congress first established JCAHO's deeming authority in 1965, 
it prohibited federal authorities from issuing standards on patient 
health and safety for hospitals higher than comparable requirements for 
hospital accreditation by JCAHO in deference to the expertise of 
professional accreditation organizations sponsored by medical and 
hospital associations. See Pub. L. No. 89-97, § 102(a), 79 Stat. 286, 
315 (1965). Subsequent legislation removed the prohibition and required 
JCAHO to demonstrate that its standards were at least equivalent to any 
such higher standards issued by the Secretary in order to have deeming 
authority in that area. See Pub. L. No. 92-603, § 244(c), 86 Stat. 
1329, 1423 (1972). 

[11] AOA solely accredits approximately 2 percent of hospitals and 
JCAHO and AOA jointly accredit less than 1 percent of hospitals. While 
JCAHO and AOA are currently the only hospital accrediting 
organizations, federal law permits CMS to approve any other national 
accreditation body that demonstrates that Medicare requirements will be 
met by hospitals it accredits. 

[12] See 42 U.S.C. § 1395ll(b). 

[13] For this report, we will refer to traditional validation surveys 
as validation surveys.

[14] 42 C.F.R. 488.8(e).

[15] Between fiscal years 2000 and 2002, JCAHO used more recent life 
safety code standards than state survey agencies performing validation 
surveys. CMS stated that these differences could account for some of 
the disparate findings between JCAHO's surveys and state agency 
validation surveys. However, CMS considered these different standards 
in determining whether JCAHO had not detected serious deficiencies in 
the life safety code.

[16] U.S. General Accounting Office, Medicare Home Health Agencies: 
Weaknesses in Federal and State Oversight Mask Potential Quality 
Issues, GAO-02-382 (Washington, D.C.: July 19, 2002) and U.S. General 
Accounting Office, Nursing Home Quality: Prevalence of Serious 
Problems, While Declining, Reinforces Importance of Enhanced Oversight, 
GAO-03-561 (Washington, D.C.: July 15, 2003).

[17] All six states conducted at least 15 validation surveys from 
fiscal year 2000 through 2002.

[18] In 2004 and 2005, JCAHO will continue to conduct its accreditation 
surveys on an announced basis. 

[19] For example, we found in our nursing home survey work in 1999 that 
state surveyors may perform their tasks more attentively when they are 
being observed by federal surveyors than they would if performing their 
surveys unobserved, thus masking a state surveyor's typical 
performance. U.S. General Accounting Office, Nursing Home Care: 
Enhanced HCFA Oversight of State Programs Would Better Ensure Quality, 
GAO/HEHS-00-6 (Washington, D.C.: Nov. 4, 1999).

[20] Whenever CMS considers, approves or removes an accreditation 
organization's deeming authority, the agency is required to publish 
detailed notices in the Federal Register, and consider public comment. 
See 42 U.S.C. § 1395bb(b)(3); 42 C.F.R. § 488.8(b) and (f)(7).

[21] Beginning in 1995, JCAHO-accredited hospitals have assessed their 
own compliance with the life safety code and developed correction 
plans, which JCAHO must approve. If hospitals are in compliance with 
their correction plans, JCAHO's surveyors do not record outstanding 
life safety code deficiencies.

[22] The example is based on the analysis of the rate of disparity in 
American Institutes of Research, Measurement and Evaluation of Revised 
Accredited Hospital Validation and Oversight (Washington, D.C.: Nov. 6, 
2002). 

[23] For example, CMS does not measure and report the precision of the 
estimates from the sample of validation surveys through the use of 
confidence intervals or margins of error, which define the range of 
estimates that sample results would yield given different random 
samples for a specified level of certainty.

[24] These estimates were developed assuming that the validation 
surveys are conducted on a simple random sample of JCAHO-accredited 
hospitals and a 95 percent confidence level.

[25] In a probability sample, each eligible hospital accredited in a 
given year would have to have a known, nonzero chance for selection in 
the sample. 

GAO's Mission:

The Government Accountability Office, the investigative arm of 
Congress, exists to support Congress in meeting its constitutional 
responsibilities and to help improve the performance and accountability 
of the federal government for the American people. GAO examines the use 
of public funds; evaluates federal programs and policies; and provides 
analyses, recommendations, and other assistance to help Congress make 
informed oversight, policy, and funding decisions. GAO's commitment to 
good government is reflected in its core values of accountability, 
integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony:

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains 
abstracts and full-text files of current reports and testimony and an 
expanding archive of older products. The Web site features a search 
engine to help you locate documents using key words and phrases. You 
can print these documents in their entirety, including charts and other 
graphics.

Each day, GAO issues a list of newly released reports, testimony, and 
correspondence. GAO posts this list, known as "Today's Reports," on its 
Web site daily. The list contains links to the full-text document 
files. To have GAO e-mail this list to you every afternoon, go to 
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order 
GAO Products" heading.

Order by Mail or Phone:

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to:

U.S. Government Accountability Office

441 G Street NW, Room LM

Washington, D.C. 20548:

To order by Phone:

	

Voice: (202) 512-6000:

TDD: (202) 512-2537:

Fax: (202) 512-6061:

To Report Fraud, Waste, and Abuse in Federal Programs:

Contact:

Web site: www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470:

Public Affairs:

Jeff Nelligan, managing director,

NelliganJ@gao.gov

(202) 512-4800

U.S. Government Accountability Office,

441 G Street NW, Room 7149

Washington, D.C. 20548: