Evidence Report/Technology Assessment

Number 13

Prepared for:
Agency for Healthcare Research and Quality

Department of Health and Human Services
U.S. Public Health Service
2101 East Jefferson Street
Rockville, MD 20852
www.ahrq.gov

Contract No. 290-97-0020

Prepared by:
ECRI, Plymouth Meeting, PA
Health Technology Assessment Group

AHRQ Publication No. 00-E013

May 2000

The Agency for Healthcare Research and Quality (AHRQ), through its Evidence-Based Practice Centers (EPCs), sponsors the development of evidence reports and technology assessments to assist public- and private-sector organizations in their efforts to improve the quality of health care in the United States. The reports and assessments provide organizations with comprehensive, science-based information on common, costly medical conditions and new health care technologies. The EPCs systematically review the relevant scientific literature on topics assigned to them by AHRQ and conduct additional analyses when appropriate prior to developing their reports and assessments.

To bring the broadest range of experts into the development of evidence reports and health technology assessments, AHRQ encourages the EPCs to form partnerships and enter into collaborations with other medical and research organizations. The EPCs work with these partner organizations to ensure that the evidence reports and technology assessments they produce will become building blocks for health care quality improvement projects throughout the Nation. The reports undergo peer review prior to their release.

AHRQ expects that the EPC evidence reports and technology assessments will inform individual health plans, providers, and purchasers as well as the health care system as a whole by providing important information to help improve health care quality.

We welcome written comments on this evidence report. They may be sent to: Director, Center for Practice and Technology Assessment, Agency for Healthcare Research and Quality, 6010 Executive Blvd., Suite 300, Rockville, MD 20852.

Douglas B. Kamerow, M.D.	John M. Eisenberg, M.D.
Director, Center for Practice and Technology Assessment	Director
Agency for Healthcare Research and Quality	Agency for Healthcare Research and Quality

The authors of this report are responsible for its content. Statements in the report should not be construed as endorsement by the Agency for Healthcare Research and Quality or the U.S. Department of Health and Human Services of a particular drug, device, test, treatment, or other clinical service.

The key question of this report, as posed by the Social Security Administration, asks, "Do the current criteria cited in SSA's Listing of Impairments ("Listings") for chronic renal failure correlate with an inability to work for 12 consecutive months?" The current Listings assume that all patients on dialysis have an impairment severe enough to prevent them from doing substantial gainful activity. Our goals were to determine whether existing evidence supported or refuted this assumption and to determine if there were alternative criteria that might be more predictive of inability to work.

Renal failure may be "acute," occurring from a sudden injury or illness, such as a blow to the abdomen, bacterial infection, or drug overdose or "chronic" as a result of conditions, such as glomerulonephritis, diabetes, hypertension, and heart disease. When chronic renal failure (CRF) becomes severe enough that replacement therapy (dialysis or kidney transplant) is required, the functional diagnosis is end-stage renal disease (ESRD).

Approximately 110 out of every 100,000 people are diagnosed with ESRD, and the United States Renal Data System (USRDS) estimated that more than 300,000 individuals in the United States had ESRD as of 1997. The average patient with ESRD has a survival time between 19 and 47 percent of the age-, sex-, and race-matched U.S. population.

This project was divided into two phases. For Phase 1, we sought evidence in the published literature. We searched 27 databases (including MEDLINE and EMBASE) for relevant information. Most of these databases were last searched in late 1998, at the end of Phase 1 of this contract. Searches of the World Wide Web were also conducted.

Individual patient data of the USRDS Dialysis Morbidity and Mortality Study (DMMS) Wave 2 database were analyzed in Phase 2 of this contract

The search strategies identified 3,492 documents, books, and World Wide Web resources. A total of 503 documents were ordered and read in full. Fourteen studies were identified that analyzed predictors of employment, all of which attempted to correlate physiological, functional, and/or psychological factors with employment status.

Limitations in the 14 studies identified in Phase 1 precluded using them to evaluate the current criteria cited in the SSA Listing of Impairments for CRF. Therefore, we examined data in the DMMS Wave 2 special study of the USRDS. Patients in the DMMS Wave 2 study were followed prospectively for 1 year, and assessed physiologically, functionally, and psychologically at the beginning and end of the 1-year period. More than 300 variables were included in this database of 4,026 patients. Outcome variables included self-reported ability to work and work status.

We conducted numerous de novo statistical calculations on these data to determine the external validity, construct validity, reliability, and reproducibility of statistical analyses that could be performed using this database. We also conducted an illustrative analysis, including missing data imputation, data recoding, regression analysis, and assessment of diagnostic efficacy.

These analyses demonstrated that neither the published literature nor the DMMS Wave 2 data could be used to answer the key question posed by SSA. The primary limitation of the DMMS database arose because, although 4,026 patients were initially included, approximately 43 percent were over age 65 and not eligible for our analysis, and an additional 37 percent were lost to followup. This reduced the data set to a small number of patients and a large number of potentially relevant variables. This, combined with the different results obtained with randomly selected halves of the database rendered the results of our analyses unreliable.

Currently available evidence does not allow us to answer the key questions of this evidence report. Answering these questions would require a large-scale prospective study of patients who are followed rigorously for at least 1 year to monitor their functional, physiological, and disability status.

This document is in the public domain and may be used and reprinted without permission, except for those copyrighted materials noted for which further reproduction is prohibited without the specific permission of copyright holders. AHRQ appreciates citation as to source, and the suggested format is provided below: ECRI Health Technology Assessment Group. Determinants of Disability in Patients With Chronic Renal Failure. Evidence Report/Technology Assessment No. 13 (Prepared by ECRI under Contract No. 290-97-0020). AHRQ Publication No. 00-E013. Rockville, MD: Agency for Healthcare Research and Quality. May 2000.

The purpose of this report is to evaluate the U.S. Social Security Administration's (SSA's) current Listing of Impairments ("Listings") for determining disability in individuals with chronic renal failure (CRF). Renal failure occurs when the kidneys lose their ability to filter wastes from the blood. It can occur as a result of chronic conditions such as primary kidney disease (e.g., glomerulonephritis), diabetes, hypertension, and heart disease. Renal failure can also be acute, occurring from a sudden injury or illness, such as a blow to the abdomen, bacterial infection, or drug overdose. This report focuses on CRF, which is much more common than acute renal failure. When the severity of CRF reaches the point at which the patient can no longer function without a kidney transplant or dialysis, this is functionally defined as end-stage renal disease (ESRD).

Approximately 110 out of every 100,000 people are diagnosed with ESRD. The United States Renal Data System (USRDS) has estimated that more than 300,000 individuals in the United States had ESRD as of 1997. Determining the prevalence of CRF is more difficult because many people who suffer from earlier stages of the disease have either not been diagnosed or have not sought treatment; therefore, reliable prevalence statistics are not available. SSA's current Listings consider every patient requiring dialysis or kidney transplantation as a result of CRF to be unable to perform gainful activity for at least 12 consecutive months, and therefore "disabled" at the third step in SSA's sequential disability evaluation process. Our goal was to determine if there were alternative criteria that might be more predictive of inability to work.

This report focuses solely on those patients undergoing dialysis; patients with a renal transplant have not been considered in this report. Our goal was to determine whether the CRF criteria for disability status, last revised in 1979, are still applicable, as newer treatment modalities may have allowed some patients to continue normal daily activities within the limitations of their disease. Our approach was to analyze the scientific evidence in the published literature that pertained to the ability of patients with CRF to maintain or resume their working status. We specifically sought evidence about what physiological and functional status measurements could predict an individual's inability to work.

Information in the published literature was too sparse to answer the questions addressed in this report. We then examined data from the USRDS, a nationwide registry of dialysis centers. Our goal was to perform multivariate statistical analyses on the raw data in order to identify the best physiological and functional predictors of inability to work. Because of difficulties measuring inability to work directly, we used vocational status and self-reported ability to work as indirect measurements. To test the validity and reliability of these data, we performed numerous different original statistical analyses, including bivariate Spearman's rho and Pearson's r correlations of more than 200 variables, analysis of variance (ANOVA), and Kolmogorov-Smirnov nonparametric statistics, as well as a sample regression analysis comparing the results from two random halves of the database.

Although USRDS data provided a lot of interesting and useful information and demonstrated adequate construct and external validity, the reliability of any disability-related complex multivariate analyses of these data was considered suspect. This is due to a large amount of missing data, as many patients did not answer certain essential questions. We therefore were unable to answer the questions that SSA asked. We do, however, present data about the vocational characteristics of patients with ESRD and an illustrative complex statistical analysis, which may help focus future research.

The key question posed by SSA for this report was:
Do the current criteria cited in SSA's Listing of Impairments for CRF correlate with an inability to work for 12 consecutive months?

Measurement of "inability to work" creates difficulties in answering this question. Inability to work is a concept that is difficult to measure quantitatively, and therefore has not been directly addressed in either the published literature or the USRDS. Surrogate measures such as self-reported ability to work and work status have been used in the literature instead. Therefore, SSA's followup questions to the above key question are:

This evidence report is epidemiological in nature, assessing physiologic and functional predictors of inability to work in a quantitative fashion. We do not seek to compare the effectiveness of any therapeutic interventions, modalities, or technologies, because treatment efficacy is only a very indirect measure of a patient's true state of health. The population of interest in this report included all patients with CRF, including those with ESRD. However, due to limitations in the data available, we focused primarily on patients with ESRD.

Due to the above-mentioned difficulties in directly measuring inability to work, the outcomes of interest are indirect measures of inability to work. We therefore focused on three different indirect outcome measurements: self-reported ability to work, work status, and functional status (as measured by the Kidney Disease Quality-of-Life Questionnaire).

This project was divided into two phases. During Phase 1, we assessed the availability and quality of data contained in the published literature pertaining to this topic. During Phase 2, we assessed the feasibility of using individual patient registry data to conduct de novo statistical analyses in order to answer the key question.

We searched 27 databases for relevant information. We searched for information in each database from the date of its inception; therefore all records in these databases were considered:

ABI/Inform (through November 12, 1998)

Abledata (NARIC) (through November 12, 1998)

The Cochrane Database of Systematic Reviews (through 1999 Issue 2)

The Cochrane Registry of Clinical Trials (through 1999 Issue 2)

The Cochrane Review Methodology Database (through 1999 Issue 2)

Combined Health Information Database (CHID) (through November 2, 1998)

CRISP (through December 3, 1998)

Current Contents (through June 1999)

Database of Reviews of Effectiveness (Cochrane Library) (through 1999 Issue 2)

DIRLINE (through November 1998)

ECRI Library Catalog (through October 29, 1998)

EMBASE (Excerpta Medica) (1980 through November 23, 1998)

Health Care Financing Administration Database (through April 22, 1999)

Health Devices Alerts (1977 through June 1999)

Healthcare Standards (1975 through June 1999)

Health Devices Sourcebase (through June 1999)

HealthSTAR (Health Services, Technology, Administration, and Research) (1980 through April 13, 1999)

HSRProj (through November 4, 1998)

Hypertension, Dialysis and Clinical Nephrology© (HDCN) (through June 1999)

International Health Technology Assessment (IHTA)© (1990 through June 1999)

MEDLINE (1980 through April 13, 1999)

Nursing and Allied Health (NAHL/CINAHL) (1980 through October 20, 1998)

PsycINFO (1980 through December 2, 1999)

RehabDATA (NARIC) (through November 12, 1998)

Sci Citation Index (through October 22, 1998)

Social SciSearch (through October 21, 1998)

TARGETTM (through October 6, 1999)

The search strategies employed a number of free-text keywords as well as controlled vocabulary terms including (but not limited to) the following concepts:

• Study design-Controlled trials: Randomized controlled; controlled clinical trials; meta-analysis; random allocation; single-blind method; double-blind method, evidence-based medicine (includes randomized controlled trials, outcomes research, and meta-analysis).
• Disability: Disabled; disability; disability evaluation.
• Disorders: ESRD; end-stage renal disease; ESRF; end-stage renal failure; kidney failure, chronic.
• Interventions: Dialysis; haemodialysis; hemodialysis; peritoneal dialysis; renal replacement therapies; kidney transplantation.
• Miscellaneous: Educational status; patients; patient compliance; patient participation; predictive value of tests; quality-of-life; QOL; sex factors; social class; socioeconomic factors; time factors.
• Work: Employment; employability; employment status; job re-entry; re-employment; unemployment; vocational rehabilitation; work capacity evaluation; workload; work scheduling.

In general, the searches were restricted to studies examining human subjects. Case reports were excluded.

Searches of the World Wide Web were also conducted using various search engines including (but not limited to) AltaVista, Hotbot, Infoseek, Magellan, and Yahoo!. Pertinent Web sites included:

American Association of Kidney Patients (www.aakp.org)

About Epogen (wwwext.Amgen.com/cgi-bin/genobject/productEpogen/tig__5Znvfv)

Directory of Kidney and Urologic Diseases Organizations (www.niddk.nih.gov/health/kidney/pubs/kuorg/kuorg.htm)

Forum of End Stage Renal Disease Networks (www.esrdnetworks.org/)

Hypertension, Dialysis, & Clinical Nephrology (www.hdcn.com/)

Kidney and Urologic Diseases Statistics for the United States (www.niddk.nih.gov/health/kidney/pubs/kustats/kustats.htm)

National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK) (www.niddk.nih.gov/)

National Kidney Foundation (www.kidney.org/)

The Nephron Information Center (nephron.com/

Nephrology News and Issues (www.medicalnews.com/)

RENALNET (www.renalnet.org/renalnet/renalnet.cfm)

United States Renal Data System (www.med.umich.edu/usrds/)

Disability Resources Monthly (DRM) Guide to Resources on the Internet (www.geocities.com/~drm/)

Disability Statistics Center (dsc.ucsf.edu/)

Employment Project's Homepage. Efforts to remove work disincentives (www.teleport.com/~enygma/employ/)

National Institute on Disability and Rehabilitation Research (NIDRR) (www.ed.gov/offices/OSERS/NIDRR)

National Organization on Disability (www.nod.org/)

National Rehabilitation Information Center (NARIC) (www.naric.com)

Research Institutes, Universities, Rehabilitation Centres (www.gladnet.org/research.htm)

Vocational Evaluation and Work Adjustment Association (VEWAA) (www.vewaa.org/)

Nonjournal publications and conference proceedings from professional organizations, private agencies, and government agencies maintained in ECRI's collections were routinely reviewed.

Other mechanisms were used to retrieve additional relevant information, including review of bibliographies/reference lists from peer-reviewed and gray literature. (Gray literature includes reports, studies, etc. produced by local government agencies, private organizations, educational facilities, and corporations, etc., that do not appear in the peer-reviewed literature.)

These searches identified 3,492 documents, books, and World Wide Web resources. A research analyst reviewed the search results to identify relevant documents and to ensure that all relevant information was retrieved using such search strategies. Input from technical experts and members of an internal review committee also helped revise the search strategies. Through these processes, new searches were conducted, and a total of 503 documents were ordered and read in full. Fourteen studies were identified that contained any analysis of predictors of employment in individuals with CRF. All of these 14 studies pertained solely to adult ESRD patients and attempted to correlate physiological, functional, and psychological factors with employment status.

Through careful analysis of these studies, we determined that there are limitations to all of the published literature that preclude its analysis for answering SSA's key and followup questions:

• Most of the variables examined in the published literature were demographic (e.g., race, ethnicity) or psychological and, therefore, not ethically or easily incorporated into the SSA disability evaluation process.
• None of these studies was longitudinally designed to allow assessment of predictive value of independent variables at time 1 for outcome at time 2.
• Patients reported on in the published literature were contacted at many different time points after the start of dialysis. Most patients were examined or interviewed 5 to 6 years after beginning dialysis. This does not approximate the time frame of interest to SSA (1 year after beginning dialysis).
• Most studies used univariate statistical tests (e.g., chi-square or t-tests), which do not control for the effects that other variables might have upon the outcomes.

Because of these limitations, there are currently no published data available to either support or refute SSA's current Listings. As a result, we proceeded to Phase 2 of this project-an evaluation of whether individual patient data from the USRDS database are sufficient to answer the key questions of this project.

The USRDS collects patient and facility data from every Medicare-approved dialysis facility in the United States. This registry is sponsored by the National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK) and the Health Care Financing Administration (HCFA), and from 1995 to 1999 was coordinated by the University of Michigan. We were primarily interested in a subset of these data, a special study known as the Dialysis Morbidity and Mortality Study (DMMS) Wave 2. This was a prospective, longitudinal study that tracked 4,026 incident dialysis patients in 1996 and 1997, measuring physiological, functional, and quality-of-life variables. This patient subset was advantageous for our purposes because its time frame approximated the time frame of interest for SSA (1 year from onset of ESRD). Therefore, it seemed possible that these data could be used in a de novo regression analysis to determine the best predictors of inability to work, as measured indirectly by the variables "self-reported ability to work full time" and "work status." The purpose of the regression analysis would be to identify a set of diagnostic criteria that would maximize the accurate inclusion of truly disabled patients with ESRD while minimizing the inappropriate inclusion of patients who are able to work.

In order to determine if such an analysis was warranted, we conducted numerous validity and reliability tests on the data. It was important to determine whether the patients in the database were similar enough to the general population of dialysis patients for results to be generalizable ("external validity"); that the variables measured were internally consistent, such that similar measures produced similar results ("construct validity"); and that any analysis of the data would produce consistent results that could be reproduced by other researchers.

To assess this database in these ways, numerous different statistical calculations were employed, including, but not limited to, the following: bivariate correlations of more than 300 variables, Kolmogorov-Smirnov nonparametric comparisons of more than 100 nominal categorical variables, ANOVAs of more than 50 continuous quantitative variables, and a sample regression analysis. Overall, to assess the validity and reliability of this database, more than 500 de novo analyses were conducted. The results of these analyses were reviewed by three physicians in the fields of nephrology and pathology, as well as by several biostatisticians.

We also performed an analysis to determine whether a regression analysis done on one random half of the database would find the same results as a regression analysis done on the other random half of the database. This was an important consideration because, although the database initially followed 4,026 patients, this was reduced to 546 eligible patients. Many were excluded because they were over age 65, lost to followup, or did not fully complete the 1-year followup questionnaire. The final group of patients eligible for our main analysis comprised only 546 individuals, with fragmented data availability, thus reducing the statistical power of any complex multivariate analyses.

• The portion of the DMMS Wave 2 database of the USRDS relevant to this evidence report demonstrated acceptable external and construct validity.
• Results of multivariate regression analyses were not reliable. Results for one random half of the database did not approximate the results of the other half.
• Because the main analysis proposed for this report could not be conducted, we could not reach conclusions about SSA's current Listings for patients with CRF.
• An illustrative regression and receiver operating characteristic (ROC) analysis are included that provide some inconclusive information about the usefulness of the outcome measures and whether laboratory and physiological measures alone can be used to predict inability to work, in isolation from sociodemographic variables.
• Summary statistics indicated that while approximately 42 percent of ESRD patients were employed full time before beginning dialysis, only 21 percent were employed when they began dialysis, and only 13 percent were employed a year later. Those patients who continued working while on dialysis were most likely to be professional/white collar workers (49 percent).

Although the DMMS Wave 2 was a well-designed epidemiological study, its purpose was not to assess disability. Furthermore, only 42 percent of patients completed the entire followup patient questionnaire. Studying disability using data such as the DMMS Wave 2 requires one to "follow" a large number of patients who were working at the time they began dialysis until the time they completed the followup questionnaire. However, both the number of patients initially working and the number of initially working patients who completed the followup questionnaire were relatively small.

Limitations in existing research that should be addressed by future researchers include the use of univariate statistics, retrospective study design, and a focus on demographic and social variables. It would be most useful, for the purposes of answering the questions addressed here, for a study to collect multiple variables longitudinally such that, as did DMMS Wave 2, the independent variables are measured at time 1 and the dependent/outcome variable at time 2. An analysis that includes laboratory and physiologic measurements is essential. Information about each individual's Social Security disability status is also essential in order to evaluate the Listings.