Summary
Historically, the proportion of the Social Security Administration's (SSA) disability benefits claims that were approved has been lower for African-Americans than for whites. In 1992, GAO found that racial differences, largely at the Administrative Law Judge (ALJ) level, could not be completely explained by factors related to the decision-making process. This report examines how race and other factors influence ALJ decisions and assesses SSA's ability to ensure the accuracy and fairness of ALJ decisions.
GAO controlled for factors that are related to the disability decision-making process at the Administrative Law Judge level and found: (1) no statistically significant difference in the likelihood of being allowed benefits between white claimants and claimants from other, non-African-American racial/ethnic groups; and between white claimants and African-American claimants who were represented by attorneys; (2) statistically significant differences between white and African-American claimants who were not represented by attorneys (specifically, among claimants without attorneys, African-American claimants were significantly less likely to be awarded benefits than white claimants); and (3) other factors--including sex, income, and the presence of a translator at a hearing--also had a statistically significant influence on the likelihood of benefits being allowed. Due to the inherent limitations of statistical analysis, one cannot determine whether these differences by race, sex, and other factors are a result of discrimination, other forms of bias, or variations in currently unobservable claimant characteristics. Analytical, sampling, and data weaknesses in SSA's approach to quality assurance reviews limit its ability to ensure the accuracy and fairness of ALJ decisions. Analytic weaknesses: SSA analyzes ALJ decisions by various factors, such as SSA region, but not by the claimant's race. Sampling weaknesses: SSA currently excludes cases that have been appealed to the Appeals Council from the pool of ALJ cases that undergoes the quality assurance review. The exclusion of these cases could mean that the sample used by SSA in its quality assurance review is not representative of all ALJ decisions. While GAO did not find large differences in the sample of cases from 1997 to 2000 that it used for its analysis, the continued, systematic exclusion of cases that are under appeal could in the future result in an unrepresentative sample of all ALJ decisions. Data limitations: even if SSA wanted to conduct analyses by race/ethnicity, it would encounter difficulties doing so in the near future because, since 1990, SSA significantly scaled back its collection of race/ethnicity data. Although GAO had sufficient race data for its study, the scaled back collection of race/ethnicity data will impact SSA's future efforts to study ALJ benefit decisions by race. During GAO's review, however, SSA decided to collect race/ethnicity data for persons applying for Social Security benefits.
Recommendations
Our recommendations from this work are listed below with a Contact for more information. Status will change
from "In process" to "Implemented" or "Not implemented" based on our follow up work.
Director:
Team:
Phone:
Daniel Bertoni
Government Accountability Office: Education, Workforce, and Income Security
(202) 512-9889
Recommendations for Executive Action
Recommendation: To improve SSA's ability to ensure the accuracy and fairness of ALJ decisions, the agency should conduct ongoing analyses of ALJ decisions by race/ethnicity, as well as by other claimant groups (such as claimants with attorneys and nonattorneys, with translators, with low incomes, from certain regions and claimants who are men). In doing so, it should collect data on the types and sources of medical evidence in the claimant's file to better understand the agency's and attorney's role in the development of evidence.
Agency Affected: Social Security Administration
Status: In process
Comments: In June 2003, SSA convened an Agency workgroup tasked with developing recommendations on how we can collect meaningful data on race and ethnicity so we will have the information needed to analyze any adverse effects of our program policies and rules. SSA considered this a very complicated project that affects SSA's major programmatic systems and expects this will be a multi-year project. Upon completion of this systems project, SSA noted that it should be in a better position to respond to the GAO recommendations. At the same time, SSA has indicated that it discontinued its biennial evaluation of a representative sample of ALJ decisions for accuracy, which may diminish SSA's ability to conduct future tests for fairness of ALJ decisions by race and other factors. Despite the merit of this recommendation, as of September 2008, SSA reported that it had not taken action on this recommendation.
Recommendation: To improve SSA's ability to ensure the accuracy and fairness of ALJ decisions, the agency should conduct ongoing analyses of ALJ decisions by race/ethnicity, as well as by other claimant groups (such as claimants with attorneys and nonattorneys, with translators, with low incomes, from certain regions and claimants who are men). In doing so, it should analyze differences in support (accuracy) rates, in addition to differences in allowance decisions.
Agency Affected: Social Security Administration
Status: In process
Comments: In June 2003, SSA convened an Agency workgroup tasked with developing recommendations on how we can collect meaningful data on race and ethnicity so we will have the information needed to analyze any adverse effects of our program policies and rules. SSA considered this a very complicated project that affects SSA's major programmatic systems and expects this will be a multi-year project. Upon completion of this systems project, SSA noted that it should be in a better position to respond to the GAO recommendations. At the same time, SSA has indicated that it discontinued its biennial evaluation of a representative sample of ALJ decisions for accuracy, which may diminish SSA's ability to conduct future tests for fairness of ALJ decisions by race and other factors. Despite the merit of this recommendation, as of September 2008, SSA reported that it had not taken action on this recommendation.
Recommendation: To improve SSA's ability to ensure the accuracy and fairness of ALJ decisions, the agency should conduct ongoing analyses of ALJ decisions by race/ethnicity, as well as by other claimant groups (such as claimants with attorneys and nonattorneys, with translators, with low incomes, from certain regions and claimants who are men). In doing so, it should over-sample the selection of ALJ decisions by African-American claimants and, to the extent possible, other racial/ethnic groups to ensure that SSA has a sufficient number of cases to conduct analyses of ALJ decisions by race.
Agency Affected: Social Security Administration
Status: In process
Comments: In June 2003, SSA convened an Agency workgroup tasked with developing recommendations on how we can collect meaningful data on race and ethnicity so we will have the information needed to analyze any adverse effects of our program policies and rules. SSA considered this a very complicated project that affects SSA's major programmatic systems and expects this will be a multi-year project. Upon completion of this systems project, SSA noted that it should be in a better position to respond to the GAO recommendations. At the same time, SSA has indicated that it discontinued its biennial evaluation of a representative sample of ALJ decisions for accuracy, which may diminish SSA's ability to conduct future tests for fairness of ALJ decisions by race and other factors. Despite the merit of this recommendation, as of September 2008, SSA reported that it had not taken action on this recommendation.
Recommendation: To improve SSA's ability to ensure the accuracy and fairness of ALJ decisions, the agency should conduct ongoing analyses of ALJ decisions by race/ethnicity, as well as by other claimant groups (such as claimants with attorneys and nonattorneys, with translators, with low incomes, from certain regions and claimants who are men). In doing so, it should publish methods used and results as part of its biennial reporting on the findings of its disability hearings quality review process.
Agency Affected: Social Security Administration
Status: In process
Comments: In June 2003, SSA convened an Agency workgroup tasked with developing recommendations on how we can collect meaningful data on race and ethnicity so we will have the information needed to analyze any adverse effects of our program policies and rules. SSA considered this a very complicated project that affects SSA's major programmatic systems and expects this will be a multi-year project. Upon completion of this systems project, SSA noted that it should be in a better position to respond to the GAO recommendations. At the same time, SSA has indicated that it discontinued its biennial evaluation of a representative sample of ALJ decisions for accuracy, which may diminish SSA's ability to conduct future tests for fairness of ALJ decisions by race and other factors. Despite the merit of this recommendation, as of September 2008, SSA reported that it had not taken action on this recommendation.
Recommendation: To improve SSA's ability to ensure the accuracy and fairness of ALJ decisions, the agency should conduct ongoing analyses of ALJ decisions by race/ethnicity, as well as by other claimant groups (such as claimants with attorneys and nonattorneys, with translators, with low incomes, from certain regions and claimants who are men). In doing so, it should, if needed, take actions to correct and prevent any unwarranted differences in allowance and support rates among racial/ethnic and other claimant groups.
Agency Affected: Social Security Administration
Status: In process
Comments: In June 2003, SSA convened an Agency workgroup tasked with developing recommendations on how we can collect meaningful data on race and ethnicity so we will have the information needed to analyze any adverse effects of our program policies and rules. SSA considered this a very complicated project that affects SSA's major programmatic systems and expects this will be a multi-year project. Upon completion of this systems project, SSA noted that it should be in a better position to respond to the GAO recommendations. At the same time, SSA has indicated that it discontinued its biennial evaluation of a representative sample of ALJ decisions for accuracy, which may diminish SSA's ability to conduct future tests for fairness of ALJ decisions by race and other factors. Despite the merit of this recommendation, as of September 2008, SSA reported that it had not taken action on this recommendation.
Recommendation: To further ensure the accuracy and fairness of ALJ decisions for various claimant groups, SSA should conduct in-depth investigations of cases (e.g., case studies) to better understand differences in ALJ allowances for certain claimant groups, including claimants with and without an attorney. The results of these investigations should also be published in the biennial reports. If needed, SSA should take actions to correct and prevent any unwarranted differences in allowance rates among these claimant groups.
Agency Affected: Social Security Administration
Status: In process
Comments: In June 2003, SSA convened an Agency workgroup tasked with developing recommendations on how we can collect meaningful data on race and ethnicity so we will have the information needed to analyze any adverse effects of our program policies and rules. SSA considered this a very complicated project that affects SSA's major programmatic systems and expects this will be a multi-year project. Upon completion of this systems project, SSA noted that it should be in a better position to respond to the GAO recommendations. At the same time, SSA has indicated that it discontinued its biennial evaluation of a representative sample of ALJ decisions for accuracy, which may diminish SSA's ability to conduct future tests for fairness of ALJ decisions by race and other factors. Despite the merit of this recommendation, as of September 2008, SSA reported that it had not taken action on this recommendation.
Recommendation: To ensure that SSA uses a sample that is representative of all ALJ decisions in its quality assurance review, the agency should restructure its sampling process to incorporate cases that are appealed to SSA's Appeals Council in the quality assurance review sample. These appealed cases should be analyzed together with, rather than separate from, the rest of SSA's quality assurance sample.
Agency Affected: Social Security Administration
Status: Implemented
Comments: SSA addressed the shortcomings in its ongoing quality assurance process for ALJs, thereby improving its assessment of ALJ decision-making accuracy, by making appropriate changes to its data collection to ensure a representative sample. Specifically, the agency improved its collection of cases that are denied by ALJs and subsequently appealed to the Appeals Council--a subset that was formerly excluded from their sample. It then conducted statistical analyses of these cases, comparing the decision support rates for denied appealed cases with the support rates for cases that were not appealed to the Appeals Council. They also included the appealed cases with the other cases in the computation of ALJ accuracy rates. The results, as published in its biennial report, showed that the support rates of each group did not differ significantly.
Recommendation: In light of the methodological complexities associated with analyzing ALJ decisions, SSA should establish an advisory panel comprised of external experts in a range of disciplines--including statistics/econometrics, design methodology, law, medicine, vocational training, and disability--to provide leadership, oversight, and technical assistance with respect to conducting these and other quality assurance reviews of ALJ decisions.
Agency Affected: Social Security Administration
Status: In process
Comments: In June 2003, SSA convened an Agency workgroup tasked with developing recommendations on how we can collect meaningful data on race and ethnicity so we will have the information needed to analyze any adverse effects of our program policies and rules. SSA considered this a very complicated project that affects SSA's major programmatic systems and expects this will be a multi-year project. Upon completion of this systems project, SSA noted that it should be in a better position to respond to the GAO recommendations. At the same time, SSA has indicated that it discontinued its biennial evaluation of a representative sample of ALJ decisions for accuracy, which may diminish SSA's ability to conduct future tests for fairness of ALJ decisions by race and other factors. Despite the merit of this recommendation, as of September 2008, SSA reported that it had not taken action on this recommendation.
