This is the accessible text file for GAO report number GAO-08-1053 
entitled 'Social Security Disability: Management Controls Needed to 
Strengthen Demonstration Projects' which was released on September 26, 
2008. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Report to Congressional Requesters: 

United States Government Accountability Office: 

GAO: 

September 2008: 

Social Security Disability: 

Management Controls Needed to Strengthen Demonstration Projects: 

Social Security Disability: 

GAO-08-1053: 

GAO Highlights: 

Highlights of GAO-08-1053, a report to congressional requesters. 

Why GAO Did This Study: 

Since 1980, Congress has required the Social Security Administration 
(SSA) to conduct demonstration projects to test the effectiveness of 
possible changes to its Social Security Disability Insurance (DI) and 
Supplemental Security Income (SSI) programs that could decrease 
individuals’ dependence on benefits or improve program administration. 
However, in 2004, GAO reported that SSA had not used its demonstration 
authority effectively. This follow-up report assesses (1) how SSA has 
used its demonstration authority to test DI and SSI program changes and 
what information these efforts have yielded and (2) what steps SSA has 
taken to improve the planning and management of its demonstration 
projects. 

To do this, GAO reviewed documents related to SSA’s demonstration 
project management and the steps it took to implement the 
recommendations in the 2004 report, as well as the projects’ designs, 
evaluations, and costs. GAO also interviewed officials from SSA, its 
contractors and project sites, and disability experts 

What GAO Found: 

Over the last decade, SSA has initiated 14 demonstration projects under 
its authority to test possible DI and SSI policy and program changes; 
however, these projects have yielded limited information for 
influencing program and policy decisions. Of the 14 projects, SSA has 
completed 4, cancelled 5, and had 5 projects in progress as of June 
2008. In total, SSA spent about $155 million on its projects as of 
April 2008, and officials anticipate spending another $220 million in 
the coming years on those projects currently under way. Yet, these 
projects have yielded limited information on the impacts of the program 
and policy changes they were testing. SSA did not conduct impact 
evaluations for two of its completed projects, and intended to evaluate 
five other projects, but could not do so because significant challenges 
led SSA to cancel them. SSA officials believe the five projects 
currently under way will yield useful information, but it is too early 
to tell. 

Table: Demonstration Project Costs and Information Yielded Since 1998: 

DI (trust fund); 
Funds spent (1998 to April 2008): $48.3 million; 
Projected expenditures (April 2008 forward): $174 million. 

SSI (appropriations); 
Funds spent (1998 to April 2008): $107.2 million; 
Projected expenditures (April 2008 forward): $45.6 million. 

Total; 
Funds spent (1998 to April 2008): $155.5 million; 
Projected expenditures (April 2008 forward): $219.6 million. 

Information yielded; 
Funds spent (1998 to April 2008): Impact Information–2 of 14; 
Other information–3 of 14; No information–6 of 14; 
Projected expenditures (April 2008 forward): Information not yet 
available–4 of 14. 

Source: GAO analysis of SSA deminstration project estimated project 
cost and evaluation results as of April 2008. 

Note: Projects sum to greater than 14 because one project yielded 
impact information and other information. 

[End of table] 

SSA has taken steps to improve its demonstration projects but continues 
to lack management controls to ensure that the projects yield reliable 
information for making disability policy decisions. SSA has used 
methodological designs that GAO determined were strong or reasonable 
when assessed against professional research standards for 11 of its 14 
projects. SSA has also used external research professionals to work 
with the agency on the design, implementation, or evaluation of 12 of 
the projects, and appointed new program management to oversee its 
demonstration program. However, as of August 2008, SSA had not fully 
implemented the recommendations GAO made in 2004 and did not have 
written policies and procedures governing how it should review and 
operate its demonstration project program. Specifically, SSA does not 
have written policies and procedures for its managers and project 
officers to follow as they design, implement, and evaluate its 
demonstration projects. Absent such protocols, SSA did not always apply 
standard research practices, such as conducting pilot phases or 
obtaining sufficient stakeholder input, which led to data limitations 
and project cancellations. 

What GAO Recommends: 

GAO recommends that SSA establish written policies and procedures for 
managing and operating its projects consistent with standard research 
practices and internal control standards in the federal government. In 
response, SSA generally agreed with GAO’s recommendation and 
acknowledged that its guidance is a work in progress. 

To view the full product, including the scope and methodology, click on 
[http://www.gao.gov/cgi-bin/getrpt?GAO-08-1053]. For more information, 
contact Daniel Bertoni at (202) 512-7215 or bertonid@gao.gov. 

[End of section] 

Contents: 

Letter: 

Results in Brief: 

Background: 

SSA Has Used Its Demonstration Authority to Initiate Several Projects 
Since 1998, but Efforts Have Yielded Limited Information: 

SSA Has Taken Steps to Improve Its Projects but Continues to Lack 
Management Controls: 

Conclusions: 

Recommendation for Executive Action: 

Agency Comments and Our Evaluation: 

Appendix I: Scope and Methodology: 

Appendix II: Additional Statutory Criteria for the Benefit Offset 
National Demonstration Project: 

Appendix III: SSA's Use of Research Contractors and Universities on the 
Demonstration Projects: 

Appendix IV: Comments from the Social Security Administration: 

Appendix V: GAO Contact and Staff Acknowledgments: 

Related GAO Products: 

Tables: 

Table 1: Overview of Demonstration Projects Initiated Since 1998: 

Table 2: Total Estimated Costs for Demonstration Projects Currently in 
Progress: 

Table 3: Assessment of Demonstration Projects' Designs against 
Professional Research Standards and Statutory Requirements: 

Table 4: Reasons for Demonstration Projects' Cancellations: 

Table 5: Additional Statutory Criteria for the Benefit Offset National 
Demonstration Project: 

Figure: 

Figure 1: Projects Experienced Schedule Delays and Cancellations 
Associated with Organizational Changes: 

Abbreviations: 

BOND: Benefit Offset National Demonstration: 

DI: Social Security Disability Insurance: 

DISP: Disability and Income Security Program: 

DOL: Department of Labor: 

FICA: Federal Insurance Contributions Act: 

FMFIA: Federal Managers' Financial Integrity Act of 1982: 

HOPE: Homeless Projects Outreach and Evaluation: 

OPDR: Office of Program Development and Research: 

PMU: Pediatric Medical Unit: 

SSA: Social Security Administration: 

SSI: Supplemental Security Income: 

WIPA: Work Incentives Planning and Assistance: 

United States Government Accountability Office: 

Washington, DC 20548: 

September 26, 2008: 

The Honorable Michael R. McNulty: 
Chairman: 
The Honorable Sam Johnson: 
Ranking Member: 
Subcommittee on Social Security: 
Committee on Ways and Means: 
House of Representatives: 

As of October 2007, the Social Security Administration (SSA) reported 
that nearly 8.9 million disabled workers and their dependents were 
receiving benefits under the Social Security Disability Insurance (DI) 
program, while approximately 7.4 million individuals received federally 
administered Supplemental Security Income (SSI) payments. In our prior 
work, we have raised concerns about significant growth in the 
disability rolls and challenges in devising return-to-work policies. To 
address these challenges and encourage individuals capable of working 
to return to work, Congress has required SSA to conduct demonstration 
projects to test the effectiveness of possible program changes that 
could decrease individuals' dependency on benefits. Over the last 
decade, SSA has spent about $155 million dollars on these projects and 
anticipates that it will spend another $220 million on those currently 
under way. 

To conduct these demonstrations, Congress authorized SSA to waive 
certain program rules temporarily and use trust fund dollars and 
appropriated funds to finance their development. Congress also required 
that DI demonstration projects be of sufficient scope and conducted on 
a wide enough scale to ensure a thorough evaluation of the program or 
policy change under consideration. However, in 2004, we reported that 
SSA had not used its DI demonstration authority effectively and that 
the demonstration projects conducted in the late 1980s and early 1990s 
had little effect on SSA's and Congress's consideration of disability 
policy issues.[Footnote 1] We have also placed federal disability 
programs on our high-risk list--a list of federal programs that need 
attention and transformation. We did so, in part because of concerns 
about significant growth in the disability rolls and challenges in 
devising return-to-work policies.[Footnote 2] As you requested, we 
reviewed the adequacy of SSA's current and planned demonstration 
projects for both DI and SSI programs. Specifically, this report 
assesses (1) how SSA has used its demonstration authority to test DI 
and SSI program changes and what information these efforts have yielded 
and (2) what steps SSA has taken to improve the planning and management 
of its demonstration projects. 

To complete our work, we reviewed the demonstration projects that SSA 
planned, initiated, or terminated between the calendar years 1998 and 
2008 in order to identify their key characteristics, including their 
purpose, current status, and costs. To gain a better understanding of 
the components of SSA's demonstration projects, we conducted interviews 
with staff at sites where SSA implemented 9 of the 14 demonstration 
projects. We selected sites that included ongoing, cancelled, and 
completed projects, and represented diverse geographic regions 
throughout the United States. To determine what information these 
projects have yielded about the impacts of the policies and programs 
being tested and lessons learned, we collected and reviewed available 
evaluation information on each project. We collected data from SSA on 
the funds spent to date, projected future expenditures, and total costs 
for the demonstration projects, and determined that these data were 
sufficiently reliable for the purposes of this report. We also reviewed 
documents related to SSA's planning, implementation, and evaluation of 
its demonstration projects, including the development of the projects' 
methodological designs. We reviewed the projects' methodological 
designs against professional research standards for demonstration 
projects. These research standards were derived from GAO and recognized 
academic criteria for conducting evaluation research. Our Office of 
General Counsel reviewed these standards and found them consistent with 
the authorizing statutes' methodological requirements.[Footnote 3] 
Further, we identified relevant federal statutes governing DI and SSI 
demonstration activities and examined other reviews of SSA 
demonstration projects that we and SSA's Office of the Inspector 
General had conducted. For both objectives, we interviewed SSA 
officials, project officers, and research contractors that worked on 
the demonstration projects, officials from SSA's Office of Inspector 
General, and individuals from organizations that have a key role in 
federal disability policy. 

To address SSA's planning and management of its demonstration projects, 
we interviewed SSA management and staff about the agency's policies, 
guidance and procedures for developing and implementing demonstration 
projects, and collected supporting documentation where available. We 
assessed the adequacy of SSA's internal controls using the criteria in 
GAO's Standards for Internal Control in the Federal Government, which 
provide the overall framework for establishing and maintaining internal 
control in the federal government.[Footnote 4] 

We conducted this performance audit from November 2007 to August 2008 
in accordance with generally accepted government auditing standards. 
Those standards require that we plan and perform the audit to obtain 
sufficient, appropriate evidence to provide a reasonable basis for our 
findings and conclusions based on our audit objectives. We believe that 
the evidence obtained provides a reasonable basis for our findings and 
conclusions based on our audit objectives. Appendix I discusses our 
scope and methodology in further detail. 

Results in Brief: 

Over the last decade, SSA has initiated 14 projects under its 
demonstration authority to test both DI and SSI program changes--6 
related to DI, 6 related to SSI, and 2 examining both programs jointly-
-however, these projects have yielded limited information to date. Of 
the 14 projects, SSA has completed 4, cancelled 5, and had 5 projects 
in progress as of August 2008. In general, SSA has used its 
demonstration authority to conduct projects to consider changes to 
program administration and benefits counseling, as well as to test 
options for providing employment assistance and health benefits for 
beneficiaries. In total, SSA had spent about $155 million on these 
demonstration projects as of April 2008, and officials anticipate 
spending another $220 million in the coming years on those projects 
currently in progress. Yet, SSA's demonstration projects have yielded 
limited information about the impacts of the policies and programs 
being tested. We found that SSA did not conduct impact evaluations-- 
assessments of a project's effects compared to what would have happened 
in its absence--for two of its completed projects because the data it 
planned to use was not available. In addition, SSA intended to evaluate 
the impacts of five other projects but could not because significant 
challenges led it to cancel them in the early stages, and thus no data 
was available to assess those policies and programs being tested. 
Nonetheless, SSA was able to obtain some preliminary information from 
one of its cancelled projects. SSA also obtained some information from 
the two completed projects it evaluated, but the evaluations' findings 
had limitations. In one case SSA could not separate the effects of its 
project from the effects of a similar federal project that was also 
under way. SSA has also obtained some preliminary information from one 
of the five projects currently under way, and agency officials believe 
that these projects will ultimately yield useful information. However, 
it is too early to tell whether the information obtained will be used 
to inform changes to disability program policy. These projects are 
expected to continue until 2010 or later. 

SSA has taken steps to improve its demonstration projects but continues 
to lack written management controls to ensure that the projects yield 
reliable information about their impacts. For 11 of the projects, SSA 
has used experimental or quasi-experimental designs--methods commonly 
used by research professionals for demonstration projects--that we 
determined were strong or reasonable when assessed against professional 
research standards. SSA has also used external research professionals 
to work with the agency on the design, implementation, or evaluation of 
12 of the 14 projects, and appointed new management in 2007 to oversee 
the demonstration project program. The new management team subsequently 
conducted an internal review of the 10 demonstration projects that were 
under way at the time of their appointment and cancelled 5 that they 
determined were unlikely to yield reliable information because they 
faced significant limitations or challenges, or would have been 
duplicative of other ongoing research. SSA's decision to cancel these 
projects appears to have been data-driven and reasonable. However, as 
of August 2008, SSA had not fully implemented the recommendations we 
made in 2004 to help ensure the effectiveness of the demonstration 
projects. Further, the agency does not have written policies and 
procedures governing how it should review and operate its demonstration 
program. Specifically, SSA does not have written policies and 
procedures for its managers and project officers to follow as they 
design, implement, and evaluate its demonstration projects. Absent such 
protocols, SSA sometimes did not apply standard research practices, 
such as conducting pilot phases or obtaining sufficient stakeholder 
input, which led to data limitations and project cancellations. 

We are making a recommendation to SSA that is intended to improve the 
management of its demonstration project program by establishing written 
policies, procedures, and mechanisms for managing and operating the 
demonstration projects. These policies and procedures should be 
consistent with standard research practices and internal control 
standards in the federal government. In response to our draft report, 
SSA generally agreed with our recommendation and the need to develop a 
guidebook to assist its staff in the design, implementation, and 
evaluation phases of its demonstration projects. Furthermore, SSA 
considers its current guidebook a work in progress. SSA also stated 
that the agency has taken steps in recent years to address our prior 
recommendations. While we acknowledge SSA's efforts, we maintain that 
additional steps are needed to fully implement them. SSA's comments are 
reproduced in appendix IV. 

Background: 

Under the Social Security Act of 1935, as amended, SSA administers two 
federal disability programs--DI and SSI--intended to provide benefits 
to individuals with disabilities who are unable to work. The DI 
insurance program provides monthly cash benefits to individuals who 
have a Social Security work record and the amount of benefits is 
related to prior earnings.[Footnote 5] The DI program is funded 
primarily through a payroll tax required by the Federal Insurance 
Contributions Act (FICA) and is levied on most workers, and DI benefits 
are based on an individual's career earnings. The SSI program is a 
means-tested entitlement program that provides monthly benefits to 
aged, blind, or disabled individuals who have very limited income and 
assets. The SSI program is funded through general revenues. Unlike the 
DI benefit, the federal SSI benefit is a flat amount (adjusted for 
other income the individual may have) and is not related to prior 
earnings. 

During the 1970s, as the number of disability awards and costs were 
increasing significantly for the DI program, Congress enacted 
legislation providing various work incentives to encourage 
beneficiaries to return to work and, potentially, leave the benefit 
rolls. To further these efforts, in 1980, Congress provided SSA with 
the authority to conduct demonstration projects to evaluate the 
effectiveness of policy alternatives that could encourage both DI and 
SSI beneficiaries to re-enter the workforce.[Footnote 6] Under this 
authority, SSA can temporarily waive DI and SSI program rules, 
including rules regarding program eligibility and benefit 
administration, in order to test the effect certain program changes 
would have on beneficiaries' return-to-work rates and the size of the 
DI and SSI benefit rolls. Because Congress has historically granted SSA 
DI demonstration authority on a temporary basis, it is subject to 
periodic review and renewal. Since first providing this authority in 
1980, Congress has renewed it in 1986, 1989, 1994, 1999, and 2004. 
However, in 2004, Congress only extended SSA's DI demonstration 
authority through December 2005. As a result, SSA cannot initiate new 
DI demonstration projects but can continue those projects that were 
initiated on or before the December 2005 expiration date. However, SSA 
can continue to initiate demonstration projects under its SSI 
authority.[Footnote 7] In 2008, SSA requested that Congress reauthorize 
its DI demonstration authority, and a bill was introduced to do so. 

SSA's DI demonstration projects--unlike other SSA research activities-
-are paid for via the DI trust fund. Therefore, SSA is not required to 
obtain congressional approval for DI demonstration expenditures, 
although it is required to obtain approval from the Office of 
Management and Budget for an annual apportionment of the trust funds 
for these demonstrations. Unlike the DI projects, SSI demonstration 
projects are funded from SSA's overall congressional research 
appropriation. Although SSA's DI and SSI demonstration authorities are 
separate, the agency's disability demonstration projects are sometimes 
jointly authorized when they involve both DI and SSI beneficiaries and 
applicants. When a demonstration project is conducted jointly under the 
DI and SSI demonstration authorities, funding for the project is split 
between trust fund (i.e., DI) and appropriated (i.e., SSI) sources. 

SSA's Office of Program Development and Research (OPDR) provides 
program analysis in support of DI and SSI programs. As part of their 
responsibilities, OPDR--sometimes with the assistance of outside 
research organizations--identifies the requirements for individual 
disability program demonstration projects, including the basic 
objectives, scope, and methodological standards for these projects. 
OPDR project officers are primarily responsible for overseeing the 
projects to ensure that they meet SSA's technical and programmatic 
requirements. 

As we have previously reported, demonstration projects examining the 
impact of social programs aim to provide evidence of the feasibility or 
effectiveness of a new approach or practice and are inherently complex 
and difficult to conduct.[Footnote 8] Measuring outcomes, ensuring the 
consistency and quality of data collected at various site locations, 
establishing a causal connection between outcomes and program 
activities, and separating out the influence of extraneous factors can 
raise formidable technical and logistical problems. Although the 
legislation granting SSA its demonstration authority does not require 
the use of particular methodological approaches, SSA has historically 
recognized that the law's general requirement for its demonstration 
projects requires SSA to conduct its projects in a rigorous manner that 
provides the agency with a reliable basis for making policy 
recommendations. According to professional research standards, a 
rigorous study should include a clearly stated research question and 
methodology, including plans for data collection and evaluation, as 
well as appropriate controls to determine if a relationship exists 
between observed outcomes and the program change under examination (see 
app. I). 

As part of our prior work related to SSA DI demonstration authority we 
reviewed two DI demonstration projects that SSA conducted in the late 
1980s and early 1990s.[Footnote 9] At that time, we found that SSA had 
not used its demonstration authority to extensively evaluate a wide 
range of DI policy areas dealing with return to work and found that the 
demonstration projects had little impact on SSA's and Congress's 
consideration of DI policy issues. To facilitate close congressional 
oversight and provide greater assurance that SSA will make effective 
use of its authority, we recommended that SSA develop a formal agenda 
for its demonstration projects, establish an expert panel to guide the 
design and implementation of its demonstration projects, and establish 
formal processes to ensure full consideration of demonstration project 
results. We also identified several matters for Congress to consider, 
including continuation of DI demonstration authority on a temporary 
basis, establishment of additional reporting requirements for 
demonstrations, and clearer specification of the methodological and 
evaluation requirements of demonstrations. 

SSA Has Used Its Demonstration Authority to Initiate Several Projects 
Since 1998, but Efforts Have Yielded Limited Information: 

Over the last decade, SSA has initiated 14 demonstration projects to 
test policy and program changes, of which SSA has completed 4, 
cancelled 5, and had 5 projects in progress as of August 2008. In 
total, SSA had spent about $155 million on these demonstration projects 
as of April 2008, and officials anticipate spending another $220 
million in the coming years on those projects currently in progress. 
However, these projects have yielded limited information for 
influencing program and policy decisions. We found that SSA did not 
conduct impact evaluations for two of its completed projects and 
cancelled five projects prior to conducting formal evaluations; thus, 
limited information is available. 

Since 1998, SSA Has Initiated 14 DI and SSI Demonstration Projects in 
an Effort to Test Policy and Administrative Changes: 

Since 1998, SSA has initiated 14 projects under its demonstration 
authority to test both DI and SSI program changes--6 related to DI, 6 
related to SSI, and 2 examining both programs jointly (see table 1 for 
an overview of each project). As of April 2008, SSA spent $80.3 million 
on its completed projects, $7.1 million on cancelled projects, and 
$68.2 million on those currently in progress. 

Table 1: Overview of Demonstration Projects Initiated Since 1998: 

Project: Projects conducted under DI demonstration authority: 
Accelerated Benefits; 
Policy issues studied: Projects conducted under DI demonstration 
authority: Focused on provision of immediate health benefits to newly 
entitled DI beneficiaries to test whether this results in improved 
health and better return to work outcomes; 
Status: Projects conducted under DI demonstration authority: Initiated 
2004, in progress; 
Funds spent (as of April 2008): Projects conducted under DI 
demonstration authority: $3.6 million from the DI trust fund; 
Information available: Projects conducted under DI demonstration 
authority: Not yet available; 
Completion expected 2011. 

Project: Projects conducted under DI demonstration authority: Benefit 
Offset - 4 State Pilot; 
Policy issues studied: Projects conducted under DI demonstration 
authority: Focused on assessment of pilot of a benefit offset to inform 
the design of the Benefit Offset National Demonstration (BOND); 
Status: Projects conducted under DI demonstration authority: Initiated 
2003, in progress; 
Funds spent (as of April 2008): Projects conducted under DI 
demonstration authority: $4.5 million from the DI trust fund; 
Information available: Projects conducted under DI demonstration 
authority: Yes; 
preliminary process and outcome information available showing some 
increase in earnings by pilot participants and identifying operational 
challenges, such as income reporting, encountered when implementing the 
benefit offset; 
Completion expected 2009. 

Project: Projects conducted under DI demonstration authority: Benefit 
Offset National Demonstration (BOND); 
Policy issues studied: Projects conducted under DI demonstration 
authority: Focused on assessment of a benefit offset that would allow 
beneficiaries to remain eligible for DI benefits when earning above the 
Substantial Gainful Activity level while offsetting these benefits by 
$1 for every $2 in earnings; 
it would also assess benefit counseling in conjunction with the offset; 
Status: Projects conducted under DI demonstration authority: Initiated 
1999, in progress; 
Funds spent (as of April 2008): Projects conducted under DI 
demonstration authority: $6.7 million from the DI trust fund; 
Information available: Projects conducted under DI demonstration 
authority: Not yet available; Completion date unknown[A]. 

Project: Projects conducted under DI demonstration authority: 
California RISE (CA HIV/AI) Demonstration; 
Policy issues studied: Projects conducted under DI demonstration 
authority: Focused on provision of additional health and employment 
services for DI beneficiaries with Human Immunodeficiency Virus (HIV) 
or Auto-Immune (AI) deficiencies; 
Status: Projects conducted under DI demonstration authority: Initiated 
2003, cancelled 2007; 
Funds spent (as of April 2008): Projects conducted under DI 
demonstration authority: $2.1 million from the DI trust fund; 
Information available: Projects conducted under DI demonstration 
authority: No; project was not implemented. 

Project: Projects conducted under DI demonstration authority: Early 
Intervention Demonstration; 
Policy issues studied: Projects conducted under DI demonstration 
authority: Focused on provision of employment services to DI applicants 
as an alternative to the DI program; 
Status: Projects conducted under DI demonstration authority: Initiated 
2000, cancelled 2007; 
Funds spent (as of April 2008): Projects conducted under DI 
demonstration authority: $2 million from the DI trust fund; 
Information available: Projects conducted under DI demonstration 
authority: No; project was not implemented. 

Project: Projects conducted under DI demonstration authority: Mental 
Health Treatment Study; 
Policy issues studied: Projects conducted under DI demonstration 
authority: Focused on provision of mental health and employment 
services to DI beneficiaries with a primary impairment of schizophrenia 
or affective disorder; 
Status: Projects conducted under DI demonstration authority: Initiated 
2003, in progress; 
Funds spent (as of April 2008): Projects conducted under DI 
demonstration authority: $17.2 million from the DI trust fund; 
Information available: Projects conducted under DI demonstration 
authority: Not yet available; Completion expected 2011. 

Project: Projects conducted under SSI demonstration authority: 
Disability Program Navigator,; conducted with the Department of Labor 
(DOL); 
Policy issues studied: Projects conducted under DI demonstration 
authority: Focused on expansion of capacity in workforce investment 
system to serve people with disabilities; 
Status: Projects conducted under DI demonstration authority: Initiated 
2002, completed 2007; 
Funds spent (as of April 2008): Projects conducted under DI 
demonstration authority: $12 million from appropriations; 
Information available: Projects conducted under DI demonstration 
authority: No; project was not evaluated; SSA officials told us that 
DOL is pursuing a separate evaluation. 

Project: Projects conducted under SSI demonstration authority: Early 
Identification and Intervention Demonstration; 
Policy issues studied: Projects conducted under DI demonstration 
authority: Focused on provision of funding for programs that increase 
developmental screening and early identification of children with 
developmental delays and/or disabilities; 
Status: Projects conducted under DI demonstration authority: Initiated 
2004, cancelled 2007; 
Funds spent (as of April 2008): Projects conducted under DI 
demonstration authority: $50,000 from appropriations; 
Information available: Projects conducted under DI demonstration 
authority: No; project was not implemented. 

Project: Projects conducted under SSI demonstration authority: Florida 
Freedom Initiative; 
Policy issues studied: Projects conducted under DI demonstration 
authority: Focused on assessment of waivers of SSI program rules to 
foster greater self-sufficiency among SSI recipients in the state of 
Florida; 
Status: Projects conducted under DI demonstration authority: Initiated 
2003, completed 2007; 
Funds spent (as of April 2008): Projects conducted under DI 
demonstration authority: $100,000 from appropriations; 
Information available: Projects conducted under DI demonstration 
authority: No; project was not evaluated because too few participants 
enrolled to produce enough data for the evaluation. 

Project: Projects conducted under SSI demonstration authority: Homeless 
Outreach Projects and Evaluation; 
Policy issues studied: Projects conducted under DI demonstration 
authority: Focused on provision of services to people with disabilities 
who are chronically homeless, including assistance with filing 
disability benefit applications and accessing mainstream treatment and 
services; 
Status: Projects conducted under DI demonstration authority: Initiated 
2003, completed 2008; 
Funds spent (as of April 2008): Projects conducted under DI 
demonstration authority: $21.4 million from appropriations; 
Information available: Projects conducted under DI demonstration 
authority: Yes; impact information available showing that project 
participants received faster decisions from SSA about whether to 
allow/deny benefits; however, the contractor identified limitations 
with its evaluation because a similar program that another federal 
agency initiated after the SSA project began could have influenced the 
impact on the Homeless Outreach Projects and Evaluation participants. 

Project: Projects conducted under SSI demonstration authority: Interim 
Medical Benefits; 
Policy issues studied: Projects conducted under DI demonstration 
authority: Focused on provision of medical benefits to child and adult 
SSI applicants who are likely to be awarded benefits following the 
disability determination process; 
Status: Projects conducted under DI demonstration authority: Initiated 
2004, cancelled 2007; 
Funds spent (as of April 2008): Projects conducted under DI 
demonstration authority: $0; 
Information available: Projects conducted under DI demonstration 
authority: No; project was not implemented. 

Project: Projects conducted under SSI demonstration authority: 
Pediatric Medical Unit (PMU); 
Policy issues studied: Projects conducted under DI demonstration 
authority: Focused on provision of pediatric expertise to disability 
adjudicators to assist in making determinations on childhood disability 
cases; 
Status: Projects conducted under DI demonstration authority: Initiated 
2004, cancelled 2008; 
Funds spent (as of April 2008): Projects conducted under DI 
demonstration authority: $2.9 million from appropriations; 
Information available: Projects conducted under DI demonstration 
authority: Yes; some information from implementation experiences 
identified potential problems with childhood disability applications; 
PMU information could inform future research on interdisciplinary 
approaches to disability determination. 

Project: Projects jointly authorized under DI and SSI demonstration 
authorities: State Partnership Initiative; 
Policy issues studied: Projects conducted under DI demonstration 
authority: Focused on assessment of waivers of SSI benefit rules and 
other employment support interventions for DI and SSI beneficiaries 
provided in 12 states; 
Status: Projects conducted under DI demonstration authority: Initiated 
1998, completed 2006; 
Funds spent (as of April 2008): Projects conducted under DI 
demonstration authority: $46.7 million,; $8.5 million from the DI trust 
fund and $38.2 million from appropriations; 
Information available: Projects conducted under DI demonstration 
authority: Yes; some information from implementation experiences, as 
well as impact information showing that benefits counseling and 
employment services sometimes increased employment, but reduced 
earnings or did not affect them. However, the contractor identified 
some limitations with its impact evaluation because it is based on 
preliminary data. 

Project: Projects jointly authorized under DI and SSI demonstration 
authorities: Youth Transition Demonstration; 
Policy issues studied: Projects conducted under DI demonstration 
authority: Focused on provision of services to help youth SSI 
beneficiaries and other youth with disabilities make the transition 
from school to work and/or continue their education; it would also 
assess various waivers of disability program rules; 
Status: Projects conducted under DI demonstration authority: Initiated 
2003, in progress; 
Funds spent (as of April 2008): Projects conducted under DI 
demonstration authority: $36.2 million,; $3.7 million from the DI Trust 
fund and $32.5 million from appropriations; 
Information available: Projects conducted under DI demonstration 
authority: Not yet available; Completion expected 2014. 

Project: Total; 
Policy issues studied: Projects conducted under DI demonstration 
authority: [Empty]; 
Status: Projects conducted under DI demonstration authority: 4 
completed; 5 cancelled; 5 in progress; 
Funds spent (as of April 2008): Projects conducted under DI 
demonstration authority: $155.5 million; ($48.3 million from DI trust 
fund, $107.2 million from appropriations); 
Information available: Projects conducted under DI demonstration 
authority: Impact information-2 of 14; Other information-3 of 14; No 
information-6 of 14; 
Information not yet available-4 of 14. 

Source: GAO analysis of SSA's demonstration projects and budget data. 

Notes: In the column Information Available, projects sum to greater 
than 14 because 1 project yielded impact information and other 
information. In some cases, SSA officials provided us with estimated 
initiation dates because the agency had not maintained complete 
documentation on all demonstrations projects initiated since 1998. For 
the State Partnership Initiative, SSA officials were unable to verify 
the actual budget amounts spent in 1998 and 1999 for this project 
because SSA no longer maintains this information. 

[A] As of August 2008, SSA had not determined an expected completion 
date for the BOND project. 

[End of table] 

While SSA initiated 14 projects over the past 10 years, the agency has 
only completed 4 of them to date. These completed projects generally 
focused on reducing individuals' dependency on the SSI program by 
primarily testing program waivers and other changes in program 
administration, as outlined in its SSI demonstration 
authority.[Footnote 10] We also found that SSA cancelled five projects 
during this period, citing significant challenges that would have 
limited the agency's ability to obtain reliable information from them. 

SSA had five projects in progress as of August 2008. These projects 
generally addressed topics outlined in the authorizing legislation for 
DI demonstrations and included strategies to return individuals to work 
and reduce the growth of certain subgroups of beneficiaries. For 
example, the legislation required projects to test various incentives 
to increase DI beneficiaries work activity.[Footnote 11] In addition, 
the Ticket to Work and Work Incentives Improvement Act of 1999 provided 
demonstration authority for a benefit reduction, rather than complete 
benefit termination, when beneficiaries had earnings that exceeded a 
certain level.[Footnote 12] To address this provision, SSA initiated 
the Benefit Offset National Demonstration (BOND) shortly after passage 
of the statute. Another project in progress, the Mental Health 
Treatment Study, is focused on identifying strategies for providing 
mental health treatment and employment supports for certain DI 
beneficiaries with mental illnesses.[Footnote 13] As of April 2008, 
officials estimated that the total costs for the five projects 
currently in progress would be about $288 million--about $220 million 
more than the $68 million already expended (see table 2). 

Table 2: Total Estimated Costs for Demonstration Projects Currently in 
Progress: 

Project: Accelerated Benefits; 
Year of initiation: 2004; 
Completion date: 2011; 
Total estimated cost: $45 million. 

Project: Benefit Offset - 4 State Pilot; 
Year of initiation: 2005; 
Completion date: 2009; 
Total estimated cost: $8.5 million. 

Project: Benefit Offset National Demonstration (BOND); 
Year of initiation: 1999; 
Completion date: unknown[A]; 
Total estimated cost: $95 million. 

Project: Mental Health Treatment Study; 
Year of initiation: 2003; 
Completion date: 2011; 
Total estimated cost: $52.5 million. 

Project: Youth Transition Demonstration; 
Year of initiation: 2003; 
Completion date: 2014; 
Total estimated cost: $86.7 million; ($78 million from appropriations 
and $8.7 million from DI trust fund). 

Project: Total; 
Year of initiation: [Empty]; 
Completion date: [Empty]; 
Total estimated cost: $287.7 million; ($209.7 million from DI trust 
fund, $78 million from appropriations). 

Source: GAO analysis of SSA's demonstration projects and budget data. 

Notes: Expected total cost figures provided by SSA are estimates. 

[A] As of August 2008, SSA has not determined an expected completion 
date for the BOND project. 

[End of table] 

Demonstration Projects Have Yielded Limited Information to Influence 
Program and Policy Changes: 

Despite using its demonstration authority to examine various issues, 
SSA's demonstration projects have yielded limited information for 
influencing program and policy decisions. As required under its 
demonstration authority, SSA's demonstration projects should be 
conducted in such a way to permit a thorough evaluation of alternative 
methods under consideration.[Footnote 14] However, we found that SSA 
had not conducted impact evaluations--assessments of a project's 
effects compared to what would have happened in its absence--for two of 
its completed projects, the Disability Program Navigator and the 
Florida Freedom Initiative. Thus, no information about the impacts of 
the program and policy changes being tested was available for making 
decisions about disability policy. The Disability Program Navigator 
project, which SSA conducted with the Department of Labor (DOL), was 
not evaluated because the evaluation contractor could not meet SSA's 
data security requirements established after the project was already in 
progress, and thus could not access the necessary data.[Footnote 15] 
SSA developed a plan to evaluate the Florida Freedom Initiative after 
they became concerned about the state's evaluation plans. However, SSA 
did not conduct an evaluation because staff at the state level 
conducting the project did not enroll enough participants in the 
project to meet sample size requirements. Thus, there was not enough 
data available to conduct a reliable evaluation. 

Furthermore, SSA intended to evaluate the impacts of policies and 
programs being tested in five other projects but could not do so 
because the significant challenges those projects faced led SSA to 
cancel them in the early stages. Specifically, four of these projects 
were cancelled prior to implementation, and thus no data was available 
to conduct the evaluations of those policy and programs being tested. 
The other cancelled project--the Pediatric Medical Unit demonstration-
-was partially implemented but not evaluated because the project did 
not establish the comparison group needed for the analysis. The project 
also did not it enroll enough participants at some implementation sites 
to meet the sample size requirements needed to generate data for a 
reliable evaluation. However, SSA was able to obtain some preliminary 
information on how the project's strategy appeared to be working at two 
site locations and is considering how to use it. 

Although SSA did conduct evaluations for two of the completed projects-
-the Homeless Outreach Projects and Evaluation (HOPE) project and the 
State Partnership Initiative project--we found that these projects also 
yielded little information about the impacts of the strategies being 
tested because the reported evaluation results could not reliably 
demonstrate the projects' effects. For example, an outcome evaluation 
of the HOPE project showed that although disability program applicants 
assisted by the project received faster decisions from SSA about 
whether to allow or deny benefits, another federal agency initiated a 
similar project even though the HOPE project was under way. Therefore, 
SSA's evaluation results were weakened, in part because researchers 
could not separate the effects of the SSA project from the effects of 
the other federal project. While SSA did not obtain reliable impact 
evaluation results from this project, agency officials told us that 
they did obtain a great deal of information about the process of 
conducting this type of demonstration project. 

For the State Partnership Initiative, we found that SSA did conduct an 
impact evaluation when the project ended, but data available at that 
time were incomplete, and thus information about the impact of the 
project may not be a reliable indicator of the project's long-term 
effects. SSA's contractors recommended that a final evaluation be 
conducted once all the data were collected to assess whether the 
preliminary results were valid. However, SSA management chose not to 
pursue further evaluation because the preliminary results indicated 
that the project was not successful at increasing earnings enough to 
allow individuals returning to work to exit the rolls and no longer be 
dependent on disability benefits. Nonetheless, SSA's contractors and 
agency officials said that lessons learned from implementing the State 
Partnership Initiative have influenced the agency's subsequent approach 
to return beneficiaries to work. For example, SSA used the job 
descriptions of benefits planners, as well as data systems from this 
project, to design the agency's national Benefits Planning and Outreach 
program.[Footnote 16] 

SSA has also begun to obtain some information from one of the five 
projects currently under way. SSA has used preliminary results of the 
Benefit Offset - 4 State Pilot to aid in the design of the BOND 
project. Each of the four states conducting this pilot has provided an 
interim report to SSA detailing lessons learned from the implementation 
of this project. Because the pilot and BOND both test a benefit offset 
in conjunction with other DI program changes, SSA officials and the 
BOND project contractor believe that states' experiences implementing 
this pilot will help SSA identify and resolve operational issues before 
rolling BOND out nationally. In addition, the four states have 
conducted preliminary impact evaluations for the pilot project and 
expect to complete final evaluations once the project's implementation 
and data collection phases are over. 

SSA also plans to conduct impact evaluations of the other demonstration 
projects it had in progress as of August 2008. While they have the 
potential to yield reliable results, it is too early to tell whether 
they will ultimately be useful for informing DI and SSI policy and 
program changes. These projects address issues outlined in the 
demonstration authority statutes and disability programs more broadly, 
and SSA officials believe they will yield useful information. For 
example, SSA officials anticipate that the results of the Accelerated 
Benefits demonstration project could help policymakers determine 
whether to eliminate the 24 month waiting period for Medicare that DI 
beneficiaries encounter under current law. SSA officials also 
anticipate that demonstration projects in progress could yield key 
information on how to improve outcomes for certain subgroups of 
beneficiaries. For example, SSA officials said that the Youth 
Transition Demonstration, which targets young people with disabilities 
as they transition from school to work, could identify strategies for 
improving the self-sufficiency of these beneficiaries and thus reduce 
their dependence on the disability programs. Most of SSA's current 
demonstration projects are expected to continue until 2010 or later 
before generating final evaluation results that could inform changes to 
disability program policy. 

SSA Has Taken Steps to Improve Its Projects but Continues to Lack 
Management Controls: 

SSA has taken steps to improve its demonstration projects, in part by 
applying more rigorous methodologies than it did for the projects SSA 
initiated prior to 1998; however, it has not fully implemented GAO's 
recommendations from 2004 and does not have written policies and 
procedures in place to ensure that projects are routinely reviewed and 
effectively managed so that they yield reliable information about their 
impacts. As a result, some projects faced challenges, such as low 
participation rates or data collection problems, which were significant 
enough to hinder the agency's ability to evaluate the projects' impacts 
as planned. In addition, without comprehensive written policies and 
procedures governing how SSA manages and operates its demonstration 
program, the project objectives, designs, and evaluation plans may be 
impacted during times of organizational change. 

SSA Has Taken Steps to Improve Its Demonstration Projects: 

SSA has improved its demonstration projects by applying more rigorous 
methodologies than it did prior to 1998, contracting with professional 
researchers and appointing new management for the program. 
Specifically, SSA is applying more rigorous evaluation methodologies to 
the projects it has initiated since 1998 than it did to the projects 
initiated in the late 1980s and early 1990s. At the time of our prior 
report, SSA officials acknowledged that the limited rigor of those 
earlier projects reduced their usefulness and indicated that the agency 
had placed a new emphasis on ensuring that its projects going forward 
would be more rigorously designed.[Footnote 17] Of the 14 projects that 
SSA has initiated since 1998, 13 were early enough in the planning or 
design stages at that time to give SSA an opportunity to make such 
improvements.[Footnote 18] Since that time, SSA has completed much of 
the design work for its 14 projects and provided us with detailed 
design information for 12 of them, enabling us to assess the rigor of 
these projects' designs for our current review. 

Our current analysis shows that SSA did use more rigorous methodologies 
for the projects initiated over the last decade than for its earlier 
projects. SSA is now using methodologies known as experimental or quasi-
experimental designs, which are commonly used by research professionals 
conducting demonstration projects to estimate the impacts of program or 
policy changes. On the basis of our assessment, we determined that 11 
of the 12 projects' designs were strong or reasonable when assessed 
against professional research standards (see table 3).[Footnote 19] We 
compared each project's design against GAO and recognized academic 
criteria for conducting evaluation research, which were also consistent 
with statutory requirements that DI projects be generally sufficient in 
scope and planned in such a way to permit a thorough evaluation of the 
program or policy changes under consideration. We also determined that 
the projects currently under way could provide some reliable results if 
implemented and evaluated as designed. 

Despite this progress, we found that SSA did not always meet additional 
DI and SSI statutory requirements regarding the general applicability 
of the projects' results and the use of expert advice, respectively. 
The authorizing statute for DI demonstration projects requires that the 
results derived from the projects will obtain generally in the 
operation of the disability program.[Footnote 20] While one of the six 
DI projects, the BOND project, has been designed to yield nationally 
representative information about the impacts of the project, the 
statute does not require that the results be applicable to all DI 
beneficiaries nationwide.[Footnote 21] However, the results should 
apply to a larger group of beneficiaries than just those that 
participated in the demonstration project, and SSA may be able to apply 
the results from three other DI projects--the Accelerated Benefits 
demonstration project, the Benefit Offset - 4 State Pilot, and the 
Mental Health Treatment Study--more generally because it plans to 
implement and evaluate the projects in a consistent manner at multiple 
sites. In addition, one of the two jointly authorized projects--the 
State Partnership Initiative--did not yield generally applicable 
results because the projects were not implemented consistently across 
each state.[Footnote 22] The authorizing statute for SSI projects 
requires the Commissioner of SSA to obtain the advice and 
recommendations of specialists who are competent to evaluate the 
proposed projects as to the soundness of their design, the 
possibilities of securing productive results, the adequacy of resources 
to conduct the proposed research or demonstrations, and their 
relationship to other similar research or demonstrations already 
completed or in process before entering into a contract, grant, or 
cooperative agreement for the project.[Footnote 23] However, SSA 
obtained advice from experts for only two of the six SSI projects. 
Finally, SSA generally met other design criteria required by statute 
for the BOND project (see app. II).[Footnote 24] 

Table 3: Assessment of Demonstration Projects' Designs against 
Professional Research Standards and Statutory Requirements: 

Project: Projects conducted under DI demonstration authority: 
Accelerated Benefits (In progress); 
Methodology: Experimental; 
Assessment against professional research standards[A]: Strong; 
Assessment against additional DI statutory requirements[B]: Meets to 
some extent; 
Assessment against additional SSI statutory requirements[C]: NA. 

Project: Projects conducted under DI demonstration authority: Benefit 
Offset - 4 State Pilot[D[(IN] In progress); 
Methodology: Experimental; 
Assessment against professional research standards[A]: Reasonable; 
Assessment against additional DI statutory requirements[B]: Meets to 
some extent; 
Assessment against additional SSI statutory requirements[C]: NA. 

Project: Projects conducted under DI demonstration authority: Benefit 
Offset National Demonstration (In progress); 
Methodology: Experimental; 
Assessment against professional research standards[A]: Strong; 
Assessment against additional DI statutory requirements[B]: Meets; 
Assessment against additional SSI statutory requirements[C]: NA. 

Project: Projects conducted under DI demonstration authority: 
California RISE (CA HIV/AI) (Cancelled); 
Methodology: Experimental; 
Assessment against professional research standards[A]: Reasonable; 
Assessment against additional DI statutory requirements[B]: Does not 
meet; 
Assessment against additional SSI statutory requirements[C]: NA. 

Project: Projects conducted under DI demonstration authority: Early 
Intervention Demonstration (Cancelled); 
Methodology: Experimental; 
Assessment against professional research standards[A]: Strong; 
Assessment against additional DI statutory requirements[B]: Does not 
meet; 
Assessment against additional SSI statutory requirements[C]: NA. 

Project: Projects conducted under DI demonstration authority: Mental 
Health Treatment Study (In progress); 
Methodology: Experimental; 
Assessment against professional research standards[A]: Reasonable; 
Assessment against additional DI statutory requirements[B]: Meets to 
some extent; 
Assessment against additional SSI statutory requirements[C]: NA. 

Project: Projects conducted under SSI demonstration authority: 
Disability Program Navigator (Completed); 
Methodology: Unknown; 
Assessment against professional research standards[A]: Could not 
assess; 
Assessment against additional DI statutory requirements[B]: NA; 
Assessment against additional SSI statutory requirements[C]: Meets to 
some extent. 

Project: Projects conducted under SSI demonstration authority: Early 
Identification and Intervention Demonstration (Cancelled); 
Methodology: Unknown; 
Assessment against professional research standards[A]: Could not 
assess; 
Assessment against additional DI statutory requirements[B]: NA; 
Assessment against additional SSI statutory requirements[C]: Does not 
meet. 

Project: Projects conducted under SSI demonstration authority: Florida 
Freedom Initiative (Completed); 
Methodology: Quasi- experimental; 
Assessment against professional research standards[A]: Relatively weak; 
Assessment against additional DI statutory requirements[B]: NA; 
Assessment against additional SSI statutory requirements[C]: Does not 
meet. 

Project: Projects conducted under SSI demonstration authority: Homeless 
Outreach Projects and Evaluation (Completed); 
Methodology: Quasi-experimental; 
Assessment against professional research standards[A]: Reasonable; 
Assessment against additional DI statutory requirements[B]: NA; 
Assessment against additional SSI statutory requirements[C]: Does not 
meet. 

Project: Projects conducted under SSI demonstration authority: Interim 
Medical Benefits (Cancelled); 
Methodology: Quasi- experimental; 
Assessment against professional research standards[A]: Reasonable; 
Assessment against additional DI statutory requirements[B]: NA; 
Assessment against additional SSI statutory requirements[C]: Does not 
meet. 

Project: Projects conducted under SSI demonstration authority: 
Pediatric Medical Units (Cancelled); 
Methodology: Experimental; 
Assessment against professional research standards[A]: Reasonable; 
Assessment against additional DI statutory requirements[B]: NA; 
Assessment against additional SSI statutory requirements[C]: Meets. 

Project: Projects jointly authorized under DI and SSI demonstration 
authorities: State Partnership Initiative[D,E[(completed); 
Methodology: Mixed-experimental; 
quasi-experimental; 
Assessment against professional research standards[A]: Mixed-some 
strong; 
some reasonable; 
Assessment against additional DI statutory requirements[B]: Does not 
meet; 
Assessment against additional SSI statutory requirements[C]: Meets to 
some extent. 

Project: Projects jointly authorized under DI and SSI demonstration 
authorities: Youth Transition Demonstration[E[(IN] In progress); 
Methodology: Experimental; 
Assessment against professional research standards[A]: Strong; 
Assessment against additional DI statutory requirements[B]: Meets to 
some extent; 
Assessment against additional SSI statutory requirements[C]: Meets. 

Source: GAO analysis of SSA's demonstration projects' designs. 

[A] Appendix I discusses professional research standards in further 
detail. 

[B] The Social Security Act § 234(b) requires that in addition to the 
demonstration projects being of sufficient scope and carried out on a 
wide enough scale to permit a thorough evaluation of the alternative 
methods under consideration, the projects should also give assurance 
that the results derived from the projects obtain generally in the 
operation of the disability program. 

[C] The Social Security Act § 1110(a)(1) requires the Commissioner of 
SSA to obtain the advice and recommendations of specialists who are 
competent to evaluate the proposed projects as to the soundness of 
their design, the possibilities of securing productive results, the 
adequacy of resources to conduct the proposed research or 
demonstrations, and their relationship to other similar research or 
demonstrations already completed or in process. 

[D] Multiple evaluations were conducted for this demonstration project. 

[E] This project must meet the requirements specified in both the DI 
and SSI statutes because it is authorized under both of them. 

[End of table] 

Evaluation methodologies: 
Experimental designs involve random assignment of study participants to 
either a treatment group or a control group. The treatment group is 
subjected to the new program or policy, and the control group is not. 
The strength of the experimental design is in its assurance that those 
who experience the treatment are like those that do not experience the 
treatment in all important ways, except for the difference of receiving 
the treatment itself. Thus, after the study is over, researchers can 
usually attribute observed differences in outcomes between the 
treatment and control groups to the new policy or program. Research 
experts generally consider experimental designs to be the "gold 
standard" for impact evaluations. However, they may not always be 
feasible or may not necessarily provide the optimal approach for 
conducting a demonstration because of various implementation or ethical 
considerations. Quasi-experimental methods-- reasonable but less 
rigorous alternatives to experimental designs-- include study 
participants that receive a treatment, but do not involve the use of 
randomized control groups and instead rely on the nonrandom selection 
of a comparison group of nonparticipants with characteristics similar 
to the study participants. Quasi-experimental methods may instead be 
preferable or may need to be used in conjunction with experimental 
methods to produce the most rigorous results. For instance, quasi-
experimental methods may be preferred when there is a chance that a 
randomized experimental approach will not remain intact, such as when 
participants in the control group are likely to receive elements of the 
policy or program under evaluation. However, because quasi-experiments 
do not randomly assign participants to treatment and control groups, 
the assurance that the two groups are actually similar in all critical 
ways except for receiving the treatment is missing, and they cannot 
establish cause and effect between the program under study and the 
observed outcomes. Nonetheless, they can be used to determine whether a 
relationship exists between the strategy being tested and the observed 
outcomes. 

To further improve the demonstration projects' planning and 
methodological rigor, SSA has used external research professionals to 
work with the agency on the design, implementation, or evaluation of 12 
of the 14 projects. SSA officials have acknowledged the need for 
additional expertise to design and implement methodologically rigorous 
demonstration projects.[Footnote 25] Thus, SSA has awarded, or planned 
to award, contracts and cooperative agreements to research consultants 
and universities with such expertise to evaluate 12 of its 14 projects 
(see app. III). In nine cases, these researchers also worked on the 
design or implementation of the projects. For example, for the 
Accelerated Benefits demonstration project, an SSA research contractor 
also designed how and where the project would be conducted and managed 
its implementation so that the data needed for the evaluation it plans 
to conduct will be available. We also found that SSA and most of these 
researchers communicated regularly when collaborating on these projects 
and researchers submitted monthly or quarterly progress reports to SSA, 
which included information on expenditures, progress, and areas of 
concern that needed to be addressed. 

SSA also appointed new program management in 2007. Since that time, the 
new management team has conducted an internal review of the 10 
demonstration projects that were under way at the time of their 
appointment. SSA officials told us that all projects underwent a 
thorough review that was conducted by the Acting Associate Commissioner 
for Program Development and Research and others with appropriate 
expertise. Documents we obtained indicate that the review identified 
the projects' strengths, weaknesses, and whether they were likely to 
yield reliable, useful results. For example, SSA considered whether a 
project's sample size and site selection were appropriate, if it had 
been implemented in accordance with its design, and if it faced any 
challenges that would prevent researchers from conducting a rigorous 
evaluation of its results. SSA concluded that five of its projects 
would continue, expecting that they were likely to yield reliable 
impact information; these five projects are currently in 
progress.[Footnote 26] However, the agency did need to make significant 
changes to strengthen one of the five projects' designs--the BOND 
project. At an earlier point in the design phase, SSA expanded the BOND 
project's scope to include multiple components in addition to the 
benefit offset, such as a health benefits package. The new management 
team subsequently determined that the cost estimates and program 
complexity associated with several of those components raised questions 
about the feasibility of implementing the project and significantly 
scaled back the scope of the study. 

For the other five projects the new management was reviewing, SSA 
determined they faced significant limitations or challenges--such as 
poorly chosen implementation sites and low participation--that made it 
highly unlikely for them to obtain reliable results or would have been 
duplicative of other ongoing research (see table 4). Thus, SSA 
cancelled those projects in 2007 and 2008. Although the only 
information SSA has obtained from these projects are some lessons 
learned from the Pediatric Medical Unit project, the agency projected 
that it would have spent another $82 million had it not canceled 
them.[Footnote 27] Based on the information SSA provided us about the 
challenges facing the projects and the expected future costs of 
conducting them, its decisions to cancel the five projects appear to 
have been data-driven and reasonable. 

Table 4: Reasons for Demonstration Projects' Cancellations: 

Project: California RISE (CA HIV/AI) Demonstration; 
Program focus: DI; 
Phase of project when cancelled: Design; 
Amount spent prior to cancellation: $2.1 million; 
Reasons SSA cited for cancellation: Reliable evaluation results were 
unlikely because similar services were already available in the state 
selected for implementation, potentially introducing control group 
contamination and problems recruiting enough participants for the 
study. Also, nationally representative results were unlikely because of 
unique conditions in the state; Significant coordination problems also 
existed between the implementation and evaluation components of the 
project. 

Project: Early Identification and Intervention Demonstration; 
Program focus: SSI; 
Phase of project when cancelled: Solicitation of proposals; 
Amount spent prior to cancellation: $50,000; 
Reasons SSA cited for cancellation: Reliable evaluation results were 
unlikely because the Request for Proposal for this study did not 
require the use of a rigorous evaluation methodology; The project also 
had little connection to SSA's mission, and there were competing needs 
for SSA's research resources. 

Project: Early Intervention Demonstration; 
Program focus: DI; 
Phase of project when cancelled: Design; Amount spent prior to 
cancellation: $2 million; 
Reasons SSA cited for cancellation: This project was initiated as a 
stand-alone demonstration project but was incorporated into the BOND 
project in 2005. It added significant complexity to BOND and became 
very costly to carry it out. 

Project: Interim Medical Benefits; 
Program focus: SSI; 
Phase of project when cancelled: Prior to solicitation of proposals; 
Amount spent prior to cancellation: $0; 
Reasons SSA cited for cancellation: This project was very similar to 
the Accelerated Benefits demonstration project, and information from it 
would have been duplicative. 

Project: Pediatric Medical Unit (PMU); 
Program focus: SSI; 
Phase of project when cancelled: Implementation; 
Amount spent prior to cancellation: $2.9 million; 
Reasons SSA cited for cancellation: While the design for this project 
was reasonable in that it specified the use of an experimental design 
and establishment of a comparison group, when SSA implemented the 
project, the comparison group was not established. This prevented SSA 
from obtaining the level of rigor it wanted for this project; 
Insufficient sample size also made reliable evaluation results unlikely 
because some of the participating Disability Determination Services 
were not referring enough cases to the PMU for review. In addition, the 
low referral rates made the cost per participant very high. 

Project: Total; 
Program focus: [Empty]; 
Phase of project when cancelled: [Empty]; 
Amount spent prior to cancellation: $7.05 million; ($4.1 million from 
DI trust fund, $2.95 million from appropriations); 
Reasons SSA cited for cancellation: [Empty]. 

Source: GAO analysis of SSA's demonstration projects and budget data. 

[End of table] 

Further, SSA consolidated its research expertise by merging the Office 
of Disability and Income Security Programs (DISP) with SSA's Office of 
Policy in February 2008, creating the Office of Retirement and 
Disability Policy. As of June 2008, each office's research unit remains 
intact and no formal organizational changes were made to the 
demonstration program, but agency officials told us that the merger has 
facilitated communication and strengthened relationships between 
researchers within the agency.[Footnote 28] For example, experts from 
the former Office of Policy's research unit routinely review and 
provide input on the demonstration projects' designs and evaluations. 

SSA Continues to Lack Policies, Procedures, and Other Mechanisms to 
Ensure Projects Yield Reliable Information: 

While SSA is taking steps to generally improve its demonstration 
projects' designs and address specific project limitations, it does not 
have policies, procedures, and mechanisms to ensure that demonstration 
projects will yield reliable information about the impacts of the 
programs they are testing. According to internal control standards in 
the federal government, federal agencies should have policies, 
procedures, and mechanisms in place to provide reasonable assurance 
that a program's objectives are being achieved.[Footnote 29] However, 
we found that, as of August 2008, SSA had not fully implemented the 
recommendations we made in 2004 to help ensure the effectiveness of the 
demonstration projects.[Footnote 30] Specifically, SSA continues to 
lack: 

* a formal, comprehensive, long-term agenda for conducting 
demonstration projects; 

* an expert panel to review and provide regular input on the design and 
implementation of demonstration projects from the early stages of a 
project through its final evaluation; and: 

* a formal process for fully considering the potential policy 
implications of its demonstration projects' results and fully apprising 
Congress of the results and their policy implications. 

We did find that SSA has developed a limited research agenda for its 
projects that lacks basic details about the projects, including their 
objectives, schedules, and costs. The agenda was also developed without 
broadly consulting key internal and external stakeholders to obtain 
their input. SSA officials told us that they do not plan to update the 
agenda to reflect that some of the projects have been cancelled. In 
contrast, other federal agencies that conduct research have published 
much more detailed research agendas and update them regularly. For 
example, the Department of Education's National Institute on Disability 
and Rehabilitation Research publishes a 5-year plan that outlines 
priorities for rehabilitation research, demonstration projects, 
training and related activities, explains the basis for such 
activities, and publishes the plan for public comment before submitting 
it to Congress. 

In addition, agency officials said that SSA planned to continue using 
experts only on an ad hoc basis, citing how potential conflicts of 
interest could pose challenges to serve on an expert panel as we had 
recommended.[Footnote 31] SSA established ad hoc panels or consulted 
with outside experts for 8 of its 14 demonstration projects, including 
4 of those that are currently in progress. Many of SSA's project 
officers and contractors reported that this was a positive experience, 
and SSA management has told us that they plan to continue using experts 
in the future. However, under this approach SSA may miss the 
opportunity to obtain advice more broadly on the demonstration program. 
For example, the panels and experts that SSA used were brought on board 
after the agency decided to initiate the demonstration projects; 
therefore, they were not in position at an early enough point in time 
to help SSA consider whether a demonstration project or an alternative 
research approach was, in fact, the best way to meet the agency's 
needs. Furthermore, SSA did not regularly seek input from the Social 
Security Advisory Board or the National Council on Disability, which 
both play key roles in federal disability policy and could be in a 
position to advise SSA more broadly on the demonstration 
projects.[Footnote 32] 

In addition, our prior work found that SSA had not sufficiently 
provided information on the status and results of its demonstration 
projects to Congress. In our current review, we found that SSA 
regularly submits annual reports about the DI demonstration projects to 
its congressional oversight committees. While SSA meets its statutory 
requirements by submitting these reports, the information in them is 
generally limited to descriptions of the projects' objectives and the 
dates of upcoming milestones. Similarly, the information that SSA 
reports about its SSI demonstration projects is limited to brief 
descriptions in the agency's annual congressional budget 
justifications. Key information that could help Congress monitor the 
progress of the demonstration projects--including project costs, 
potential risks and obstacles to their success, or the policy 
implications of their results--was rarely included in the annual 
reports or budget justifications. However, SSA officials also told us 
that they sometimes share additional information with Congress about 
the demonstration projects. For example, SSA officials told us they met 
with congressional committees in October and November 2007 to share 
information about its design plans for the BOND project. 

In addition to not fully addressing our prior recommendations, SSA does 
not have written policies and procedures governing how it should review 
and operate its demonstration program. Accordingly, SSA does not have a 
written policy requiring SSA management to review its project officer's 
demonstration projects on a regular basis. Standards for internal 
control in the federal government state that managers should compare a 
program's actual performance against expected targets and analyze 
significant differences.[Footnote 33] Although the new program 
management team reviewed each of the demonstration projects at the time 
of their appointment, SSA does not have a written policy requiring such 
a review process periodically throughout the design, implementation, 
and evaluation phases of each project. SSA's lack of a policy to 
systematically review each project on a periodic basis contributed to 
problems sometimes going undetected after the projects were 
implemented, and they did not yield the data needed for their 
evaluations. For example, because SSA was not actively involved in 
implementing or monitoring the Florida Freedom Initiative demonstration 
project, it was not in a position to take steps to ensure that the 
project proceeded as planned, and staff at the state level failed to 
enroll enough participants to generate data for the evaluation that SSA 
planned. Therefore, no evaluation was conducted for this project. 

In addition, we found that SSA does not have written procedures for its 
project officers to follow as they design, implement, and evaluate its 
demonstration projects. Such procedures could be used to ensure that 
standard research practices, such as conducting pilot phases and 
including internal and external stakeholders, are applied when planning 
and implementing the demonstration projects. Specifically, SSA does not 
require staff to regularly use pilots to test projects' underlying 
assumptions, operational logistics, or feasibility before they are 
implemented. As a result, SSA planned or conducted pilots or phased 
implementations for only 8 of its 14 projects, although GAO criteria 
for evaluation research emphasize the importance of conducting pilots, 
as they are a critical test of a project's design.[Footnote 34] At 
least four of the projects that did not include pilot phases 
experienced the type of logistical challenges that pilots are intended 
to identify. For example, the Homeless Outreach Projects and Evaluation 
demonstration project experienced start-up delays because of 
compatibility problems between the contractor's online data collection 
system and the computer systems at the 41 sites where the project was 
implemented. If SSA had conducted a pilot phase for this project, it 
may have detected these issues at a smaller number of sites and 
developed a plan to resolve them prior to implementing the full 
project. 

In addition, SSA does not have written procedures directing its project 
officers or contractors to routinely consult with internal and external 
stakeholders when planning the demonstration projects. We found that at 
least 11 of the 14 projects experienced challenges or limitations 
because SSA had not obtained sufficient input from, or coordinated 
effectively with, internal and external stakeholders. For example, SSA 
officials told us that they were aware that the Benefit Offset - 4 
State Pilot project and the Benefit Offset National Demonstration 
project required a change to internal SSA processes for calculating DI 
benefits but did not coordinate with key internal stakeholders early on 
to determine how to make this change, and no systematic process was put 
in place. As a result, SSA had to calculate payments by hand for the 
Benefit Offset - 4 State Pilot, and the BOND project's implementation 
has been delayed while SSA now works with its internal stakeholders to 
determine how to make the needed changes. Furthermore, SSA officials, 
contractors, and representatives from the various demonstration 
projects' implementation sites told us that there was little input from 
internal stakeholders and that internal coordination problems existed 
on at least seven of these projects, including three of those that were 
cancelled. We also found that lack of coordination or communication 
with external stakeholders led to challenges in at least seven of these 
projects, including four of those that were cancelled. For example, 
coordination problems between the two contractors for the California 
Rise project resulted in the project's components being designed in 
isolation from each other, which complicated the evaluation plans and 
eventually contributed to the project's cancellation. In addition, SSA 
did not always include the prospective implementation sites in the 
planning and design of its projects, although they could have provided 
insight into the feasibility and logistical requirements of the 
project. 

While SSA has periodically provided direction informally to its project 
officers, some project officers told us that more formal guidance would 
have helped them to better understand what steps were necessary and 
expected, and we concluded from our discussions with others that such 
guidance would have been helpful. SSA recognizes that the program's 
lack of written procedures is a limitation and is drafting a guidebook 
on standard research practices for staff to follow when planning and 
designing demonstration projects. Although an SSA official told us that 
this document is a work in progress, it appears that the handbook will 
include key procedures for designing a project, such as identifying 
what data is needed for the evaluation and how it should be obtained. 
It also directs project officers to assemble a team for project 
development that includes staff from across the agency who will be able 
to collaborate and provide the input necessary to address the multiple 
components of the demonstration project. The draft guidebook also 
includes provisions for assembling a research panel composed of 
internal and external experts. This panel would review proposed 
research projects and identify those that present the most promising 
opportunities, taking into consideration the extent to which prior 
research has already addressed the topic. However, in its current form, 
the guidebook provides little direction for the implementation or 
evaluation phases of the demonstration projects, and SSA officials had 
not finalized it as of May 2008. 

Without comprehensive written policies and procedures governing how SSA 
manages and operates its demonstration programs, the project 
objectives, designs, and evaluation plans may be impacted during times 
of organizational change. Because government operating conditions 
continually change, agencies should have mechanisms in place to 
identify and address any special risks arising from such changes, 
especially those caused by hiring new personnel to occupy key positions 
in the agency.[Footnote 35] However, because SSA lacks mechanisms such 
as a standing advisory panel or written policies and procedures to 
provide continuity for its demonstration program when organizational 
changes occur, it cannot guarantee that institutional knowledge about 
the projects is shared or that the impacts of such changes are 
considered as the projects progress. SSA has experienced several 
organizational changes since the first of these projects was initiated 
in 1998, which have included the demonstration program's relocation 
from the Office of Policy to the Office of Disability and Income 
Support Programs in 2002, program management's replacement in 2007, and 
the Office of Policy and Office of Disability and Income Support 
Program's merger in 2008. At least six of SSA's projects experienced 
schedule delays and cancellations, in part because newly appointed 
officials made significant changes to some projects or determined that 
because others faced significant limitations or potential challenges it 
was not in the agency's interests to continue them (see fig. 1). While 
certain management actions may be reasonable, SSA's lack of written 
policies and procedures governing how such steps are taken leaves 
current and future projects vulnerable to disruption. For example, we 
found that the Benefit Offset National Demonstration project is still 
in the design phase after 9 years, during which time it has gone 
through numerous revisions by different program managers and was moved 
from one office to another. As of August 2008, an interagency working 
group was determining how to implement this administratively complex 
project. SSA has put the project's implementation and evaluation on 
hold until this issue has been resolved. 

Figure 1: Projects Experienced Schedule Delays and Cancellations 
Associated with Organizational Changes: 

This figure is a chart showing projects which experienced schedule 
delays and cancellations associated with organizational changes. 

[See PDF for image] 

Source: GAO analysis of SSA data. 

[A] As of August 2008, SSA had not determined the expected 
implementation or evaluation date for the BOND project. 

[End of figure] 

Conclusions: 

For over two decades, SSA has had the authority to conduct 
demonstration projects to test strategies that could address the 
challenges posed by the low rate of return into the workforce and the 
growing number of applicants and disabled beneficiaries. However, the 
agency has missed opportunities to identify ways to modernize DI and 
SSI programs and policies because it has generally not conducted the 
demonstration projects effectively. Since 1998 alone, SSA has spent 
over $150 million on 14 demonstration projects; yet these projects have 
generated limited information about the impacts of the strategies that 
were being tested. Although many of these projects were generally well 
designed, SSA's lack of written polices, procedures, and mechanisms for 
managing and operating these projects is one of the key reasons the 
projects SSA has completed and cancelled to date were generally not 
implemented and evaluated in a way that yielded reliable, data-driven 
impact information. As a result, Congress, SSA, and other organizations 
that play a critical role in federal disability policy continue to lack 
key information about important issues, such as the impact of providing 
health care or employment supports to DI and SSI beneficiaries as a 
means to help beneficiaries achieve self sufficiency and leave the 
rolls. 

SSA's five demonstration projects currently in progress have the 
potential to identify solutions to some of SSA's challenges. However, 
if SSA does not address the limitations in the way it manages and 
operates demonstration projects, these projects may encounter the same 
challenges that past projects have faced, and SSA could again have 
little to show for its efforts. Given that SSA estimates it will spend 
approximately $220 million over the next several years to complete 
these projects, it is important that steps be taken to make the 
projects less vulnerable to the challenges and organizational changes 
they could encounter in the future. SSA's actions to review its 
demonstration projects and to begin drafting guidance to help staff 
better plan and design its projects are encouraging first steps. As SSA 
officials work toward finalizing this guidance, it is also necessary 
for the agency to address its lack of written policies and procedures 
for managing and operating its projects during their implementation and 
evaluation phases. SSA should also take action to fully implement the 
recommendations we made in 2004. Implementing those recommendations by 
fully developing its research agenda, establishing an expert panel to 
advise it about the projects on a regular basis, and improving its 
communications with Congress could help improve the effectiveness and 
transparency of its demonstration program going forward. 

Recommendation for Executive Action: 

To improve SSA's management of its demonstration projects we recommend 
that the Commissioner of Social Security direct the Deputy Commissioner 
for the Office of Retirement and Disability Policy to: 

* establish written policies, procedures, and mechanisms for managing 
and operating its demonstration projects that are consistent with 
standard research practices and internal control standards in the 
federal government, including those for coordinating with internal and 
external stakeholders and sharing information with Congress. 

Agency Comments and Our Evaluation: 

We obtained written comments on a draft of this report from SSA, which 
are reproduced in appendix IV. We incorporated technical comments we 
received throughout the report, as appropriate. 

In response to our draft report, SSA generally agreed with our 
recommendation and acknowledged the need to develop a guidebook to 
assist its staff in the design, implementation, and evaluation phases 
of its demonstration projects. SSA further discussed its existing 
processes and written procedures for managing and reviewing its 
programs, including the demonstration project program. While 
noteworthy, we continue to believe SSA needs to establish written 
procedures that incorporate professional research standards and 
internal control mechanisms for ensuring that the demonstration 
projects yield reliable information about their impacts. SSA considers 
its current guidebook a work in progress. Further, SSA stated that the 
agency has taken steps in recent years to address our prior 
recommendations. While we acknowledge SSA's efforts, the agency needs 
to take additional steps to fully implement them. For example, SSA 
continues to lack a standing expert panel to review and provide regular 
input on the demonstration projects overall, even though it has 
employed subject matter experts for some of its demonstration projects. 
Although SSA officials have raised concerns about the difficulty of 
establishing an expert panel because research contractors serving on 
the panel would be precluded from working on individual projects, we 
continue to believe that such a panel could be established. As 
previously recommended, this panel should also include SSA's key 
research personnel and outside disability experts in addition to 
researchers. 

We are sending copies of this report to the Commissioner of SSA and 
other interested parties. We will also make copies available to others 
upon request. In addition, the report will be available at no charge on 
the GAO Web site at [hyperlink, http://www.gao.gov]. 

If you or your staffs have any questions concerning this report, please 
contact me at (202) 512-7215. Contact points for our offices of 
Congressional Relations and Public Affairs may be found on the last 
page of this report. GAO staff who made key contributions to this 
report are listed in appendix V. 

Signed by: 

Daniel Bertoni: 

Director, Education, Workforce, and Income Security: 

[End of section] 

Appendix I: Scope and Methodology: 

To determine how SSA has used its current demonstration authority, we 
reviewed legislation authorizing the Social Security Administration 
(SSA) to conduct Disability Insurance (DI) and Supplemental Security 
Income (SSI) demonstration projects, prior GAO report and SSA's Office 
of the Inspector General (OIG) reports, and reports to Congress on the 
demonstration authority. We reviewed documents from SSA and from 
publicly available sources, including the Federal Register and reports 
by other research organizations. We interviewed current and former SSA 
officials in the Office of Disability and Income Security 
Programs,[Footnote 36] specifically, the Office of Program Development 
and Research (OPDR) and the Office of Research, Evaluation and 
Statistics who had responsibility for, or involvement in, the 
demonstration projects. We also interviewed research contractors that 
worked on the demonstration projects and individuals from organizations 
that have a key role in federal disability policy. In addition, we 
interviewed staff from sites where SSA implemented 9 of the 14 
demonstration projects. We selected sites that included ongoing, 
cancelled and completed projects, and represented diverse geographic 
regions throughout the United States. 

To better understand SSA's DI and SSI demonstration projects, we 
reviewed SSA documents describing the purpose, design, and status for 
all demonstration projects that were in progress, completed, or had 
been cancelled prior to completion. These documents included requests 
for proposals, project plans and schedules, interim or final project 
reports, and reports to Congress from 1998 to 2008. We used these 
documents to identify key characteristics of the projects, including 
the policy issues addressed, use of contractors, the authority used to 
conduct each project, project timelines, and information resulting from 
each project. We also examined the issues SSA tested or was in the 
process of testing in its demonstration program. We reviewed the 
authorizing statutes for the DI and SSI demonstration programs, as well 
as requirements for specific demonstrations included in the Ticket to 
Work and Work Incentives Improvement Act of 1999 to determine the 
extent to which the projects in SSA's demonstration program address 
statutory requirements. For projects cancelled during this time period, 
we collected cancellation memos and other documentation to determine 
SSA's reasons for the cancellations. 

To describe the costs associated with the program, we collected 
expenditures data from SSA for each project--including funds spent to 
date for each project and total anticipated funding for the projects 
that are currently in progress. We assessed the reliability of the 
budget data by (1) manually checking the required data elements, (2) 
reviewing existing information about the data and the system that 
produced them, and (3) interviewing agency officials knowledgeable 
about the data. When we found discrepancies, we brought them to the 
attention of SSA officials and worked with them to correct the 
discrepancies before conducting our analyses. Based on these efforts, 
we determined that the data were sufficiently reliable for the purposes 
of this report. 

To assess the extent to which SSA's demonstration projects were 
designed in accordance with professional research standards and 
statutory criteria, we reviewed the most current information that SSA 
provided about each project, either an evaluation design or final 
evaluation report. These design and evaluation methodologies were 
assessed against professional research standards, consistent with the 
authorizing statutes' methodological requirements and GAO's and 
recognized academic criteria for conducting evaluation 
research.[Footnote 37] Key components of the professional research 
standards include: 

* methodological rigor of the project's design and evaluation and their 
appropriateness given the purpose of the research (e.g. use of an 
experimental or quasi-experimental design for an impact evaluation); 

* appropriate handling of any problems encountered when implementing 
the evaluation's design, such as participant attrition or insufficient 
sample sizes; 

* appropriate handling of any problems encountered with the data, such 
as missing values or variables; 

* appropriate variables to ensure internal and external validity, given 
the evaluation's design; 

* appropriate data analysis and statistical models, such as frequencies 
or multivariate analysis, given the evaluation's design; and: 

* overall strength of the evaluation design and analysis. 

To assess the appropriateness of each study's methodology for answering 
the research questions, we developed two data collection instruments 
based on these professional research standards--one for evaluation 
designs and one for the final evaluation reports. We then examined the 
strengths and weaknesses of the evaluation designs and final reports, 
taking into consideration the project's objectives, resource 
constraints, methodological approach, technical adequacy of plans for 
data collection and analysis, and when available, the presentation of 
the findings. A social scientist read and coded each evaluation design 
or final report. A second social scientist reviewed each completed data 
collection instrument and the relevant documentation to verify the 
accuracy of every coded item. For each DI demonstration project, we 
also reviewed the reports to ascertain whether they met statutory 
requirements that the project's results be broadly applicable to 
relevant segments of the DI beneficiary population, not just the 
project participants. For each SSI demonstration project, we also 
interviewed agency officials to determine whether SSA met its statutory 
obligation to obtain the advice and recommendations of specialists who 
are competent to evaluate the projects as to the soundness of their 
design, the possibilities of securing productive results, the adequacy 
of resources to conduct them, and their relationship to other similar 
research or demonstrations already in progress. We also identified key 
provisions of the demonstration authority statutes to assess SSA's 
compliance with congressional reporting requirements. 

To address SSA's planning and management of its demonstration projects, 
we interviewed SSA management and staff about the agency's policies, 
guidance and processes on developing and implementing demonstration 
projects, and collected supporting documentation where available. We 
assessed the adequacy of SSA's internal controls using the criteria in 
GAO's Standards for Internal Control in the Federal Government, GAO/ 
AIMD 00-21.3.1, dated November 1999. These standards, issued pursuant 
to the requirements of the Federal Managers' Financial Integrity Act of 
1982 (FMFIA), provide the overall framework for establishing and 
maintaining internal control in the federal government. Also pursuant 
to FMFIA, the Office of Management and Budget issued Circular A-123, 
revised December 21, 2004, to provide the specific requirements for 
assessing the reporting on internal controls. Internal control 
standards and the definition of internal control in Circular A-123 are 
based on the GAO Standards for Internal Control in the Federal 
Government. 

[End of section] 

Appendix II: Additional Statutory Criteria for the Benefit Offset 
National Demonstration Project: 

In addition to meeting professional research standards and the mandated 
methodological requirements for the DI demonstration projects, the 
Benefit Offset National Demonstration (BOND) project is required to 
address a number of specific issues.[Footnote 38] We found that BOND 
met all of these requirements except the one to test the project's 
effect on induced entry to and reduced exit from the DI rolls (table 
5). 

Table 5: Additional Statutory Criteria for the Benefit Offset National 
Demonstration Project: 

Statutory requirement: The effects, if any, of induced entry into the 
project and reduced exit from the project; 
SSA action: SSA considered including an experimental analysis of 
whether a benefit offset would induce entry to the DI program. In 
consultation with their research contractor, agency officials 
determined that the best possible approach would be cost prohibitive in 
terms of resources and time, and very complex, and still would be 
unlikely to produce solid, reliable estimates of what induced demand 
would occur if a DI benefit offset were implemented nationally. SSA 
will explore alternative approaches to estimating potential entry 
effects. 

Statutory requirement: The extent, if any, to which the project being 
tested is affected by whether it is in operation in a locality within 
an area under the administration of the Ticket to Work and Self- 
Sufficiency Program established under section 1148 of the Social 
Security Act; 
SSA action: An SSA official told us the effects of the Ticket to Work 
(Ticket) program could not be evaluated as intended in the statute 
because the two programs did not run concurrently, although it was 
originally expected that they would. The statutory criteria presume 
that the Ticket program would be operating in some of the same 
locations as the BOND project, but not all, so that some BOND 
participants would be eligible to participate in the Ticket program, 
but others would not be eligible. The Ticket program has now been 
implemented nationwide, and most SSA beneficiaries are now eligible to 
participate in it, thus the two groups needed for this evaluation no 
longer exist. However, the agency official and SSA's research 
contractor told us that SSA will be able to determine which BOND 
participants are also participating in the Ticket program, and the 
influence of the Ticket program on these participants would be 
considered in the BOND evaluation. 

Statutory requirement: The savings that accrue to the Federal Old-Age 
and Survivors Insurance Trust fund, the Federal Disability Insurance 
Trust fund, and other federal programs under the project being tested; 
SSA action: The BOND project design includes a benefit-cost analysis 
that will estimate the net budgetary effects of the benefit offset and 
counseling provided in each treatment package separately for 
participants, society, SSA, the federal government, and state and local 
governments. Agency officials and the research contractor told us that 
the savings that accrue to the Federal Old-Age and Survivors Insurance 
Trust fund, the Federal Disability Insurance Trust fund, and other 
Federal programs will be part of that analysis. 

Statutory requirement: The annual cost (including net cost) of the 
project and the annual cost (including net cost) that would have been 
incurred in the absence of the project; 
SSA action: The BOND project design includes a benefit-cost analysis 
that will estimate the net budgetary effects of the benefit offset and 
counseling provided in each treatment package separately for 
participants, society, SSA, the federal government, and state and local 
governments. Agency officials and the research contractor told us that 
the annual cost (including net cost) of the project and the annual cost 
(including net cost) that would have been incurred in the absence of 
the project will be part of that analysis. 

Statutory requirement: The determinants of return to work, including 
the characteristics of the beneficiaries who participate in the 
project; 
SSA action: The BOND project design includes a participation analysis 
that will examine the proportion of beneficiaries that take up the 
offer of the benefit offset and/or counseling, the timing and intensity 
of their participation, the characteristics of participants and 
nonparticipants, and the reasons for participation and 
nonparticipation. 

Statutory requirement: The employment outcomes, including wages, 
occupations, benefits, and hours worked, of beneficiaries who return to 
work as a result of participation in the project; 
SSA action: The BOND project design includes an impact analysis of the 
benefit offset and benefits counseling on levels of employment, 
earnings and benefits, wages, and occupations. Other employment 
analyses such as effects on regularity of work will also be conducted. 
Impacts on other dimensions will also be investigated, such as health 
status, function in activities of daily living, taxes paid, and family 
income. 

Source: GAO analysis of statutory requirements and SSA data. 

[End of table] 

In addition, while not included as part of the BOND demonstration, SSA 
plans to conduct research on variations in the amount of the offset as 
a proportion of earned income to determine the appropriate offset 
disregard, in accordance with the DI authorizing statute's 
requirements.[Footnote 39] 

[End of section] 

Appendix III: SSA's Use of Research Contractors and Universities on the 
Demonstration Projects: 

Project: Projects conducted under DI demonstration authority: 
Accelerated Benefits; 
Research consultant or university: MDRC (prime contractor); Mathematica 
Policy Research (subcontractor); 
Role on the demonstration project: SSA awarded MDRC a contract in 2006 
for the project's design, implementation, and evaluation. MDRC's design 
for the project and evaluation expanded on preliminary design criteria 
SSA developed between 2004 and 2005, and published in a Request for 
Proposal. MDRC finalized the design in collaboration with Mathematica 
Policy Research and SSA. MDRC and Mathematica began enrolling 
participants in the project in October 2007. Implementation will 
continue through 2011. MDRC is conducting the impact evaluation and 
Mathematica is conducting the process evaluation. SSA expects the 
evaluation to be completed in 2011. 

Project: Projects conducted under DI demonstration authority: Benefit 
Offset - 4 State Pilot; 
Research consultant or university: None; 
SSA awarded contracts directly to the states that are implementing the 
pilot; 
Role on the demonstration project: NA. 

Project: Projects conducted under DI demonstration authority: Benefit 
Offset National Demonstration (BOND); 
Research consultant or university: Abt Associates; 
Role on the demonstration project: SSA awarded Abt Associates a 
contract in 2004 for the project's design and will award Abt a second 
contract for its implementation and evaluation contingent on successful 
completion of the design. Abt's design for the project and evaluation 
is based on preliminary design criteria SSA developed between 1999 and 
2004 and published in a Request for Proposal. Abt is finalizing the 
design and planning for implementation in collaboration with SSA. 

Project: Projects conducted under DI demonstration authority: 
California RISE (CA HIV/AI) Demonstration; 
Research consultant or university: Mathematica Policy Research; 
SSA also awarded the California Department of Rehabilitation a 
cooperative agreement in 2006 to design the employment supports for the 
project, and implement the project; 
Role on the demonstration project: SSA awarded Mathematica Policy 
Research a contract in 2006 to design and implement the health services 
part of the project and to design and conduct the evaluation. 
Mathematica's design for the heath services component and the 
evaluation is based on preliminary design criteria SSA developed 
between 2004 and 2006 and published in a Request for Proposal. 
Mathematica worked on the design in 2006 and 2007, and SSA reviewed and 
provided input on it. Mathematica expected to implement the project in 
2007 and complete its evaluation in 2011, but SSA cancelled the project 
in 2007 before implementation began. 

Project: Projects conducted under DI demonstration authority: Early 
Intervention Demonstration; 
Research consultant or university: Disability Research Institute 
(Rutgers University); Abt Associates; 
Role on the demonstration project: SSA awarded the Disability Research 
Institute a contract in 2000 for Rutgers University to design, 
implement, and evaluate an Early Intervention pilot project. Rutgers 
proposed a design for the project and evaluation and submitted an 
evaluation design report for SSA's approval in 2002. Rutgers planned to 
implement the project in 2003 and begin its evaluation in 2004, but the 
project remained under design until the contract expired in 2005; SSA 
modified Abt's contract for the BOND project in 2005 to incorporate 
Early Intervention into BOND's design, implementation, and evaluation. 
Abt proposed options to incorporate the Early Intervention project, but 
SSA later decided to cancel it and eliminated it from Abt's contract in 
2007. 

Project: Projects conducted under DI demonstration authority: Mental 
Health Treatment Study; 
Research consultant or university: Westat; 
Role on the demonstration project: SSA awarded a contract to Westat in 
2005 to implement and evaluate a project SSA designed between 2003 and 
2005. Westat began working with the project sites in 2005 to prepare 
for implementation and began enrolling participants in the project in 
2006. Implementation is scheduled to continue through 2010. Westat 
began the evaluation in 2008 and will complete it in 2011. 

Project: Projects conducted under SSI demonstration authority: 
Disability Program Navigator; (with Department of Labor (DOL)); 
Research consultant or university: University of Iowa College of Law, 
Law Health Policy and Disability Center; Department of Labor also 
awarded states cooperative agreements to implement the project; 
Role on the demonstration project: DOL awarded a contract to the 
University of Iowa College of Law, Law Health Policy and Disability 
Center to provide technical assistance for the project's implementation 
and to evaluate the project. SSA provided partial funding for the 
contract through an Interagency Agreement with DOL in 2002, 2003, and 
2004. Implementation began in 2003. The University of Iowa did not 
evaluate the impact of the project because it could not meet SSA's data 
security requirements in order to obtain data needed for the 
evaluation. SSA officials told us that DOL plans to evaluate the 
project under a new contract with Mathematica Policy Research, but SSA 
is not funding that evaluation. 

Project: Projects conducted under SSI demonstration authority: Early 
Identification and Intervention Demonstration; 
Research consultant or university: George Washington University; 
Role on the demonstration project: SSA awarded a contract to George 
Washington University in 2004 to conduct preliminary research related 
to the Early Identification and Intervention project. SSA designed the 
demonstration project between 2004 and 2006 and issued a Request for 
Proposal for its implementation in 2007, but did not award a contract 
to implement or evaluate the project. 

Project: Projects conducted under SSI demonstration authority: Florida 
Freedom Initiative; 
Research consultant or university: Mathematica Policy Research; 
Role on the demonstration project: SSA funded a contract for 
Mathematica to design an evaluation for the project through an 
Interagency Agreement with the Department of Health and Human Services 
in 2004. Due to low enrollment, data needed for the evaluation was not 
collected by the State of Florida, which designed and implemented the 
project between 2003 and 2007, and the evaluation was not conducted. 

Project: Projects conducted under SSI demonstration authority: Homeless 
Outreach Projects and Evaluation; 
Research consultant or university: Westat; 
Role on the demonstration project: SSA awarded a contract to Westat in 
2004 to evaluate a project SSA designed between 2003 and 2004. SSA 
implemented the project in 2004. Westat completed its evaluation in 
2007. 

Project: Projects conducted under SSI demonstration authority: Interim 
Medical Benefits; 
Research consultant or university: None-project cancelled before a 
contract was awarded; 
Role on the demonstration project: SSA planned to issue a Request for 
Proposal to hire a professional research organization to design, 
implement, and evaluate the project but cancelled it before it awarded 
the contract. 

Project: Projects conducted under SSI demonstration authority: 
Pediatric Medical Unit; 
Research consultant or university: Association of University Centers on 
Disabilities (AUCD); 
Role on the demonstration project: SSA awarded a contract to AUCD in 
2006 to design, implement, and evaluate the project; however, AUCD's 
design for the project began in 2004 under an extension of another 
contract. AUCD partially implemented the project in 2006 and 2007. SSA 
cancelled the project in 2008 before AUCD completed the implementation 
or evaluation. 

Project: Projects jointly authorized under DI and SSI authorities: 
State Partnership Initiative; 
Research consultant or university: Virginia Commonwealth University 
(VCU) (prime contractor); Mathematica Policy Research (subcontractor); 
SSA also awarded 12 states cooperative agreements to design and 
implement the project; 
Role on the demonstration project: SSA awarded multiple contracts to 
VCU starting in 1998 to provide technical assistance to the states that 
were implementing the project and to conduct an evaluation. VCU 
completed its evaluation in 2006. SSA awarded three contracts executed 
by VCU's subcontractor, Mathematica: one in 1998 to provide technical 
assistance to the states that were implementing the project, another in 
1999 to design the evaluation, and one in 2003 to conduct the 
evaluation. Mathematica completed its evaluation in 2005. 

Project: Projects jointly authorized under DI and SSI authorities: 
Youth Transition Demonstration; 
Research consultant or university: Mathematica Policy Research (prime 
contractor); MDRC (subcontractor); 
Role on the demonstration project: SSA originally designed and 
implemented the project in 2003, but due to methodological limitations, 
SSA awarded a contract to Mathematica in 2005 to redesign, implement, 
and evaluate the project. Mathematica began implementing the new design 
in 2006. Implementation will continue until 2013. Mathematica will 
complete its evaluation in 2014. 

Source: GAO analysis of SSA data. 

[End of table] 

[End of section] 

Appendix IV: Comments from the Social Security Administration: 

Social Security: 

Social Security Administration: 
Baltimore MD: 
21235-0001: 

The Commissioner: 

September 16, 2008: 

Mr. Daniel Bertoni: 
Director, Education, Workforce, and Income Security Issues: 
U.S. Government Accountability Office: 
441 G Street, NW: 
Washington, D.C. 20548: 

Dear Mr. Bertoni: 

Thank you for the opportunity to review and comment on the draft 
report, "Social Security Disability: Management Controls Needed to 
Strengthen Demonstration Projects" (GAO-08-1053). 

Enclosed are our detailed comments to the draft report recommendations 
along with suggested technical revisions. 

If you have any questions, please contact Ms. Candace Skurnik, 
Director, Audit Management and Liaison Staff, at (410) 965-4636. 

Sincerely, 

Signed by: 

Michael J. Astrue: 

Enclosure: 

Comments On The Government Accountability Office (GAO) Draft Report, 
"Social Security Disability: Management Controls Needed To Strengthen 
Demonstration Projects" (GAO-08-1053): 

Thank you for the opportunity to review and comment on this draft 
report. We appreciate the report's recognition of our efforts to use 
the demonstration authorities we were provided and to include more 
rigorous methodologies in doing so. We are in general agreement with 
the tenor of the report and are pleased with the report's 
acknowledgement of management's efforts over the past year to improve 
the management of demonstration projects. We also appreciate that the 
report recognizes that while experimental designs are the "gold 
standard" for impact evaluations, quasi-experimental methods can be 
appropriate given all the considerations of our demonstration projects. 

Although this report contains only one recommendation for executive 
action, it does reference and comment on our progress in implementing 
three recommendations from a prior audit report (GAO-05-19). The 
comments that follow address all four recommendations. Following these 
comments, we have included other comments and suggested revisions on 
the content of the report. 

Current Recommendation: 

To improve the Social Security Administration's (SSA) management of its 
demonstration projects we recommend that the Commissioner of Social 
Security direct the Deputy Commissioner for the Office of Retirement 
and Disability Policy to: 

* Establish written policies, procedures, and mechanisms for managing, 
and operating its demonstration projects that are consistent with 
standard research practices and internal control standards in the 
federal government, including those for coordinating with internal and 
external stakeholders and sharing information with Congress. 

Comment: 

Although we generally concur with this recommendation, the report does 
not address existing processes and written procedures for managing and 
reviewing demonstration projects. For example, for staff and managers 
responsible for demonstrations, the performance management system 
provides a means for routinely reviewing demonstrations from project 
planning through implementation and evaluation. Our management and the 
Office of Management and Budget (OMB) routinely review demonstrations 
for the budget process, including the development and review of all 
documents required for the conduct of demonstration projects and all 
material associated with the budget cycle. We have extensive 
information on our intranet website to help Contracting Officer's 
Technical Representatives who are working on demonstration projects 
understand the OMB clearance process for Federal Register notices and 
survey instruments for data collection. 

Other efforts that facilitate the effective management of demonstration 
projects include receiving a monthly update on each demonstration 
project and including demonstration projects in the annual 
work plan for the Office of Program Development and Research. For some 
demonstrations, management requires more frequent updates and becomes 
more directly involved at critical points. For example, for the Benefit 
Offset National Demonstration, our management meets with the project 
team at least on a weekly basis. 

We agree with the need to develop a guidebook to assist our staff in 
the design, implementation, and evaluation phases of demonstration 
projects and the value of piloting demonstration projects before 
proceeding with full implementation (page 28). As the report indicates, 
we have developed a guidebook. However, our guidebook does not provide 
as much information as this report recommends and we consider it a work 
in progress. We also believe our Temporary Assistance to Needy Families-
Supplemental Security Income (TANF-SSI) Disability Transition Project, 
which is a collaborative effort between us and the Department of Health 
and Human Services' Administration for Children and Families, may serve 
as a model for developing possible interventions for demonstrations and 
add to the information and guidance contained in the guidebook. 

Prior Recommendations for GAO Audit Report GAO-05-19: 

Recommendation 1: 

Develop a formal, comprehensive, long-term agenda for conducting 
demonstration projects. 

Comment: 

Our current demonstration project documentation, as presented in our 
work plan and budget, does provide a significant amount of information 
related to our demonstration project agenda. However, our ability to 
develop new demonstration projects and a corresponding agenda is 
limited by our statutory authority and available resources. 

As this report indicates, Section 234 of the Social Security Act, as 
amended, expired in December 2005. Since that time, we have not had the 
statutory authority to initiate new demonstration projects under title 
II of the Act. This impediment has a significant impact on the scope of 
demonstration projects we can undertake. 

In addition, as recognized in this report, demonstration projects 
require extensive resources and are "inherently complex and difficult 
to conduct" (page 7). 

Demonstration projects must be conducted on a scale that allows us to 
make reasonable assumptions regarding the applicability of the outcomes 
to the general beneficiary population. For example, the total estimated 
cost of our Accelerated Benefits Demonstration project is $43 million. 
However, we will use nearly $32 million to provide health benefits to 
the 1,200 participants. In addition, we are providing benefits 
counseling and employment supports to a subset of those applicants. 
Therefore, the primary costs associated with these projects are the 
interventions provided and the need to provide those interventions to a 
large sample of beneficiaries. We are, therefore, limited in the number 
of demonstration projects we can implement at any given time. 

Recommendation 2: 

Establish an expert panel to review and provide regular input on the 
design and implementation of demonstration projects from the early 
stages of a project through its final evaluation. 

Comment: 

We responded to the earlier audit report by employing a Technical 
Advisory Group (TAG) for demonstration projects. Each TAG consists of 
nationally renowned subject matter experts for the particular 
demonstration project. By using experts in a specific field for each 
project or development of a particular intervention, we obtain the 
specific expertise necessary, as opposed to a standing panel of experts 
who may not have the specific expertise. In addition, we have worked 
with experts from the research community to discuss specific 
demonstration project issues. Finally, changes in our management and 
component organization, as mentioned in the report, have now made it 
possible for us to utilize the expertise available within the agency. 

As noted in the report, current procedures in the guidebook call for 
the use of expert panels that include internal and external experts for 
the development of demonstration projects. The guidebook also 
references the need for input and review of the draft statement of work 
from external experts. We intend to follow these procedures in 
approaching new demonstration projects. In discussing this 
recommendation, the report notes that we did not regularly seek input 
from key stakeholders for Federal disability policy, such as the Social 
Security Advisory Board. The only new work that we have underway that 
may result in a demonstration project is the TANF-SSI Disability 
Transition Project. This work responds to a prior recommendation from 
not only the GAO, but also the Social Security Advisory Board. We also 
recently discussed plans for the Benefit Offset National Demonstration 
with the Social Security Advisory Board. Before any new demonstration 
project is initiated, we will follow the existing written instructions 
in the guidebook that call for collaboration with external 
stakeholders. 

Recommendation 3: 

Establish formal processes to ensure that, at the conclusion of each 
demonstration project, SSA fully considers and assesses the policy 
implications of its demonstration results, and clearly communicates 
SSA's assessment to Congress. 

Comment: 

As the report indicates, our current demonstration projects include 
explicit provisions for fully analyzing project results, and we will 
clearly communicate the results and the policy implications to 
Congress. We have also reinforced this need by placing the statutory 
reporting requirements in our guidebook. 

The current report, in discussing this prior recommendation, criticizes 
the content of our annual reports to Congress. We agree with the 
overall suggestion that the reports could be more informative. We have 
already recognized this need, and our recent reports have attempted to 
provide Congress with more information regarding project status and 
potential issues.

[End of section] 

Appendix V: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Daniel Bertoni, (202) 512-7215 or bertonid@gao.gov: 

Staff Acknowledgments: 

In addition to the individual named above, key contributions to this 
report were made by Michael Collins, Assistant Director; Jason Holsclaw 
and Anne Welch, Analysts-in-Charge; Dana Hopings; Annamarie Lopata; 
Jean McSween. Additional support was provided by Kenneth Bombara; 
Daniel Concepcion; Jennifer Cook; Cindy Gilbert; Sharon Hermes; Joanie 
Lofgren; Joel Marus; Mimi Nguyen; Patricia Owens; Daniel Schwimer; Kris 
Trueblood; Kathy White; Charles Willson; Elizabeth Wood; and Jill Yost. 

[End of section] 

Related GAO Products: 

Federal Disability Programs: More Strategic Coordination Could Help 
Overcome Challenges to Needed Transformation. GAO-08-635. Washington, 
D.C.: May 20, 2008. 

Social Security Disability: Better Planning, Management, and Evaluation 
Could Help Address Backlogs. GAO-08-40. Washington, D.C.: December 7, 
2007. 

High Risk Series: An Update. GAO-07-310. Washington, D.C.: January 
2007. 

Federal Disability Assistance: Wide Array of Programs Needs to be 
Examined in Light of 21st Century Challenges. GAO-05-626. Washington, 
D.C.: June 2, 2005. 

Social Security Disability: Improved Processes for Planning and 
Conducting Demonstrations May Help SSA More Effectively Use Its 
Demonstration Authority. GAO-05-19. Washington, D.C.: November 4, 2004. 

SSA's Rehabilitation Programs. GAO/HEHS-95-253R. Washington, D.C.: 
September 7, 1995. 

Impact of Vocational Rehabilitation Services on the Social Security 
Disability Insurance (DI) Program. GAO/HRD-T-88-16. Washington, D.C.: 
May 26, 1988. 

Social Security: Little Success Achieved in Rehabilitating Disabled 
Beneficiaries. GAO/HRD-88-11. Washington, D.C.: December 7, 1987. 

Social Security: Observations on Demonstration Interviews with 
Disability Claimants. GAO/HRD-88-22BR. Washington, D.C.: December 3, 
1987. 

Social Security: Demonstration Projects Concerning Interviews with 
Disability Claimants. GAO/HRD-87-35. Washington, D.C.: February 19, 
1987. 

[End of section] 

Footnotes: 

[1] GAO, Social Security Disability: Improved Processes for Planning 
and Conducting Demonstrations May Help SSA More Effectively Use Its 
Demonstration Authority. GAO-05-19 (Washington, D.C.: Nov. 4, 2004). 

[2] GAO, High Risk Series: An Update, GAO-07-310 (Washington, D.C.: 
January 2007). 

[3] GAO, Designing Evaluations, GAO/PEMD-10.1.4 (Washington, D.C.: 
March 1991); GAO, Assessing the Reliability of Computer-Processed Data, 
GAO/OP-8.1.3 (Washington, D.C.: April 1991); P.H. Rossi, M.W. Lipsey, 
and H.E. Freeman, Evaluation: A Systematic Approach (California, 2004); 
B.R. Worthen, J.R. Sanders, and J.L. Fitzpatrick, Program Evaluation: 
Alternative Approaches and Practical Guidelines (New York, 1997). 

[4] GAO, Standards for Internal Control in the Federal Government, GAO/ 
AIMD-00-21.3.1 (Washington, D.C.: November 1999). These standards, 
issued pursuant to the requirements of the Federal Managers' Financial 
Integrity Act of 1982 (FMFIA), provide the overall framework for 
establishing and maintaining internal control. Also pursuant to FMFIA, 
the Office of Management and Budget issued Circular A-123, revised 
December 21, 2004, to provide the specific requirements for assessing 
the reporting on internal controls. Internal control standards and the 
definition of internal control in Circular A-123 are based on the GAO 
Standards for Internal Control in the Federal Government. 

[5] To be eligible for DI benefits, a worker must be (1) insured and 
(2) disabled according to the definition of disability. To be insured, 
they must have worked a minimum amount of time in employment covered by 
Social Security (similar to eligibility for Old-Age and Survivors 
Insurance benefits). However, for disability benefits, if an individual 
does not have 40 quarters of coverage (generally about 10 years), they 
must have 1 quarter of coverage (1 quarter of coverage is equal to 
$1,050 in 2008 and indexed to the annual increase in wages) for each 
year after 1950 or from age 21 up to the onset of disability. 

[6] DI demonstration authority was provided under § 505(a) of the 
Social Security Disability Amendments of 1980 (Pub. L. No. 96-265). 
Another provision of the act, § 505(b), amended §1110 of the Social 
Security Act of 1935 to provide SSA with similar demonstration 
authority for the SSI program, including authority to waive SSI program 
rules. 

[7] Pub. L. No. 108-203. 

[8] GAO-05-19. 

[9] GAO-05-19. 

[10] SSI demonstration authority focuses, in part, on prevention and 
reduction of dependency, aid in effecting coordination of planning 
between private and public welfare agencies, improving the 
administration and effectiveness of programs carried on or assisted 
under the Social Security Act, and ascertaining the feasibility of 
treating alcoholics and drug addicts to prevent the onset of 
irreversible medical conditions which may result in permanent 
disability. 42 U.S.C §1310. 

[11] 42 U.S.C. § 434. 

[12] Pub. L. No. 106-170 § 302. 

[13] Individuals with serious mental illnesses now represent over a 
quarter (27 percent) of all DI recipients, and they account for the 
single largest diagnostic group (34 percent) on the SSI rolls. As part 
of our ongoing work on modernization of federal disability programs, we 
have raised the issue of planning for growth in the demand for services 
and benefits. See GAO, Federal Disability Assistance: Wide Array of 
Programs Needs to be Examined in Light of 21st Century Challenges. GAO-
05-626 (Washington, D.C.: June 2, 2005). 

[14] 42 U.S.C. 434 § 234(b). 

[15] Several of the demonstration projects experienced challenges 
meeting new data security requirements that SSA established in 2006 
following the theft of a Department of Veterans Affairs computer, which 
heightened SSA's concerns about data security. SSA officials told us 
that DOL plans to evaluate the project under a new contract with 
Mathematica Policy Research, but SSA is not funding that evaluation and 
has a very limited role in planning and designing it. It is unclear 
what information will be available to use for making disability program 
and policy decisions. 

[16] This program has been renamed the Work Incentives Planning and 
Assistance (WIPA) program. 

[17] GAO-05-19. 

[18] The State Partnership Initiative was nearing completion at this 
time, and it was too late to make such improvements to it. 

[19] We did not assess the evaluation designs for the Disability 
Program Navigator and the Early Identification and Intervention 
Demonstration projects because SSA could not provide us with needed 
information. The Disability Program Navigator project concluded without 
an evaluation, and the Early Identification and Intervention 
Demonstration was cancelled before its design was finalized. 

[20] Social Security Act § 234(b). 

[21] Research experts have noted that there is an inherent tradeoff 
between using experimental designs and generalizing results. 

[22] Each state designed its own project for the State Partnership 
Initiative. 

[23] Social Security Act § 1110(a)(2) 

[24] Pub. L. No. 106-170 § 302. 

[25] GAO-05-19. 

[26] According to our analysis, the designs of the projects currently 
under way are considered strong or reasonable when assessed against 
professional research standards. 

[27] In response to our draft report SSA provided technical comments 
claiming an additional $600 million in savings resulting from a project 
cancellation. After analyzing the supporting documentation, we 
determined that the cost estimates that SSA provided did not directly 
support the proposed cost savings; therefore, we did not incorporate 
the suggested change into the report. 

[28] In 2007, SSA had moved DISP's operational component to another 
office in response to findings and recommendations made by its 
Inspector General. Agency officials told us that following the 
reorganization, DISP and the Office of Policy had similar missions, 
especially in regard to research. 

[29] GAO/AIMD-00-21.3.1. 

[30] GAO-05-19. 

[31] GAO-05-19. 

[32] The Social Security Advisory Board is an independent, bipartisan 
board created by Congress and appointed by the President and Congress 
to advise the President, Congress, and the Commissioner of Social 
Security on matters related to the Social Security and Supplemental 
Security Income programs. The National Council on Disability is an 
independent federal agency, composed of members appointed by the 
President by and with the advice and consent of the U.S. Senate, to 
provide advice to the President, Congress, and executive branch 
agencies to promote policies, programs, practices, and procedures that 
guarantee equal opportunity for all individuals with disabilities. 

[33] GAO/AIMD-00-21.3.1. 

[34] GAO/PEMD-10.1.4. 

[35] GAO-AIMD-00-21.3.1. 

[36] This office was merged with SSA's Office of Policy in February 
2008 and is now called the Office of Retirement and Disability Policy 
(ORDP). 

[37] GAO, Designing Evaluations, GAO/PEMD-10.1.4 (Washington, D.C.: 
March 1991); GAO, Assessing the Reliability of Computer-Processed Data, 
GAO/OP-8.1.3 (Washington, D.C.: April 1991); P.H. Rossi, M.W. Lipsey, 
and H.E. Freeman, Evaluation: A Systematic Approach (California, 2004.) 
B.R. Worthen, J.R. Sanders, and J.L Fitzpatrick, Program Evaluation: 
Alternative Approaches and Practical Guidelines (New York, 1997). 

[38] Pub. L. No. 106-170 § 302. 

[39] Social Security Act of 1935 § 234(1)(C). 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability.  

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each 
weekday, GAO posts newly released reports, testimony, and 
correspondence on its Web site. To have GAO e-mail you a list of newly 
posted products every afternoon, go to [hyperlink, http://www.gao.gov] 
and select "E-mail Updates."  

Order by Mail or Phone: 

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to:  

U.S. Government Accountability Office: 
441 G Street NW, Room LM: 
Washington, D.C. 20548:  

To order by Phone: 
Voice: (202) 512-6000: 
TDD: (202) 512-2537: 
Fax: (202) 512-6061:  

To Report Fraud, Waste, and Abuse in Federal Programs:  

Contact:  

Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: 
E-mail: fraudnet@gao.gov: 
Automated answering system: (800) 424-5454 or (202) 512-7470:  

Congressional Relations:  

Ralph Dawn, Managing Director, dawnr@gao.gov: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, D.C. 20548:  

Public Affairs: 

Chuck Young, Managing Director, youngc1@gao.gov: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, D.C. 20548: