Ticket to Work Evaluation (January 2006)

APPENDIX C
National Beneficiary Survey Methodology and Data Tables

This appendix describes the National Beneficiary Survey methods and presents more detailed information and data that support the survey findings reported in Chapter III. The appendix is organized as follows:

• Section A discusses the purpose of the survey and presents summary statistics on the survey sample.
  
  
• Section B describes the sample design, including the selection of primary sampling units, clustered and unclustered components, target population, strata, and sample sizes.
  
  
• Section C discusses the content and design of the survey questionnaire.
  
  
• Section D describes the data collection process and procedures.
  
  
• Section E presents additional information and data that support the findings presented in Chapter III.
  
  

1. Purpose of the Survey

As part of the TTW evaluation, MPR is conducting the National Beneficiary Survey (NBS). The survey, sponsored by SSA’s Office of Disability and Income Security Programs, collects data from a nationally representative sample of SSA disability beneficiaries (hereinafter referred to as the Representative Beneficiary Sample), and two cross-sectional samples of TTW participants (hereinafter referred to as the Ticket Participant Sample) across four years. In addition, cohorts of TTW participants will be followed longitudinally. In all, four rounds of interviews will be conducted annually beginning in 2004. Data are collected by means of a computer-assisted telephone interview (CATI) with a computer-assisted personal interview (CAPI) followup for beneficiaries who did not respond to the CATI interview or who requested an in-person interview to facilitate their participation in the survey. Whenever possible, the interview was attempted with the sample person. If the sample person was unable to complete either a telephone or in-person interview due to their disability, a proxy respondent was sought.

The overall purpose of the NBS is to obtain data necessary to address the evaluation questions outlined by Congress in the Ticket Act that are not available from SSA administrative sources. In particular, the survey has five key objectives:

• To provide critical data on the work-related activities of SSI and DI beneficiaries, particularly as they relate to the implementation of TTW Work
  
  
• To describe the characteristics and program experiences of beneficiaries who use their Ticket
  
  
• To gather information about beneficiaries who do not use their Ticket and the reasons for this choice
  
  
• To evaluate the employment outcomes of Ticket users and other SSI and DI beneficiaries
  
  
• To collect data on service utilization, barriers to work, and perceptions about TTW and other SSA programs designed to help SSA beneficiaries with disabilities find and keep jobs
  

The survey data will be combined with SSA administrative data to provide critical information on access to jobs and employment outcomes for disability beneficiaries, including those who participate in the TTW program and those who do not. In addition to use in the TTW evaluation, the survey data will give SSA information about a nationally representative sample of working-age disability beneficiaries. These data may therefore by useful for other policy making and program planning efforts, and by external researchers interested in disability and employment issues.

2. Summary Statistics

People in both the Representative Beneficiary Sample and the Ticket Participant Sample receive the same survey questionnaire. Round 1 CATI data collection for both samples began in February 2004. Beginning in May 2004, in-person CAPI interviews were conducted concurrent with CATI interviews. In-person interviews were conducted with sample persons who requested an in-person interview, those who needed an in-person interview to accommodate a disability, and telephone nonrespondents. CATI and CAPI Round 1 data collection was completed in October 2004. A total of 7,603 interviews were completed with individuals in both samples: 6,302 cases were completed by telephone and 1,301 were completed by CAPI. An additional 531 sample persons were determined to be ineligible to participate in the survey.¹ The overall unweighted response rate for the combined sample was 77.2 percent.² The overall weighted response rate was 77.6 percent.³ Proxy interviews were completed for 1,997 sample persons. Interviews were completed with 6,520 individuals in the Representative Beneficiary Sample and 1,083 persons in the Ticket Participant Sample. An additional 458 beneficiaries and 73 Ticket participants were determined to be ineligible to participate in the survey. The unweighted and weighted response rates for the Representative Beneficiary Sample were 77.0 percent and 77.5 percent, respectively. The unweighted and weighted response rates for the Ticket Participant Sample were 78.9 percent and 80.9 percent, respectively.

1. Overview of the Design

The survey design for the NBS calls for interviewing four national cross-sectional samples of SSA disability beneficiaries—one each for 2003, 2004, 2005, and 2006—and two cross-sectional samples of Ticket to Work participants in each of three groups of states (Phase 1, 2, and 3 states) chosen to represent implementations of Ticket to Work. In addition, sample members in the first Phase 1 cross-sectional Ticket Participant Sample for each phase will be followed longitudinally until 2006. Thus, two samples were fielded for Round 1, the first Representative Beneficiary Sample, and the first Phase 1 cross-sectional sample of Ticket participants.

For Round 1, primary sampling units (PSUs) were formed in every state based on counts of the number of beneficiaries in each county based on data from SSA. A three-stage sample design was used to select the Representative Beneficiary Sample:

• In the first stage, the number of PSUs to be selected from each of the Phase 1, Phase 2, and Phase 3 states was identified. The total number of PSUs to be selected was 80.
  
  
• In the second stage, PSUs were selected with probability proportional to the size of the beneficiary population in the PSUs. Because one PSU was selected twice given the large number of beneficiaries in the included county, the final number of PSUs selected was 79. In the two largest PSUs (which were selected with certainty), second-stage sampling units were formed within the PSUs based on zip code; two secondary units were selected in one of these PSUs and four secondary units were selected in the other PSU.
  
  
• In the third stage, the Representative Beneficiary Sample was selected in four age-specific strata. The final sample size for the Representative Beneficiary Sample was 9,064.
  

The Ticket Participant Sample comprised both a clustered and an unclustered sample. The clustered Ticket Participant Sample was selected in the same manner as the Representative Beneficiary Sample, using the same PSUs, but due to the small number of Ticket participants, the secondary sampling units were not used and the sample was drawn from all participants in the PSUs. Participants were stratified by the payment type under which their Ticket had been assigned (traditional, milestone-outcome, and outcome-only) rather than by age. As described further below, an unclustered sample of participants was selected to supplement the clustered participant sample for participants who had assigned their Tickets under the outcome-only payment system. All of the participants in this payment type were selected. The final sample size for the Ticket Participant Sample was 1,466 (see Table C.1 for sample size by strata). The Survey Sample Design Report includes more detailed information regarding the selection of PSUs and the overall NBS sample design (Bethel and Stapleton 2002).

2. Target Population

The target population for both the Representative Beneficiary Sample and the Ticket Participant Sample consisted of SSI and DI beneficiaries between the ages of 18 and 64 who were in current pay status as of June 2003. The target population included beneficiaries in all 50 states and the District of Columbia. Beneficiaries in the Trust Territories and Puerto Rico were excluded from the survey target population.

For the Ticket participants, the study population was constrained by the TTW rollout schedule. The program was implemented in three phases (approximately one-third of the states in each phase). The target population for the first survey round included individuals who were participants in SSA’s TTW in the Phase 1 rollout states. A participant was assigned to a phase for this study on the basis of the person’s address at the time of program rollout regardless of the person’s current address. Thus, a “Phase 1” participant might be residing in any state at the time of the survey.

MPR processed a beneficiary universe file from SSA of approximately 9.4 million records and a participant universe file of 21,477 records. Approximately 1.9 million beneficiaries were in the selected PSUs.

Source: MPR Survey Management Data.

3. Strata Definitions and Sample Sizes

The sample is designed to be statistically and operationally efficient and to provide adequate sample sizes for the planned analyses. In order to ensure a sufficient number of persons seeking work, the Representative Beneficiary Sample was classified into sampling strata based on age, with persons in the younger age categories selected at higher rates than persons in the oldest age category. The sampling strata for the Ticket Participant Samples were defined by the payment system.

The Representative Beneficiary Sample was divided into the following age groups, 18-24, 25-39, 40-54, and 55-64, which were used as the sampling strata. The target number of completed interviews for Round 1 was 2,000 beneficiaries in each of the three younger age groups (18-24, 25-39, and 40-54). For the 50-64 age cohort, the target number of completed interview was 1,200 beneficiaries. While the focus of the survey was on working age beneficiaries who are Ticket-eligible (that is, all working age beneficiaries except those who SSA classified as “medical improvement expected” (MIE) and former youth beneficiaries without an adult continuing disability review, or CDR, allowance), a small sample of all Ticket-ineligible beneficiaries was included so that the survey results would represent the entire working age population.

Two subpopulations of beneficiaries are ineligible for Ticket assignment:

• Beneficiaries who were designated as MIE at the time they received their initial disability allowance decisions, and who have not yet completed a CDR
  
  
• Young SSI recipients who are 18 years old, are still receiving benefits because of their eligibility as children, and have not completed a re-determination under the adult eligibility criteria
  

Although these beneficiaries are not eligible for Ticket participation, they were included in the survey samples to give complete coverage of the national beneficiary population.

TTW participants can assign their Ticket under three payment systems: (1) outcomes-only; (2) milestone-outcome, or (3) under the traditional VR reimbursement system. Because the prevalence of the outcome-only payment type was low among Phase 1 participants, both clustered and unclustered samples of participants were selected for this payment type. The samples of participants using the milestone-outcome and traditional payment types were limited to the clustered sample.⁴ The target number of completed interviews for participants at Round 1 was 1,000 overall, with a target of approximately 333 in each payment type stratum.

Sample members in both the clustered and unclustered samples underwent the same level of locating activities to identify a telephone number so that a telephone interview could be attempted. For the unclustered sample, beneficiaries who could not be located or who required an in-person interview were “closed out” and classified as ineligible. For the clustered sample, beneficiaries who could not be located or who required an in-person interview were eligible for a field followup and were assigned to field locators/interviewers. The beneficiaries who were classified as ineligible in the unclustered sample because of no field followup are accounted for by the beneficiaries who had field followup in the clustered sample. This process is analogous to the accepted practice of subsampling of nonrespondents for more intensive effort and in this case is subsampling of cases for field followup.

For fielding purposes, we selected 2.5 to 3 times as many cases as we needed to ensure an adequate pool of completed interviews. These samples were randomly partitioned into subsamples (called “waves”). During the data collection period, we monitored the sample results and determined whether, and in what strata and PSUs, additional cases were needed. (Section D and Tables C.4 and C.5 provide further information about the completion rates and the breakdown of completed interviews and ineligible respondents included in the count of respondents.)

1. Questionnaire Sections

The NBS collects data on a wide range of topics, including employment, disability, experience with a variety of SSA programs, employment services used in the past year, health and functional status, health insurance, income and other assistance, and sociodemographic information. Sample persons who identify themselves as Ticket participants are asked about heir experiences in TTW. Sample persons who do not identify themselves as Ticket participants are asked about their reasons for nonparticipation. The survey items were developed and initially pretested as part of a separate contract. Revisions were made to prepare the instrument for CATI/CAPI programming and the programmed instrument was pretested prior to fielding. The survey instrument is available from MPR upon request.

The questionnaire is divided into 18 sections, A through M, which serve the following purposes:

Section A - Screener. This section confirms that the correct sample person has been contacted and verifies that the sample person is still eligible for the survey. The sample person is also administered a cognitive assessment in this section to ensure that he or she is capable of completing the interview. If the sample person does not pass the cognitive assessment, he/she is asked if there is someone else who can answer questions about his/her health, daily activities, and any jobs he/she might have (such as a friend, parent, caseworker, or payee). An interview is then pursued with the proxy respondent.

Section B - Disability and Current Work Status. This section collects information on the beneficiary’s limiting physical or mental condition(s) and current employment status. If the beneficiary is not currently employed, the section explores reasons for not working. This section also includes questions designed to determine the job characteristics that are important to beneficiaries, and collects information about work-related goals and expectations.

Section C - Current Employment. Questions in this section collect detailed information about the beneficiary’s current job(s). Respondents are asked about the type of work performed, type of employer, hours worked, benefits offered, and wages earned. The section also asks about work-related accommodations, those received, as well as those needed but not received. Other questions solicit information about job satisfaction.

Section D - Jobs/Other Jobs During 2003. This section collects information about employment during the 2003 calendar year, including: type(s) of employer(s), hours worked, wages earned, and reasons for leaving employment, if applicable. Other questions ask if beneficiaries worked or earned less than they could have (and if so, the reasons why), and collect information about experiences related to Social Security benefit adjustments due to work.

Section E - Awareness of SSA Work Incentive Programs and Ticket to Work. This section includes questions designed to assess whether the beneficiary is aware of, or is participating in, specific SSA work incentive programs and services. For the Ticket to Work program, information is collected on how beneficiaries learned about the program, and the names and dates they signed up with their current service providers.

Section F - Ticket Nonparticipants in 2003. This section collects data on reasons for non-participation in the Ticket to Work program. It asks whether the beneficiary has attempted to learn about employment opportunities (including TTW), problems the beneficiary may have had with ENs or other employment agencies, and how those problems were handled or resolved.

Section G - Employment-Related Services and Supports Used in 2003. Questions in this section ask beneficiaries about their use of employment-related services in calendar year 2003, including: the types of services received, the types of providers used, how long they received services, how the services were paid for, and reasons for and satisfaction with service utilization. Other questions ask about sources of information about services and the nature of any services that were needed but not received.

Section H - Ticket Participants in 2003. This section asks 2003 TTW participants about their experiences with the program, including information related to: how they decided to participate in the Ticket program; the kinds of information they used to pick their current service providers; development of the individual work plan (IWP); and any problems experienced with services provided by an EN. The section also includes a series of questions about how problems with ENs were resolved and overall satisfaction with the Ticket to Work program.

Section I - Health and Functional Status. This section includes questions about the beneficiary’s health status and everyday functioning, including the need for special equipment or assistive devices. Information is solicited regarding: general health status (via the SF-8TM scale⁵); difficulties with activities of daily living (ADLs) and instrumental activities of daily living (IADLs); a variety of functional limitations; substance abuse/dependence; and treatment for mental health conditions.

Section J - Health Insurance. Questions in this section collect information about the sources of health insurance coverage the beneficiary has, both at interview and during calendar year 2003.

Section K - Income and Other Assistance. Questions in this section ask about sources of income, including income received from earnings, Social Security, workers’ compensation, and other government programs and sources.

Section L - Sociodemographic Information. This section collects basic demographic information about the beneficiary, such as race, ethnicity, education, parental education, marital status, living arrangements, and household income.

Section M – Closing Information and Observations. In this section, address information is collected for the sample person, and telephone information for up to two contact people is collected for participants who may be selected for future survey rounds. The interviewer also records reasons a proxy or assistance was required, if appropriate, and documents special circumstances.

The Round 1 NBS took, on average, 60 minutes to administer. Interviews with Ticket Participant Sample members ranged from 60 to 70 minutes, while nonparticipant interviews ranged from 45 to 55 minutes. To promote response among Hispanic populations, the questionnaire was available in Spanish. Interpreters were used to conduct interviews in languages other than Spanish.

2. Respondent Types and Main Paths

Sample persons in the Representative Beneficiary Sample and the Ticket Participant Sample both receive the same version of the NBS questionnaire. All respondents are asked questions from sections A, B, E, G, I, J, K, L, and M. Only respondents who report that they are currently working are asked questions from section C. Similarly, only respondents who report working in 2003 are asked questions in section D. Section F is asked of respondents who report that they have either never tried to get a Ticket from SSA, have never tried to use a Ticket to sign up with a provider, or who were not signed up with a provider in 2003. Only respondents who report using their Tickets to sign up with a provider in 2003 are asked questions from section H. See Table C.2 for a summary description of the main questionnaire pathing.

The National Beneficiary Survey was executed as a dual-mode survey—initial interview attempts were made using CATI followed by CAPI. CATI data collection began in February 2004. CAPI interviewing of telephone nonrespondents and beneficiaries who requested an in-person interview began in May 2004 and continued, concurrent with CATI interviewing, through October 2004.⁶ In total 7,603 cases were completed (including 23 partial completes)—6,520 from the Representative Beneficiary Sample and 1,083 from the Ticket Participant Sample.

1. Pretest

A CATI pretest was conducted in December 2003 to test the programmed instrument prior to fielding. The pretest sample was selected from beneficiaries and participants who were not living in the sampled PSUs. Cases selected for the pretest were not included in the main survey sample. Given their rarity, outcomes-only cases were excluded from the pretest. Hearing-impaired respondents were oversampled so that we could test procedures for interviewing via teletypewriter (TTY). Ticket participants were also oversampled to ensure an adequate test of the participant query paths.

Overall, 74 pretest interviews were completed. Thirty-two interviews were completed with participants and 42 with nonparticipants. Of these, eight cases were completed with proxy respondents. As a result of the pretest, minor instrument changes were identified and programming problems corrected for full-scale CATI interviewing.

Source: MPR’s National Beneficiary Survey, Round One.

2. Data Collection Procedures

Prior to the interview, an advance letter and a list of frequently asked questions and answers was sent to all sampled beneficiaries for whom we had a valid address. The advance letter, printed on SSA letterhead and signed by an SSA official, identified SSA as the sponsor of the survey and MPR as the survey contractor, explained the purpose of the survey, offered assurances of confidentiality, and included a toll-free number and e-mail address that respondents could use to contact MPR with questions, and/or call to complete the interview at their own convenience. A toll-free TTY number was also included in the advance letter. To encourage participation and show appreciation for response, a post-paid incentive payment of $10 was offered to respondents who completed the survey.

Approximately three days after the advance materials were mailed, CATI calls began to all sample persons. If a sample person was not able to participate in the survey due to his or her disability, a proxy respondent was sought. If no proxy was available and an in-person interview was not possible, the final status of the case was classified as a nonresponse. Sample persons or proxies who requested an in-person interview and who were eligible for field followup were held for the start of CAPI data collection.

3. Locating

Prior to the advance material mailing, all addresses were verified or updated using a commercially available database. As addresses or telephone numbers were identified as invalid, MPR used a variety of techniques for locating updated information, including: database searches; calling relatives and friends; receiving updated contact information from SSA; and making in-person visits for field locating. Due to these efforts, approximately 92 percent of the sample was located for interviewing.

4. CATI, CAPI, and Proxy Interviewing

In total, 6,283 cases were completed by telephone. Eighty-two percent of the Representative Beneficiary Sample completes (n=5,323) and 89 percent of the Ticket Participant Sample completes (n=956) were completed via CATI. Thirty-one CATI cases were completed using TTY, Relay, or instant messaging technologies. Approximately 50 percent of the total completes were obtained before the start of CAPI data collection (May 2004).

In all, 3,109 cases, or approximately 30 percent of the total sample, were sent to the field for an in-person interview. Of these, 394 (13 percent) were eventually completed by telephone and 1,301 (42 percent) were completed by field interviewers. Most cases that were sent to the field (63 percent) were sent because they could not be located by electronic searches or telephone attempts or did not have a telephone. Another 20 percent were sent to the field because the sample person initially refused a CATI interview. An additional 16 percent were sent to the field because they were difficult to contact via telephone or had evaded contact efforts. The remaining one percent was sent to the field because they requested an in-person interview. Eighteen percent of the Representative Beneficiary Sample completes (n=1,178) and 11 percent of Ticket Participant completes (n=123) were obtained via CAPI.

Proxy interviews were completed for 1,997 sample persons (26 percent of all completes). In most cases, approximately 77 percent, a proxy was necessary because the sample person failed the cognitive assessment or was otherwise determined to be unable to respond due to a cognitive, or mental, impairment. Interviews were completed by proxy for 1,901 sample persons in the Representative Beneficiary Sample (29 percent of completes) and 96 sample persons in the Ticket Participant Sample (9 percent of completes).

An analysis of selected respondent characteristics indicates a few differences between CATI and CAPI respondents, and between respondents requiring a proxy interview and all interviews (Table C.3). Relative to CATI respondents, CAPI respondents were more likely to be: SSI-only recipients; black; younger; to have achieved lower levels of education; and to have experienced childhood onset of disability. Relative to all respondents, those requiring a proxy interview were much more likely to be sample members with mental retardation and who experienced childhood onset of disability, and were somewhat more likely to be: male; SSI-only recipients; younger; of Hispanic or Latino ethnicity; and employed at interview.

*Does not include 23 partially completed cases.
**Multiple responses possible.
Source: MPR’s National Beneficiary Survey, Round One.

A total of 6,520 cases from the Representative Beneficiary Sample were completed. An additional 458 sample persons in the Representative Beneficiary Sample were determined to be ineligible to participate in the survey. These cases included sample persons who were: deceased; no longer living in the continental United States; whose benefits status was pending; and who were incarcerated or institutionalized. The unweighted response rate for the Representative Beneficiary Sample was 77.0 percent. The weighted response rate was 77.5 percent.

A total of 1,083 cases from the Ticket Participant Sample were completed. An additional 73 were determined to be ineligible for the survey. The unweighted response rate for the Ticket Participant Sample was 78.9 percent. The weighted response rate was 80.9 percent. Table C.4 reports the final case disposition for all released cases in the sample. Table C.5 provides a breakdown of response rate by sample type and sampling strata.

Click here for Table C.4.

Source: MPR’s National Beneficiary Survey, Round One.

1. Subgroup Identification and Sample Sizes

Statistics presented in Chapter III and this Appendix are reported for all beneficiaries, and for several subgroups: DI-only beneficiaries, SSI-only beneficiaries, concurrent beneficiaries, TTW participants, and beneficiaries who indicated at the time of interview that they were employed. A respondent’s Social Security program status is based on administrative data, and reflects the status at the time the sample was drawn (June 2003). The TTW participant subgroup comprises respondents who were Phase 1 beneficiaries and members of the June 2003 TTW Participant sampling frame. These respondents are combined with those in the Representative Beneficiary Sample for purposes of computing the statistics for all groups except TTW participants. A combined sample weight was used when pooling the Ticket Participant and Representative Beneficiary Samples. Table C.6 shows the weighted and unweighted sample sizes for the full sample and the subgroups for which statistics are reported.

2. Detailed Data Supporting Chapter III Figures and Tables

Detailed data supporting the figures and tables presented in Chapter III are provided in tables C.7 through C.16. The Chapter III figures and tables to which the data correspond are noted in parentheses in the title of each table.

Source: MPR’s National Beneficiary Survey, Round One.

*Multiple responses possible.
Source: MPR’s National Beneficiary Survey, Round One.

*Own children defined as biological, adoptive, and/or foster care children of the respondent.
Source: MPR’s National Beneficiary Survey, Round One.

*Respondents may have reported multiple conditions causing current activity limitations.
Source: MPR’s National Beneficiary Survey, Round One.

Source: MPR’s National Beneficiary Survey, Round One.

Source: MPR’s National Beneficiary Survey, Round One.

Source: MPR’s National Beneficiary Survey, Round One.

Source: MPR’s National Beneficiary Survey, Round One.

* 73 Respondents (unweighted) were missing Phase information and re excluded from the phase-specific figures presented.
Source: MPR’s National Beneficiary Survey, Round One.

Source: MPR’s National Beneficiary Survey, Round One.

Source: MPR’s National Beneficiary Survey, Round One.

¹ Includes sample persons who were: deceased; no longer living in the continental United States; incarcerated or institutionalized; and those whose benefit status was pending. Return to Text.

² The unweighted response rate (completed interviews+partial completes+ineligible cases divided by all released cases) is an indicator of response among the sampled cases, but does not account for unequal sample weights or for the potential for nonresponse bias. Ineligible cases are included in the numerator for two reasons: 1) the cases classified as ineligible are part of the original sampling frame (and hence the study population). We obtained complete information to fully classify these cases (i.e., their responses to the eligibility questions in the questionnaire are complete) and therefore classify them as respondents; 2) incorporating the ineligibles in the numerator and denominator of the response rate is essentially equivalent to the definition of a response rate with these cases excluded if the persons with an additional estimation of the number of eligible cases are among those with eligibility unknown. By including the ineligible cases in the numerator and denominator, we avoid using this estimation stage and the response rate computation is more clearly explicated. Return to Text.

³ The weighted response rate (a weighted count of completed interviews divided by the weighted count of cases in the sample) provides an unbiased estimate of the proportion of the survey population that is represented by the completed cases. Therefore, the weighted response rate measures the potential for nonresponse bias. Return to Text.

⁴ For the Round Two survey, unclustered samples are required for both the outcomes-only and the milestones plus outcomes payment types. Return to Text.

⁵ SF-8TM is a trademark of QualityMetric, Inc. Return to Text.

⁶ Note that interviewing began approximately eight months after the sample was selected. Return to Text.