Keeping Track: National Disability Status and Program Performance Indicators

The National Council on Disability (NCD) is pleased to present you with Keeping Track: National Disability Status and Program Performance Indicators. This report is the result of a year-long effort by NCD to identify valid federal data and to describe the status of the U.S. population of Americans with disabilities.

During the past 30 years, advocates, policymakers, and a variety of public and private organizations have undertaken significant efforts to improve the lives of people with disabilities, culminating in the passage or improvement of legislation such as the Americans with Disabilities Act (ADA), various sections of the Rehabilitation Act, the Individuals with Disabilities Education Act, the Ticket to Work and Work Incentives Improvement Act, and others. Notwithstanding these various policies, little effort and progress has been made to measure and reflect upon the overall effectiveness and performance of these laws and policies, and their impacts on the quality of life for people with disabilities.

In 2005, the Government Accountability Office (GAO) conducted a review of 200 federal programs located in 20 agencies that served individuals with disabilities. It identified the need to transform many of the programs it reviewed to keep pace with the changing expectations and challenges of the 21st century. In addition, most participants at a 2007 GAO forum on modernizing disability programs agreed that multiple indicators were needed to measure the success of disability programs and that these measures should include not only economic measures such as income and employment, but quality of life measures as well.

Keeping Track: National Disability Status and Program Performance Indicators includes a set of statistical social indicators that NCD believes are currently able to measure the progress of people with disabilities in important areas of their life, over time. The report includes 18 indicators determined by stakeholders to measure “quality of life” using both objective and subjective measures. The indicators span a variety of life domains, including employment, education, health status and health care, financial status and security, leisure and recreation, personal relationships, and crime and safety. Collectively they will create a holistic representation of the lives of people with disabilities.

This report also lays out a roadmap for the federal government to expand the national disability indicator set, as well as a mechanism for installing the set into a key national indicator system. This report is consistent with the overall policy direction and progress advanced by the Administration through the New Freedom Initiative toward the goal of full participation for people with disabilities. NCD looks forward to working with key stakeholders in the federal government in the implementation of the recommendations in this report.

(The same letter of transmittal was sent to the President Pro Tempore of the U.S. Senate and the Speaker of the U.S. House of Representatives.)

National Council on Disability Members and Staff

Members

John R. Vaughn, Chairperson
Chad Colley, Vice Chairperson
Patricia Pound, Vice Chairperson

Milton Aponte, J.D.
Victoria Ray Carlson
Robert R. Davila, Ph.D.
Graham Hill
Marylyn Howe
Young Woo Kang, Ph.D.
Kathleen Martinez
Lisa Mattheiss
Lonnie Moore
Anne M. Rader
Cynthia Wainscott
Linda Wetters

Staff

Michael C. Collins, Executive Director
Martin Gould, Ed.D., Director of Research and Technology
Mark S. Quigley, Director of External Affairs
Julie Carroll, Senior Attorney Advisor
Joan M. Durocher, Senior Attorney Advisor
Sharon M. Lisa Grubb, Special Assistant to the Executive Director
Geraldine Drake Hawkins, Ph.D., Senior Program Analyst
Brenda Bratton, Executive Assistant
Stacey S. Brown, Staff Assistant
Carla Nelson, Secretary

Acknowledgments

The National Council on Disability (NCD) wishes to express its gratitude to the following people who helped draft this report: Nanette Goodman, Bonnie O'Day, and Allison Roche from the Cornell University Institute for Policy Research.

NCD also expresses its appreciation to the following individuals and organizations who gave freely of their time and expertise: Susan Daniels, Consultant, SM Daniels and Associates, Washington, DC; David Gray, Associate Professor of Occupational Therapy, Washington University School of Medicine, St. Louis, MO; Gerry Hendershot, Consultant on Disability and Health Statistics, Washington, DC; Andrew Houtenville, Senior Research Associate, Employment and Disability Institute, Cornell University, Ithaca, NY; Corinne Kirchner, Consultant, New York, NY; David Ruth, Consultant, Charlottesville, VA, Paraquad, St. Louis, MO; Help Yourself Inc., Washington, DC; American Association of People with Disabilities, Washington, DC; Laurie Mitchell Employment Center, Alexandria ,VA; and the Consortium of Citizens with Disabilities, Washington, DC.

Additional resources are available at http://www.ncd.gov/newsroom/publications/index.htm

APPENDIX A.           Review of Societal Indicator Systems
APPENDIX B.           Background and Methodology for Focus Groups
APPENDIX C.           National Data Collection Instruments
APPENDIX D.           Limitations of Existing Data
APPENDIX E.           Data on Individuals with Disabilities
APPENDIX F.           Program Assessment Rating Tool Outcome Measures
APPENDIX G.           National Core Indicators
APPENDIX H.           References

Executive Summary

During the past few years, policy makers and researchers have attempted to measure the status of people with disabilities compared with others in our society. They want to know if the lives of people with disabilities have improved since the passage of legislation such as the Americans with Disabilities Act (ADA), the Rehabilitation Act, the Individuals with Disabilities Education Act, and the Ticket to Work and Work Incentives Improvement Act. In addition to measuring overall status, various efforts are now being undertaken to measure the effectiveness and performance of these laws and programs, and their impacts on people with disabilities.

Measuring the status of people with disabilities and the extent to which programs are assisting their clients to achieve better outcomes is important for many reasons. The number of people with disabilities in the U.S. is large and growing: 49.7 million non-institutionalized people in the U.S. have disabilities, and about 21.5 million of these are working-age individuals (U.S. Census Bureau 2003, reported in Iezzoni and O’Day 2006; StatsRRTC 2005). These numbers are likely to increase with the aging of baby boomers. People who acquire significant physical disabilities in early life are living longer than in prior years largely because of medical advancements. Measuring the status of people with disabilities is also important because the limited data we have indicate that the employment rate, level of household income, and educational attainment are low, and the poverty and obesity rates among people with disabilities are high compared to their peers without disabilities. Program evaluations, such as those conducted by the Government Accountability Office (GAO) and the Office of Management and Budget (OMB) also show that disability programs could be improved (GAO 2005; OMB 2004).

There is much we do not know about the lives of people with disabilities. Currently, the statistics informing the policy debate are predominantly economic, such as employment and household income. Such statistics are helpful, but paint only part of the picture. Other quality of life dimensions are substantially overlooked. Although some surveys cover topics that are related to some aspects of well-being, such as income, assets, or health insurance status, they do not necessarily reflect other aspects of well-being as they would be defined by the target population.

This report describes what we know about the status of people with disabilities in the U.S., and examines current data to assess the extent to which they meaningfully measure the well-being of people with disabilities. While much useful data on the status of people with disabilities exists, significant knowledge gaps hamper efforts to improve their well-being. Additionally, social indicator systems currently used or being developed either underemphasize or do not include people with disabilities. No comprehensive report exists describing the status of people with disabilities.

This report proposes a set of statistical indicators to annually measure the status of working-age people with disabilities and compare them to working-age people without disabilities in the United States. This report focuses on the working-age population of individuals with disabilities, defined as those ages 21 to 64, because many federal programs focus on employment, making indicators for the working-age population particularly relevant. Additionally, the life circumstances of working-age people with disabilities differ enough from the circumstances of children, youth and seniors, that separate indicators are warranted; indeed, indicator systems already exist for children and seniors. The topical indicator set, which can be produced relatively quickly,
can provide a foundation for the social indicator systems being developed, and stimulate public awareness about the status of people with disabilities and the need
for improvement.

This report discusses disability as a socio-demographic characteristic. This may not reflect the full experience of disability because disability is an interaction between an individual with an impairment and the environment, rather than merely a personal characteristic. The report uses this language because it allows discussion of the disparity between people with and without disabilities in measures of well-being.

Social Indicators

Social indicator systems measure the progress of society as a whole on a given set of valued characteristics. Indicator systems can measure attributes of people, institutions, the economy, and the physical environment (GAO 2005). Existing indicator systems generally cover a particular domain, such as health or education, a particular population such as children or the elderly, or cover a particular locality, such as a city or county. In 2003, the GAO, in cooperation with the National Academies, convened a group of national leaders that ultimately began the Key National Indicators Initiative (KNII) to develop a comprehensive indicator system for the country as a whole. The goal of the initiative is to provide measures that will help Americans assess the nation’s progress in key areas, such as economic well-being, health status, and the environment. After several years of research at the request of Congress, the GAO recommended in November 2006, as part of a report on oversight priorities, that Key National Indicators be pursued.

The designers of the KNII system are using the wealth of existing data from a variety of public and private sources to develop a website with data on a set of key indicators that cover a wide range of topics from education and business to health and the environment. Unfortunately, the surveys used to collect data on many of these topics do not contain adequate measures to define disability, or meaningful measures for program outcomes. The range of different disability definitions and multiple uses of the term “disability” in laws, policies, programs, and data collection instruments make comprehensive data on people with disabilities extremely difficult to compile, which likely contributes to the fact that there is no comprehensive document that reports on the status of people with disabilities in the U.S. This lack of information limits the ability to monitor and evaluate the impact of laws, policies and programs on people with disabilities, which in turn undermines the nation’s ability to make informed programmatic and funding decisions.

This report presents a brief topical indicator set based upon available data to track key indicators over time for people with disabilities. NCD will present current data to populate these indicators in each of its Annual Progress Reports. NCD’s Progress Report offers an annual opportunity to feature the topical indicator set, and thereby stimulate interest among politicians, policy makers, the press, and the public to focus on how people with disabilities are or are not making progress as compared to people without disabilities, highlighting their continuing needs. These indicators will also provide a foundation for engaging with the KNII as it conducts its planned national civic outreach over the next two years. The proposed topical indicator set includes the following:

Employment
Indicator 1:	Employment rate
Indicator 2:	Employment rate by education status
Indicator 3:	Median annual labor earnings of full-time/full-year workers
Indicator 4:	Median annual labor earnings of full-time/full-year workers by education status
Education
Indicator 5:	Percent of people with less than a high school diploma
Indicator 6:	Percent of people with at least a college degree
Health status and health care
Indicator 7:	Obesity
Indicator 8:	Smoking
Indicator 9:	Health insurance status
Indicator 10:	Failure to get needed care because of cost by poverty status
Indicator 11:	Patients who report that doctors or other health providers always show respect for what they have to say
Financial status and security
Indicator 12:	Median household income
Indicator 13:	Poverty status
Leisure and recreation
Indicator 14:	Participate in leisure time physical activity
Personal relationships
Indicator 15:	Always or usually get the social and emotional support you need
Indicator 16:	Marital status
Crime and safety
Indicator 17:	Violent crimes per 1,000 people
Indicator 18:	Property crimes per 1,000 people

Recommendation 1: NCD recommends that the National Institute on Disability and Rehabilitation Research (NIDRR) establish and fund a coalition of disability policy makers and advocates to: 1) develop a fuller set of indicators that are important to people with disabilities, building on the indicators outlined in this report (see Chapter 5); and 2) ensure that disability is included as a demographic subgroup as the KNII is developed. The KNII offers an important opportunity to integrate disability into a larger national indicator system. When completed, the KNII will offer individuals who are looking for disability data relatively easy access to existing data. It will highlight the importance of including disability as a subgroup in analyzing the relative status and progress of the population and highlight gaps in data about people with disabilities.

Recommendation 2: Promote a standard set of disability questions. Some important federal surveys have no disability measures. When measures are included, they vary across surveys, often yielding inconsistent and confusing results. A common core of disability questions on all federal surveys would improve comparability and improve the national discourse about disability programs and policy.

The definition should identify people “who, because of their functional limitations, are at risk for the loss or restriction of opportunities to take part in the normal life of the community on an equal level with others” (Altman 2006). Questions aiming to identify this group should characterize people by the difficulties they have in undertaking basic activities, like walking, seeing, and self-care—and not questions asking if they have a condition which limits their ability to participate in society, for example to work or attend school. The analysis can then determine to what extent people with functional limitations are excluded from society. It is also important that the questions ask about functioning without assistive devices, since the presence of assistive devices can assist people with functional limitations gain access to society.

Based on ideas developed by the Washington City Group (Madans et al. 2004, Altman 2006, Mont 2007) and Stapleton et al. (forthcoming), we propose that the set of questions used to operationalize this definition meet the following criteria:

Including a set of questions meeting these criteria on all national surveys will allow us to compare the well-being of people with and without disabilities in each of the domains covered by the surveys, and to monitor changes over time in response to changes in environmental factors such as the economy and public policy. It would also ensure that data available for the general population and reported for other socio-demographic groups is also available by disability status. A short common set of questions also would allow for comparisons across surveys.

Recommendation 3: Fully disseminate disability data. Federal agencies and other organizations that conduct national surveys, such as the US Census Bureau of Labor Statistics and the US Department of Health and Human Services Administration on Healthcare Research and Quality, should provide comparisons of people with and without disabilities in their aggregated data reports and should, where sufficient data exists, offer comparisons of people with disability by gender, race, and other socio-demographic characteristics.

The Status of People with Disabilities

Some existing surveys probe deeply into certain life domains and are useful for measuring the well-being of people with disabilities. Using these data, we identify what is known about the status of people with disabilities. To find out what is important, we conducted six focus groups with individuals with disabilities, and one focus group with service providers. We asked participants what they considered to be important areas in which to assess quality of life, and what was important in each area. Based on their input, we identified 11 high priority domains or areas of life: employment, education, health, financial means, community participation, leisure and recreation, political participation, transportation, housing, personal relationships, and technology. Focus groups participants also identified 13 crosscutting individual and community dimensions that would contribute to a high quality of life.

Choice: To have the ability to select from various, acceptable options in important life areas, for example: the ability to choose whether and where to work, attend school, or participate in civic or community activities; the ability to select among various health care providers, insurance companies, and personal assistants; having options regarding where and with whom to live and how to spend money; and having the choice to marry and have children.

Spontaneity: To have the ability to make last minute plans to attend community events, go out with friends, or visit family. This requires accessible and readily available transportation and personal assistance, the ability to use a cell phone, community accessibility, and housing constructed to meet universal design requirements.

Aspirations: To have dreams and set and pursue goals for oneself in life areas such as employment, education, health care, or political participation.

Empowerment: To have: a sense of self esteem; the ability to advocate for oneself; understanding and acceptance of disability; knowledge about legal rights, including legislation such as ADA; a voice in programs that affect one’s life; and control
over finances.

Quality: To be able to: engage in employment that one finds meaningful; access timely and high caliber health care irrespective of insurance coverage; have adequate financial resources to afford to participate in community activities; have accessible, reliable transportation; and foster close, egalitarian relationships with friends and family.

Financial means: To have the ability to: purchase high quality services, including transportation, housing, and personal assistance; obtain a job that pays a decent wage with good benefits; and afford purchases required for leisure time activities.

Overall satisfaction: This category represents overall satisfaction in each domain; fulfillment in employment or education; ability to obtain high-quality health care; safe, affordable, accessible housing; participation in community events of one’s choosing; and having close relationships with friends and family.

Inclusion and integration: Participating in employment, education, recreation and religious activities along with people without disabilities; inclusion in social activities with co-workers; integrated housing and transportation; inclusion in exercise and fitness activities; having a feeling of belonging or acceptance; and having relationships with people with and without disabilities.

Assistance and support: Having access to help to meet various needs for independence, where “independence” means the opportunity to live outside externally- controlled settings, notably institutions; having assistance in making choices; getting informal support from family and friends to enable independence; and having access to needed coaching and support services in education and employment.

Public attitudes: Promoting appropriate expectations by members of the general public applied to people with disabilities in employment, education and community participation; experiencing high comfort level by members of the general public with people with all types of disabilities; having appropriate portrayals of disability in the media; increasing public regard of disability as a normal part of life; and being treated with dignity and respect.

Accessibility: Having physical and programmatic accessibility; usability of facilities; usable off-the-shelf technology by people with disabilities.

Non-discrimination: Experiencing fair treatment in employment, housing, education, health care, public and private agencies and organizations, lending and voting. This category is similar to, but goes beyond, public attitudes to encompass a legal-based definition of discrimination.

Safety/Risk: Safety includes such items as housing located in safe areas and back-up system for personal assistance services; risk includes the ability to work without fear of benefit loss.

With this feedback from focus group participants, we reviewed the existing national data in each of the 11 domains to report on what is known about the status of people with disabilities. While there is some data in each of the domains identified by the focus group participants, serious gaps exist in what the nation can measure using current data. Most questions related to the crosscutting concepts cannot be addressed with the data that are currently available.

Program Performance Measures

Program performance indicators, or indicators used for “performance measurement,” have a narrower purpose than the societal indicator systems we have been discussing. While topical and comprehensive indicator systems attempt to capture the state of society, program performance indicators are used by agencies, Congress, and OMB for ongoing monitoring and reporting of program accomplishments, particularly progress toward pre-established goals.

Most current program performance indicators measure the impact of programs on participants. They generally do not measure the number of people who could benefit from the program but who do not participate either because they do not meet eligibility requirements, are unaware that the program exists, or do not use the program because of other entry barriers such as inaccessibility or excessive red tape.

A second limitation of program performance indicators is that they do not account for the complex interaction between the program and other factors, such as the economy or other public programs. For example, the percentage of vocational rehabilitation clients obtaining competitive jobs at closure partly depends on the quality of the services delivered, but also depends on the characteristics of the users, the availability of jobs in the economy, and the barriers and disincentives to work created by Social Security Disability Insurance and other cash and medical benefits.

A third limitation is that programs serving a broader population tend to report the outcomes by all program enrollees, rather than by different demographic subsets of enrollees, and do not allow a program to determine the specific impact on people with disabilities. Outcome indicators for programs focusing on people with disabilities, particularly employment programs, are evaluated based on a narrow range of indicators that do not adequately capture the program’s impact on the quality of people’s lives.

Recommendation 4: Administrative records of all means-tested programs should include a disability indicator. Programs that serve individuals with disabilities, such as One-Stop Employment Centers, and TANF, should collect data on the number of individuals with disabilities who use their programs and compare outcomes between program users with and without disabilities. We recommend that the Interagency Committee on Disability Research (ICDR) develop a workgroup to establish criteria on which the indicator is based.

Recommendation 5: Expand the Job Training Common Indicators. NCD should ask the Department of Labor to explore options within its administrative data collection system to add questions to the Job Training Common Indicators that more adequately capture concepts important to the focus groups, including choice in job; whether the job uses the employee’s full talents and abilities; whether the wage is appropriate given their qualifications; the extent to which they are satisfied with job conditions (including place, facility, co-workers, schedule requirements, accommodations, and opportunities for advancement); and whether they have meaningful opportunities to make choices about the conditions of their work.

Recommendation 6: Agencies should consider the effects of programs on non-participants. Agencies should include participation rates for eligible individuals (or potentially eligible individuals) and measures of well-being for those who are denied services in their GPRA and PART outcome measures. Improved survey data on people with disabilities would help support such measures.

Chapter 1: Introduction

A. Need for the National Disability Performance and Indicators Project

During the past few years, policy makers and researchers have attempted to measure the status of people with disabilities as compared to others in our society. They want to know if the lives of people with disabilities have improved since the passage of legislation such as the Americans with Disabilities Act (ADA), the Rehabilitation Act, the Individuals with Disabilities Education Act, and the Ticket to Work and Work Incentives Improvement Act. In addition to measuring overall status, various efforts are now being undertaken to measure the effectiveness and performance of these laws and programs and their impacts on people with disabilities.

Measuring the status of people with disabilities and whether programs are truly
assisting their clients to achieve better outcomes is important for many reasons. First, about one-fifth of the non-institutionalized population or 49.7 million people report disabilities (U.S. Census Bureau 2003 reported in Iezzoni and O’Day 2006). Among those of working age, about 21.5 million or 13 percent have some type of disability. (See Exhibit 1.1, which presents disability by type, age, education and race among the working age population.)

The disability prevalence rate increases with age. As shown in Exhibit 1.1, seven percent of people ages 21–29 have a disability compared to 26 percent of people ages 60–64. These rates increase even more dramatically for the population over age 65: 30 percent of people ages 65–74 have a disability, and more than two-thirds (69%) of people over age 85 have some type of disability. Because the numbers of elders will grow substantially in coming decades, the number of individuals with disabilities is likely to increase. People who acquire significant physical disabilities in early life are living longer than in prior years largely because of medical advancements. Healthy aging with a disability has become an important clinical consideration and research topic, as persons with such conditions as cerebral palsy, polio, and spina bifida increasingly live into their seventh decade and beyond (Iezzoni and O’Day 2006).

Measuring the status of people with disabilities is also important because the limited data we have indicate that, in many ways, the status of people with disabilities has changed little. They remain below their non-disabled counterparts in many key areas where “more is better,” such as employment, household income, and educational attainment, and above their counterparts in other areas where a lower rate is preferable, such as poverty and material hardship. Other disparities include excess weight, reduced physical activity, increased stress, and less frequent mammograms for women over age 55 with disabilities (Iezzoni et al. 2001; Weil et al. 2002).

Program evaluations also suggest the need for improvement. The Government Accountability Office (GAO) has recently conducted a review of 200 federal programs located in 20 agencies that serve individuals with disabilities. It identified the need to transform many of the programs it reviewed to keep pace with the changing expectations and challenges of the 21st century. In addition to significant program redesign, it recommends looking across programs to determine the effectiveness of disability programs as a whole (GAO 2005). Such a review requires an examination of the entire population of those with disabilities, not just those who receive services from the individual programs.

The Office of Management and Budget (OMB) also sees need for improvement among federal programs serving people with disabilities. Through its Program Assessment Rating Tool (PART), it continues to mandate outcome measurement as a part of its government wide program review. It assesses whether agencies focus on achievable outcomes based on long-term goals and benchmarks, and mandates that agencies develop a plan of improvement to address deficits. Many of the federal programs that serve people with disabilities show need for improvement (OMB 2004).

Lack of progress on key indicators for people with disabilities may reflect two problems. One is that programs and services for people with disabilities have not kept pace with the changing expectations and aspirations of people with disabilities and the new realities of the 21st century. The other is inadequacies in our measurement and data systems. Although some existing national surveys probe deeply into certain areas and could be useful in measuring the well-being and progress of people with disabilities, they have several notable limitations, including inadequate definitions of disability and failure to measure aspects of life that are very important to people with disabilities. According to the National Council on Disability (NCD), policy issues on the forefront of the disability agenda, such as long-term care services and employment, require relevant and routinely collected measures, accurate data, sophisticated analyses, and both broad and well-targeted dissemination (NCD 1998).

There is much we do not know about the lives of people with disabilities. Currently, the statistics informing the policy debate are predominantly economic, such as employment and household income. Such statistics are helpful, but paint only part of the picture. Other dimensions of quality of life are substantially overlooked. Current survey questions do not provide a holistic picture of the disability experience. Although some surveys cover topics that are related to some aspects of well-being, such as income, assets, or health insurance status, they do not necessarily reflect other aspects of well-being as they would be defined by the target population.

Statistics can be a powerful tool for shaping the public debate. Whether viewed as “keeping score” or as a “report card,” regular reporting of numbers that can be compared over time is a technique for accountability that is likely to stimulate interest among politicians, policy makers, the press, and the public. The purpose of this report is to develop a set of statistical indicators to annually measure the status of working-age people with disabilities in the United States. NCD wishes to describe the status of working-age people with disabilities in our society and determine whether current data meaningfully measures their well-being. More specifically, the purposes of this report are to:

This report focuses on the working-age population of individuals with disabilities, defined as those ages 21 to 64. We are focusing on this population for several reasons: First, many federal programs focus on employment, making indicators for the working-age population particularly relevant. Second, the life circumstances of working-age people with disabilities differ enough from the circumstances of children, youth and seniors, that separate indicators are warranted; indeed, indicator systems already exist for children and seniors. (See Appendix A for a fuller discussion of these indicators.) Third, including seniors or children may impact the interpretation of the results. For example, including seniors, all of whom are covered by Medicare, in an indicator of insurance coverage, may mask the differential in levels of private health insurance coverage between working-age people with and without disabilities.

This report refers to disability as a socio-demographic characteristic. This may not reflect the full experience of disability because disability is an interaction between an individual with an impairment and the environment, rather than merely a personal characteristic. The report uses this language because it allows discussion of the disparity between people with and without disabilities in measures of well-being. It is the approach used to measure the disparity among racial groups. Race, which is consistently used as a socio-demographic characteristic, is also a social construction (Wilson 2003) and can be viewed as an interaction between personal characteristic and social response; we attribute social value to biological characteristics by labeling an individual with a racial identity. This report explores some of these definitional
issues throughout.

B. Current Measurement Systems

No comprehensive indicator system that describes the social conditions of the U.S. population exists, but interest in developing such a social indicator system is growing. (See Chapter 3.)

Social indicator systems are one of the chief mechanisms for assessing group status in our society. Although social data exist on many important aspects of daily life, there is no comprehensive indicator system at the federal level that describes our social condition (Miringoff et al. 2003). The official portrait of our nation’s prosperity focuses on narrowly-defined economic indicators such as Gross Domestic Product, the rate of inflation, the unemployment rate, income per capita, and poverty rates. These economic indicators give us a concrete sense of how we are doing as a nation, but important aspects of life, such as health status, psychological well-being, environmental factors, and others also deserve serious attention.

Social indicator systems measure the progress of society as a whole on a given set of valued characteristics. Indicator systems can measure attributes of people, institutions, the economy, and the physical environment (GAO 2005). Indicators are a reflection of many factors, such as the scarcity of natural resources relative to the size of the population, the state of knowledge and technology, and the structure of governance. Indicators also reflect the actions of federal, state, and local governments, a wide array of for-profit and not-for-profit entities, and individuals.

Existing indicator systems generally cover a particular domain, such as health or education, or cover a particular locality, such as a city or county. In 2003, the GAO, in cooperation with the National Academies, convened a group of national leaders that ultimately began the Key National Indicators Initiative (KNII) to develop a comprehensive indicator system for the country as a whole. The goal of the initiative is to provide measures that will help Americans assess the nation’s progress in key areas, such as economic well-being, health status, and the environment. After several years of research at the request of Congress, the GAO recommended in November 2006, as part of a report on oversight priorities, that Key National Indicators be pursued.

Social data currently collected are based on surveys without adequate disability measures. No comprehensive report exists on the status of individuals with disabilities in America. Government agencies or private organizations that serve particular segments of our society use national data to assess and report on the status of particular groups. For example the Federal Interagency Forum on Aging-Related Statistics produces Older Americans, Key Indicators of Well-Being. The Urban League annually publishes The State of Black America to focus upon the status of African Americans in the U.S. compared to society as a whole. Kids Count, a project of the Annie E. Casey Foundation, tracks the status of children in the U.S., providing policymakers and citizens with benchmarks of child well-being. These reports are a shorthand way to provide a picture of the group’s status and to educate the general public about social issues affecting particular groups. They can call attention to particular issues, but policy makers and administrators will need more complete data to make informed programmatic and funding decisions.

Unfortunately, many of the surveys used to collect social data do not contain adequate measures to define disability, or meaningful measures for program outcomes. Some existing surveys probe deeply into certain life domains and appear to be useful in measuring the well-being of people with disabilities in some domains. For example, the Cornell University Rehabilitation Research and Training Center on Disability Statistics (StatsRRTC) uses the American Community Survey (ACS) to develop “Status Reports” that provide demographic and economic indicators in several areas, including disability prevalence, employment (full and part-time), personal and household income, poverty rate, and education (www.disabilitystatistics.org ). But the ACS is constrained by the conceptual and methodological limitations of its disability questions, and because the other ACS questions cannot support a complete, holistic set of indicators. Existing social measurement systems generally do not include measures that would enable us to identify people with disabilities in a consistent and meaningful fashion, or to evaluate their quality of life. The range of different disability definitions and multiple uses of the term “disability” in laws, policies, programs, and data collection instruments make comprehensive data on people with disabilities extremely difficult to compile. This contributes to the fact that there is no comprehensive document that reports on the status of people with disabilities in the U.S., similar to those mentioned above. The lack of data limits that ability to monitor and evaluate the impact of laws, policies and programs on people with disabilities, which in turn undermines our nation’s ability to make informed programmatic and funding decisions.

Existing data on people with disabilities is focused on health and economic measures and tells us little about other things that people with disabilities think are important.

Currently available data provide information on some health and economic indicators, for example, obesity, employment, income, and assets. However, we know little about how people with disabilities perceive the quality of their lives or their social environments. Are their lives meaningful and rewarding? Are they content with the quality of their lives? Do they feel accepted by their communities? To what extent do they feel their opportunities for social participation are equal to those of their peers without disabilities? What is their perception of programs, such as those that provide rehabilitation, housing, health care and income support? These are questions for which little information is available to answer them.

Although this report focuses on outcome measurement to support indicator systems for working-age people with disabilities, many of the limitations we find are common to other indicator systems. In particular, they often do not take the perspective of the target group—how their lives are actually lived and how they perceive their own well-being.

Limitations on outcome measurement for people with disabilities apply in other contexts as well. Most importantly, programs serving people with disabilities are evaluated based on outcome measures that often poorly reflect outcomes important to their clients. For instance, a program’s success might be based on an employment placement rate relative to a benchmark (Projects with Industry, Department of Labor Customized Employment Grants); the percent who are employed for at least 90 days (vocational rehabilitation), increases in client earnings (Medicaid Buy-In programs); or reduced expenditures (Medicaid Home and Community-Based Waiver programs). These do not adequately capture the program’s impact on the quality of people’s lives, and are therefore insufficient to assess the true value of the program.

Given the inadequacies of existing indicators as just described, we can anticipate that the search for a workable indicator system for persons with disabilities will entail considering complex concepts, and ultimately will require some compromises toward practical recommendations. The next section outlines how this report tackles
those challenges.

C. Organization of the Report

Chapter Two sets the context for review of performance measurement systems, by providing a framework for discussing disability and social indicator systems. The chapter also provides a brief history of social indicator systems and a discussion of the role social indicator and program evaluation systems can play in measuring quality
of life.

Chapter Three reviews existing and emerging social indicator systems. Because no comprehensive national system yet exists, it focuses on the Key National Indicators Initiative (KNII). It also highlights lessons learned from a similar initiative in Canada. Chapter Three also reviews topical and population-specific systems related to health, education, aging, and race.

Chapter Four reviews available data on the status of people with disabilities and identifies gaps in our current knowledge. To identify knowledge gaps, we conducted six focus groups with individuals with disabilities, and one focus group with service providers. We asked them what they considered to be important areas in which to assess quality of life. Based on their input, we identified 11 high priority domains or areas of life, including employment, education, health, financial means, community participation, leisure and recreation, political participation, transportation, housing, personal relationships, and technology. Focus group participants commented on
what aspects of each area were important. With this feedback, we reviewed existing data in each of the 11 domains to report what is known about the status of people
with disabilities.

Chapter Five proposes a set of topical indicators to be used to compare people with and without disabilities in important life domains over time. These 18 indicators, along with the most recent data to populate them, provide a snapshot of the status of people with disabilities and a foundation for engaging with the KNII discussed in Chapter Three.

Chapter Six discusses the use of indicators in measuring the performance of federal programs. The chapter describes the Program Assessment Rating Tool (PART), the primary government-wide effort to measure program outcomes, as well as other program evaluation initiatives. We review several initiatives based on measures that assess the quality of life for program participants, including the National Core Indicators Project, the Supported Work Indicators, and others. Most of these indicators have focused on evaluating specific providers but they can be valuable in development of broader program evaluation measures.

Chapter Seven lists the recommendations that arise out of review, both those related to national indicator systems and those related to other data collection efforts.

Appendices A–G provide additional information on many of the issues covered in
this report.

Exhibit 1.1: Characteristics of the Working-Age Population with Disabilities (Ages 21–64)
Category	Number or percentage
Total Working-Age Population	169.8 million
Number with any Type of Disability	21.5 million
Percentage of Total Population with any Disability	13%
Disability type*
Sensory Disability	3%
Physical Disability	8%
Mental Disability	4%
“Self-care” Disability	2%
“Go-outside” Disability	3%
Employment Disability	7%
Race
White	12%
Black/African American	17%
Native American	23%
Asian	6%
Other Race or Multiple Races	12%
Age (years)
21–29 years	7%
30–39 years	10%
40–49 years	12%
50–59 years	19%
60–64 years	26%
Educational attainment
Less than High School	24%
High School Diploma or Equivalent	15%
Some College	12%
Bachelor’s Degree or More	6%

Chapter 2: Framework for Discussion of Social Reporting Systems

The purpose of this chapter is to establish the context and provide a framework for a discussion of performance measurement systems. We begin with key definitions and a brief history of social measurement systems. We discuss the purpose of social indicator systems and the role social indicator and program measurement systems can play in measuring the status of the population.

A. Concepts and Definitions

Indicators and indicator systems are the building blocks to measure quality of life and evaluate program effectiveness. They are used to measure three types of outcomes:

Both societal indicators and program performance indicators play important roles in ensuring a responsive and accountable government. Social indicators measure societal progress; program performance indicators measure the impact of specific programs on people’s lives. But the distinction is sometimes blurry. A social indicator answers a “societal outcome” question, such as “Do people have the ability to travel where and when they want to go using transportation that is affordable?” For people with disabilities, many factors play into the answer—the nature of the individual’s disability, their personal family resources, available public transportation, affordable vehicle modifications for people with physical impairments, available personal assistants or volunteer drivers, mobility training for people who are blind or have cognitive disabilities, and availability of assistive technologies. No single agency is responsible for ensuring that all of these needs are met.

Program indicators focus on a particular program or group of programs and measure the effect of the program on users’ lives. In the above example, an evaluation of a paratransit program might focus on the question “Is the program sufficiently flexible to permit spontaneous travel?” or “Do the paratransit and fixed-route transit services together provide adequate coverage throughout the transit district?” Only one program is being evaluated and, in general, the program will have its primary, and perhaps
only, impact on its participants. In both cases, measures that indicate quality should
be included.

In the examples above, both the societal indicator and the program performance indicator measure affordability, availability, and spontaneity in transportation. Quality in a specific transportation program contributes to quality transportation in the jurisdiction as a whole primarily, if not exclusively, through its impact on participants. Even when government programs are designed to address a certain issue for people with disabilities (such as improving access to transportation), they do not benefit all people with disabilities, because only a subset of the disability population uses them. As a result, a strong federal law or program may have only a small direct impact on the societal measure.

At the same time, however, such a program can have indirect consequences for society that go beyond the services provided to participants. For instance, the availability of transportation facilitates employment, shopping and participation in many other activities, and increased presence of people with disabilities in social settings can influence societal attitudes in ways that might have significant impacts on society. Such indirect impacts are captured in societal measures, but it is rarely possible to separate the indirect impacts from the total impact.

B. Brief History of Social Reporting Initiatives in the United States

There is a long history of efforts by leading democracies throughout the world to develop and sustain indicator systems. In the United States, efforts to use social indicators to provide a picture of our nation’s well-being arose during periodic crises: the rapid changes of the Progressive Era, the poverty of the Great Depression, and the social movements of the 1960s and early 1970s (GAO 2003).

Early in the twentieth century, problems of poverty and child labor raised concerns about the lack of a consistent monitoring system for the well-being of children and youth. To address this problem, the Children’s Bureau compiled a Handbook of Federal Statistics on Children, a compendium of data on infant mortality, birth rates, poverty, and other child-oriented indicators. This publication represented the first effort by the federal government to bring together social information from a variety of sources throughout the country and became an early model for social reporting in the U.S. (Miringoff et al. 2003).

During the 1930s, as the Great Depression deepened, federal mechanisms were established to gather social data and monitor social conditions. New Deal agencies began to coordinate data previously collected by the states. These agencies initiated and refined comprehensive national surveys in areas such as labor and health and began to document the nation’s social problems.

Social reporting received little attention until the 1960s, when Congress and the Administration wished to explore whether the technology used for space exploration contributed to the rise of new social problems. The Commission responsible for this investigation discovered that little of the critical information needed to assess American life was available. In 1966, President Lyndon Johnson directed the Department of Health, Education, and Welfare to explore “ways to improve the nation’s ability to chart its social progress.” The resulting study, Toward a Social Report, was viewed as a “preliminary step toward the evolution of a regular system of social reporting.” The study considered the types of indicators necessary for regular reporting, including measures of health and illness, social mobility, the environment, income and poverty, public order and safety, learning, science, and art (Miringoff et al. 2003).

After Congressional efforts to adopt a multifaceted indicator system failed to pass, the Bureau of the Census published Social Indicators, a less ambitious statistical report. Follow-up reports appeared in 1973, 1976, and 1980 but, the series was discontinued by the Reagan Administration. Today, reports on specific issue areas, such as crime, health, education, the elderly, and children are published regularly. There have been no further efforts at comprehensive, regular social reporting, but past endeavors have helped to shape the developments that are underway today.

C. Objectives of Societal Indicator Systems

The chief functions of broad societal indicators are to monitor movement relative to societal goals and to identify areas for which additional research or government attention is warranted. It is important to recognize however, that societal indicators are “indicative” of the relative status of different groups of people. They are not intended to imply causality. For example, data show that women with disabilities are less likely to have mammograms than women without disabilities (Iezzoni et al. 2001). But what causes this disparity? It might be that many mammogram machines are not physically accessible or that provider attitudes discourage use. It might be that disability is highly correlated with low income and low income leads to lower use of preventive care or lack of affordable, accessible transportation to travel to appointments. It might be that women with disabilities are more likely than others to rely on Medicaid, and there are too few Medicaid providers. Each of these possibilities leads to a different policy intervention. The social indicator is a signal that there is a problem, but additional research is needed to identify the cause.

D. Quality-of-Life Measures and Social Indicators

The primary purpose of this report is to identify quantifiable measures that represent a person’s quality of life (QOL). Various disciplines approach and define QOL, and its closely related topics of socioeconomic well-being and happiness, differently.

Philosophers have been debating the essential qualities of happiness for millennia. One school of thought believes that true happiness lay in the devotion to intellectual thought and the rejection of worldly goods (Russell 1945). Aristotle, on the other hand, thought that happiness lay in virtue (Russell 1945). The notion of happiness is subjective and everyone must find their own path to “happiness.”

We define QOL as “having the means and wherewithal to pursue happiness—however defined by each person.” This definition is consistent with the thinking of Thomas Jefferson, who included “life, liberty and the pursuit of happiness” in the Declaration of Independence, as well as with the conclusions of QOL researchers Diener and Suh (1997), and our focus group findings. When we asked people with disabilities to define “quality of life” or to answer the question, “What makes you happy?” they consistently mentioned concepts such as choice and empowerment. They wanted adequate income in order to be able to choose goods and services that would make them happy. Thinking about choice in housing, some wanted a place that would allow pets; some wanted to choose their roommate; some valued safety above all else. Consistently though, they all valued choice as critical to their quality of life.

Happiness and quality of life are often associated with the presence of favorable circumstances external to the individual, such as economic stability, a supportive family, access to health care to maximize health, as well as social, cultural and environmental conditions. These concepts are also influenced by internal factors such as attitudes and perspectives on personal circumstances, feelings of empowerment, and other intangibles that are difficult to quantify.

Social Indicators are used to quantify quality of life. While some social indicators may address factors internal to the individual, we adopt the more common approach (JCCI 2006) and focus mostly on the external environment, which is more measurable and is affected by public policies.

Objective measures, subjective well-being measures, and traditional economic indices are needed in unison to understand human quality of life, and to make informed decisions about public policy and funding (Diener and Suh 1997). Because objective measures are easier to gather than are individuals’ subjective perceptions of their social environment, comparing objective measures across subpopulations and over time is relatively straightforward. However, decisions involved in selecting and measuring the variables are not value-free, which limits the objectivity of the data.

The cost of collecting objective measures varies across measures. As a result, the choice of measures collected does not necessarily reflect their importance to society. For instance, financial transactions are much easier to document than how individuals use their time. As a result, outcome indicators based on financial transactions dominate the literature on social well-being, and statistics on time use (e.g., hours of unpaid work, time spent on self care, or visiting health care providers) are rarely reported, even though they are regarded as important by many people.

An individual’s sense of well-being is far more complex than can be quantified by even a well chosen set of objective measures based on external circumstances. Subjective measures are based on the notion that if a person experiences his or her life as good and desirable, it is assumed to be so (Diener and Suh 1997). Thus, it is appropriate to ask directly about factors such as feelings of joy, pleasure, contentment, and life satisfaction. Subjective well-being is not a state of simply being hedonistically happy. The central elements of well-being, a sense of satisfaction with one’s life and positive affective experiences, are derived from the context of one’s most important values, goals and expectations.

Despite the impression that “subjective” connotes lesser scientific credibility, studies have shown that subjective well-being measures possess adequate validity. For example, data on the Australian Unity Wellbeing Index, an index of subjective measures, has been collected since 2001. Recent studies have found that it is a valid, reliable, and sensitive instrument (Cummins et al. 2005). Despite this credibility, most measures of quality of life and well-being focus predominantly on objective rather than subjective measures.

In summary, this chapter has established a framework for the discussion presented in the remainder of the report, sorting out some useful distinctions between societal outcomes and related phenomena, specifically: individual outcomes; program evaluations; and quality-of-life assessments. We have drawn attention to objective and subjective aspects of outcomes, and measures that correspond to those aspects. This chapter also reviewed the early history of attempts at comprehensive social indicator systems, and the objectives sought by further work in that arena. The next chapter turns to current efforts to develop a social reporting system, along with some well known topical indicator systems.

Chapter 3: Existing and Emerging Societal Indicators

Many democratic nations, including Canada, Australia, South Africa and almost all European countries, are operating comprehensive key indicator systems and have been doing so for years. Even though the U.S. government uses literally hundreds of data collection instruments (surveys and administrative records) and spends over $4 billion per year collecting data, the data are not easy to find and use. The Federal Interagency Council on Statistical Policy provides a portal to data from the websites of over 100 federal agencies, including program participation data and survey data, accessible at http://www.fedstats.gov/ . This site links to a vast amount of data which, taken together, must be described as “disjointed,” with each data source using different definitions of such key concepts as disability and race. The system is difficult to navigate even for experienced researchers and policy makers.

There have been several attempts to rationalize this mass of data. For example, the Census Bureau produces the annual Statistical Abstract of the United States and the Bush administration presents selected economic and social indicators in the “White House Briefing Room” accessible at http://www.whitehouse.gov/fsbr/ssbr.html . The Briefing Room social indicators refer to crime, demography, education, and health.

The notion of disability is substantially absent in both of these compendia. In general, the briefing rooms present data for the population as a whole rather than by different demographic characteristics, and the Statistical Abstract generally includes disability as a health outcome rather than as a socio-demographic characteristic. That is, while it includes the number of people with disabilities by age group by state it does not include other characteristics of the population with disabilities, such as educational, economic or health status.

Hundreds of U.S. state and local jurisdictions have drawn from the substantial body of general population data collected at federal, state, and local levels to assemble indicators into an easily accessible tool such as a website or chartbook. Some of these indicator systems measure progress toward specific goals or benchmarks; others provide more general information about the economic, environmental, social, and cultural conditions of a jurisdiction (GAO 2004).

In addition to these broad-based collections of indicators, there are numerous topical indicator systems that are national in scale. Such systems focus on specific topics such as health care or education, or specific populations such as the elderly, children, or African Americans.

There is, however, no national indicator set that provides the public with information about the nation’s progress on a range of issues. In order to fill this gap, the Key National Indicators Initiative (KNII) undertook the challenge of developing and disseminating a set of key indicators that would encompass economic, social, cultural, and environmental outcomes. The initiative has grown to include diverse leaders from government, business, research, the media, and the nonprofit sector (GAO 2004). The KNII has recently formed a non-profit organization and is currently planning to release the first public version of its website in 2009. In this chapter, we discuss the KNII and several existing topical indicator systems with an eye toward identifying whether the best strategy for monitoring the status and progress of people with disabilities is to integrate disability into the KNII, to develop a separate topical indicator system focused on disability, or both.

A. Key National Indicators Initiative

In 2003, GAO, in cooperation with the National Academies, convened a forum of leaders from different sectors of the economy to discuss the need for and potential of a national indicator set. Since then, those individuals and institutions that chose to develop the KNII have taken a methodical approach to establishing a “credible, trusted source of unbiased, nonpartisan, data” (Hoenig 2007b). They have done extensive research on existing indicator systems (GAO 2004), engaged with diverse stakeholders, gained financial support from private foundations, established a governance structure, developed a prototype demonstration website and begun the design of a civic engagement process to help select key issues and indicators. GAO has recommended to Congress that it consider contributing funds to the initiative through a public-private partnership (GAO 2005b).

Over the past two years, the KNII has been pursuing a series of targeted efforts—under the auspices of the National Academies—aimed at testing effective and efficient ways of choosing topics and indicators as well as disseminating them in a useful manner via web-based information systems. Their first round of work on issues and indicators produced 22 topical areas and a pool of over 900 candidate indicators. Through a civic engagement process which is now being designed in detail, these topical areas will be reassessed and a smaller set of key indicators chosen to display on a website. Some examples of potential population indicators include:

For each indicator ultimately chosen, the website would provide a description of the indicator, overall statistics, breakdowns over time, geography and by demographic subgroup (where that data is available), description and limitation of the data source, and links to the source and related indicators and commentary.

To choose issues and indicators for the public release of its website, the KNII will use research, surveys, and a civic engagement process to bring together both diverse groups of Americans and experts from a variety of government agencies, research institutions, associations, non-profits, the media, and businesses. They are currently soliciting input from a broad range of groups. There has been, to date, only minor input from the disability community.

First, the goal of the KNII is to assemble information, not to collect it. Thus the indicators will be based on available data. However, the KNII is committed to highlighting the need for additional data collection for important indicators that cannot be measured with existing data (Hoenig 2007b). Second, the set of indicators is national and intended to drive decision making; thus it must be comprehensive and it must integrate the links
and interactions between component measures (GAO 2004). Third, the indicators focus on current status and past trends and not on modeling or projecting into the future (Hoenig 2007b).

To be successful at measuring the status and progress of people with disabilities, the KNII will have to be attentive to several issues. First, key indicators will need to reflect the issues that are particularly important to people with disabilities. For example, there will need to be measures of affordable housing, reliable and accessible transportation, or community accessibility. Second, since the KNII plans to present many of the population-based indicators by race, gender, and other socio-demographic characteristics, disability status must also be one of those characteristics. Even if the KNII does include indicators to address the first issue, there are significant challenges in addressing the second issue, including:

Only with the full engagement of the disability community can these issues be turned into opportunities through KNII that could increase the amount of knowledge available to the nation on people with disabilities.

B. The Canadian Experience

For the past six years, the Treasury Board of Canada has published annual Performance reports designed to provide a “whole-of-government” view of how individual departments and agencies contribute to broad outcomes in four areas—economic, social, international, and government affairs (Treasury Board of Canada 2006). It is designed as a companion document to 88 separate Departmental Performance Reports, which links each indicator with the government departments that can affect it.

While the Treasury Board developed its indicators, a coalition of public, private for-profit and not-for profit organizations, led by the Canadian Policy Research Network (CPRN) developed a related but separate indicator initiative. The CPRN set of indicators is designed to “measure what matters” (Atkinson Charitable Foundation 2007). The architects of the project note that although there are a variety of social welfare indicator systems in Canada, the question of which aspects of well-being should be monitored remains a “flashpoint of controversy” (Michalski 2002). There has been no effort to integrate the Treasury Board and the CPRN indicators.

Based on the concept that to be legitimate, societal indicators should reflect what matters most to the members of a community or a nation, the CPRN organized 40 “dialogue discussions” with nearly 350 participants in nine provinces across Canada. The participants reviewed background materials on quality of life issues and indicators, engaged in three-hour dialogue discussions, and identified their priorities for defining quality of life. These priorities serve as the basis for selecting the indicators (Michalski 2002). The final indicators are listed in Appendix A.

Most recently, the group has undertaken an initiative to aggregate the indicators into a single composite measure—the Canadian Index of Wellbeing (CIW) which they hope will be used in conjunction with the Gross Domestic Product (GDP) to measure growth in Canada. Unlike the GDP which is a single measure, the CIW is an aggregation of measures. There is still considerable disagreement over the appropriate methodology that should be used to create a statistic that is “comprehensive enough to satisfy the statisticians and policymakers but simple enough to be understood by the general public” (Laidlaw 2007).

C. Topical Indicator Systems

Topical indicator systems in the US focus on a variety of issues including the business cycle (the Conference Board Business Indicators), the environment (e.g., the State of the Nation’s Ecosystems, Environmental Protection Indicators for California and many others), education (e.g., State Education Indicators, National Science Foundation Science and Engineering Indicators), and health (the U.S. Department of Health and Human Services, Healthy People 2010).

In addition, a variety of groups, both public and private, have developed topical indicator systems focused on particular populations. For example:

The topical indicator systems that focus on African Americans, children, and the elderly do not generally identify people with disability as a subpopulation. For example, the compendia that present information on children presents the data by race, ethnicity, gender and age (where appropriate) but not by disability status. The one exception is the Older Americans chartbook, but even here, disability is presented as a health indicator rather than a subgroup characteristic. Thus, the project is limited to tracking the prevalence of disability in the aging population. It does not track other socioeconomic and health measures (such as poverty, housing expenditures, participation in the labor market, health care expenditures, or source of payment for health care) by disability status.

NOD in cooperation with the Harris Poll, has conducted five surveys between 1986 and 2004 to measure the participation of people with disabilities in American life. The survey asks dozens of questions, many of which change each time the survey is administered. However, NOD/Harris developed ten key indicators that are monitored over time. This set of indicators is potentially very useful because it includes objective and subjective measures and it captures some of the issues identified as important by our focus groups. NOD compares the value of the indicator over time for people with disabilities and the gap between the value for people with and without disabilities. However the indicators have significant limitations. Among others, the sample size of the survey is too small to detect important changes in the value of the indicator over time unless they are very large. (See Appendix A for more information on NOD indicators.)

Unlike the NOD/Harris indicators, the data for the “Healthy People 2010” indicators are drawn from data sources that are collected as part of ongoing federal surveys and administrative data. This approach has the advantage of presenting data that are based on larger sample sizes and collected regularly, but the breadth of the indicators is limited by the type of questions asked in existing surveys.

Neither existing indicator system for people with disabilities covers the breadth of issues that our focus groups indicated were important. For example, neither addresses the domains of housing, financial status and security or technology. Nor do they address the dimensions that are important to people with disabilities such as choice, spontaneity, and aspirations.

While the topical indicator systems for African Americans, children, and the elderly do not integrate disability status into their reports, they provide valuable insights about how to develop a successful topical indicator system.

Develop a coalition: Although indicator sets can be developed by governmental or non-governmental entities (e.g., The Annie E. Casey Foundation, Child Trends, the National Urban League, and the NOD), they all rely on coalitions of stakeholders to develop and disseminate the product.

Select indicators parsimoniously to provide adequate coverage of the subject with as few indicators as possible: All the topical indicator systems present a wide variety of indicators on a website or in a chartbook. The systems that receive the most attention from the media, such as Kids Count, the State of Black America and the NOD/Harris Survey of Americans with Disabilities, focus on fewer than 20
core indicators.

Avoid composite indicators: Only one indicator system, the State of Black America, includes a composite measure of well-being. It allows the National Urban League to report the status of African Americans is 73.3 percent in 2007, when compared to their white counterparts, considered as 100 percent, up from 73.0 percent in 2005 and 2006. Readers cannot identify whether the change of less than half of one percentage point represents any real change, or what type of change it might represent. The National Urban League does report sub-indices and highlights bellwether individual indicators such as unemployment, poverty, homeownership and incarceration rates (National Urban League 2007). Two well-known composite indicators of quality of life—Disability Adjusted Life Years (DALYs) and Quality Adjusted Life Years (QALYs)—illustrate one type of danger in producing a composite indicator. These measures are based on the medical model of disability and neglect the value of other outcomes for people with disabilities. (See Appendix A for a fuller discussion of these issues.)

Highlight limitations in existing data: Both the Federal Interagency Forum on Child and Family Statistics and the Federal Interagency Forum on Aging-Related Statistics include a section in their reports highlighting measures that should be developed and data limitations that need to be addressed in order to obtain a complete picture of the population (GAO 2004).

D. Conclusions and Recommendations

Recommendation 1: NCD recommends that the National Institute on Disability and Rehabilitation Research (NIDRR) establish and fund a coalition of disability policy-makers and advocates to 1) develop indicators that are important to people with disabilities, building on the indicators outlined in this report (see Chapter 5); and 2) ensure that disability is included as a subgroup characteristic as the KNII is developed. The KNII offers an important opportunity to integrate disability into a larger national indicator system, which is important for several reasons:

Chapters Two and Three of this report painted the landscape of existing and emerging social indicator systems. Although there are some topical indicator systems in discrete areas such as healthcare or on specific populations such as children, there are no comprehensive national quality of life systems in the U.S. The KNII effort shows promise but thus far has lacked meaningful involvement of disability researchers and advocates. If a comprehensive social indicator system is to include measurement of the status of people with disabilities in a meaningful way, existing surveys sponsored by government agencies must include adequate and consistent disability measures. Within this context, Chapter Four reviews available data on the status of people with disabilities and identifies gaps in our current knowledge.

Chapter 4: The Status of People with Disabilities

In this chapter, we discuss some of the limitations of existing data that should be kept in mind when reviewing statistics on disability, including problems with the definition of disability, reporting on individuals from diverse cultural or linguistic backgrounds, and various statistical limitations.

To identify knowledge gaps across available data sources, we conducted seven focus groups of individuals with disabilities. We asked them to articulate what they consider to be important areas in which to assess well-being. (See Appendix B for the focus group methodology and characteristics of participants.) We identified 11 high priority domains or areas of life, and 13 dimensions within each domain—such as choice, empowerment, and inclusion—that focus group participants said were important. We then reviewed existing data in each of the domains to report what is known about the status of people with disabilities, and to identify gaps in our knowledge. This review will assist us to identify current and potential indicators most relevant to individuals with disabilities that could be included in a national comprehensive indicator system and/or in a topical indicator system.

This review focuses on data that are currently published or easily available on websites. There are potentially useful questions included on some surveys but they are not aggregated or reported in a way that facilitates a comparison between people with and without disabilities. Even though some of these statistics could be available to researchers with computer programming skills, they are not included in our review.

Disability data is collected in a large number of federal surveys. In Appendix C, we briefly describe each survey, the sample methodology and sample size, the dissemination instruments, relevant variables, definition of disability, frequency of data collection, ability to analyze data on people with disabilities from different racial and ethnic groups, and strengths and weaknesses in ability to provide valid, reliable, and comparable disability statistics across time.

A. General Limitations of Existing Data

Data on individuals with disabilities have various limitations that affect its usefulness in measuring quality of life or in evaluating significant programs. These limitations are briefly described below, and discussed more fully in Appendix D.

Inconsistent definitions of disability: The definition of disability is quite complex and varies among different federal and state laws, public programs, insurance plans, and organizations. It is thus no surprise that surveys that were designed for varying purposes, and which generally use between one and six questions to identify whether the respondent has a disability, do not all use the same definition. Appendix C identifies the definition of disability used in each survey.

Inadequate definitions of disability: Some of the survey definitions are particularly problematic—most notably the “work limitation” definition. Several surveys ask respondents if they have a “disability or medical condition that prevents or limits the amount or type of work you can do.” This question equates disability with inability to work. Answers are likely to be sensitive to characteristics of the economic or physical environment and are likely to differ among individuals with different circumstances even with the same level of impairment. These limitations have been discussed at length by Burkhauser et al. (2002); Hale (2001), Silverstein et al. (2005) and many others.

Definitions that are too broad to provide meaningful data on a particular impairment group: As an example, the American Community Survey (ACS) combines visual and hearing disabilities into a “sensory impairment” category.

Inability to measure progress for diverse ethnic and racial groups: Although people with disabilities with diverse ethnic backgrounds constitute a disproportionate share of the disability community and may have unique needs (NCD 1993), they are often not adequately represented in surveys. Often, the sample size of specific racial or ethnic groups is too small to create separate estimates for them. In many of the larger surveys, it is technically possible to measure this subpopulation, but the reporting agencies do not cross tabulate race-ethnicity with disability status.

Limited ability to create a time series: There are few surveys with the same question, the same disability indicators and the same sampling methodology across time. The CPS is a long-established survey and has not changed the definition of disability since 1981. However, the survey uses the work-based definition discussed above, which has significant limitations.

Various types of statistical limitations: These limitations include sampling error (the sample does not accurately represent the population), coverage error (excluding part of the population from the survey, such as persons in institutions or without telephones), and nonresponse error (lack of response from certain types of individuals biases the results of the survey).

Statistics not adjusted for other characteristics. Disability is correlated with other characteristics such as age. Most data sources compare all people with disabilities to
all people without disabilities, without regard to age, rather than making adjustments
for the fact that disability prevalence increases with age. Similar criticisms apply to
other characteristics.

Data on people with psychiatric disabilities is particularly poor: Partly because there is no common, well accepted definition of psychiatric disability, there is little reliable data on prevalence and characteristics. People with psychiatric disabilities are sometimes included in the category of “mental disability” or data are only available for medically-defined subpopulations of people with specific types of psychiatric impairments, such as schizophrenia or depression. Also, because of severe stigma pertaining to psychiatric impairments, underreporting is likely by persons with the condition or by their family members.

B. Identifying Concepts That Are Important to Measure

As noted previously, we conducted seven focus groups—six groups of people with disabilities and their advocates, and one group of service providers. We structured the focus groups to yield two outcomes; first, a list of domains that participants feel are important, such as employment, housing, education, or health; and second, a list of dimensions within each domain that could be used to assess life satisfaction. (See Appendix B for a summary of the methodology and the characteristics of the focus group members.)

Domains and Dimensions Identified by Focus Group Participants: Participant comments about important quality of life issues could be divided into eleven domains, including:

Several crosscutting dimensions also emerged from the discussion, which we divided into individual and community dimensions. Although neither the domains nor the dimensions are mutually exclusive, they highlight distinctive aspects of the less tangible characteristics participants thought contributed to a high quality of life. Individual dimensions included the following:

C. Domains and Data

In this section, we describe the issues that focus group participants identified as important factors in the quality of their lives within each domain, the data currently available to measure disparity between people with and without disabilities in each domain, and knowledge gaps. Appendix E contains further detail on the data we report here, as well as data that would be available if all national surveys included a standard set of disability questions, or if all federal agency-produced statistics included disability status as a socio-demographic characteristic like age or race.

1. Employment

Issues identified by the focus groups: Focus group participants raised issues that went beyond the usual data collected on wages and hours worked. They stressed the importance of finding a job appropriate to their aspirations and long term goals; and working in a job they believed to be worthwhile. Service providers also stressed matching a client’s qualifications to a job in a field of interest and considering the client’s long-term aspirations.

Many participants emphasized the difficulty of obtaining employment due to employers’ negative attitudes and lack of awareness of the employment provisions of the Americans with Disabilities Act. They said that people with disabilities are often hired
for jobs below their qualification levels, because of employers’ low expectations of people with disabilities. Other participants thought they were paid lower wages because of their disabilities. Participants identified problems after they were hired, including unfair termination, lack of accommodation, and lack of inclusion in social activities with co-workers.

Data and gaps: The strongest source of employment data on most populations in the U.S. is the Current Population Survey (CPS). However, the definition of disability available in the CPS is a work-based definition, whose limitations were discussed above. Because of this and other controversies about measuring employment, there exists no authoritative time series of the employment rate for people with disabilities prior to 2005. For 2005 forward, we can rely on the ACS estimates. The NOD/Harris survey asks about employers’ expectations and discrimination. Other than these sources, we know very little about important aspects of employment identified by
the focus groups such as whether they believe the job is appropriate, given their aspirations.

2. Education

Issues identified by the focus groups: Participants discussed the desire to choose among various educational options, including integrated classrooms or programs that specialized in disability. Some participants identified needs for accommodations, such as tutors or slower pacing. Group members said they still confronted inaccessible school buildings and course materials. For many, cost, especially of college, limited their ability to obtain an education. They were aware of few or no financial resources to help them. Participants wanted data that would enable them to compare graduation rates between people with and without disabilities; as a woman said, “We know we’re equal when our mortar boards turn with the rest of them.”

Data and gaps: States provide annual data on early childhood education through high school in terms of the characteristics of special education students covered under the Individuals with Disabilities Education Act (IDEA) and the environment in which they are educated. IDEA reports cover graduation rates, but they calculate the rates differently from the way graduation rates are generally reported for students without disabilities so it is not possible to compare the two groups. Data we have from other sources indicates the following:

3. Health and Health Care

Issues identified by focus groups: Issues related to the quality of health care centered on communications with doctors and other medical staff. Focus group participants described situations in which medical professionals were not comfortable with disability, did not know how to respond to disability-related needs, or did not ask questions about drug use, sexual activity, or fitness they believed to be routinely asked of other patients. For some, quality health care meant finding a doctor who “treats you as a person, not as a condition.” Choosing a therapist or psychiatrist was particularly important to people with psychiatric impairments—a choice that was often absent if the service was covered by Medicaid or provided by the participant’s group home. Participants with physical or sensory impairments talked in detail about inaccessibility in health care facilities: e.g., difficulty in finding practices with adjustable exam tables or medical information in accessible formats.

Personal assistance services (PAS) were vital to the daily health of some participants with mobility impairments. As one woman said, “As a high-level quad, my quality of life is directly determined by access to affordable and reliable attendant care. If I don’t have attendant care, I can’t work and I can’t take care of my family.”

Data and gaps: There is a significant amount of data available on the health and health care of people with disabilities, largely due to sources such as the National Health Interview Survey (NHIS), the Medical Expenditure Panel Survey (MEPS), the Behavioral Risk Factor Surveillance System (BRFSS), the National Health and Nutrition Examination Survey (NHANES), and the National Survey of Family Growth (NSFG).

Despite the breadth of data available, many of the data sources used to monitor the health status of the general population do not include measures of disability. For example, the National Vital Statistics Mortality and Natality System, the National Survey on Drug Use and Health, and the National Ambulatory Care survey do not have any indicators of disability. In fact, one of the goals of Healthy People 2010 is to “include in the core of all relevant Healthy People 2010 surveillance instruments a standardized set of questions that identifies “people with disabilities.” As of 2005, only 17 percent of the instruments included such a measure (Healthy People 2010, goal 6.1).

Although none of the current data sources directly measures choice, spontaneity, aspirations, empowerment, quality, assistance and support, accessibility or non-discrimination, they do provide data on quality, finance, satisfaction, public attitudes and, to a lesser extent, inclusion/integration and safety/risk. Because of the quantity of data available on health and health care, we present only a sampling of the information available on insurance, expenditures, and access. For example:

4. Financial Status and Security

Issues identified by the focus groups: Many participants articulated the negative effects of their low income on their quality of life. Some saw having more money as an integral part of getting the community to listen to and meet their needs. Many participants wanted to work to increase their income, but said they were trapped on the benefit rolls. In addition to the ability to earn money, participants saw choice in how to spend one’s own money and how to self-direct service dollars, such as for personal assistance services, as an essential part of quality of life.

Data and gaps: The ACS collects detailed information about family and household income which is used in reports that include median income, income distribution, poverty status, ratio of income to poverty by race, age, education and other variables. Data about the causes of limited income, such as being trapped on the benefit rolls or restrictions on how the money can be used, are not available.

5. Community Participation

Issues identified by the focus groups: Participants defined community participation as more than just being involved in community activities. They stressed the importance of physical and program accessibility, including communication access for people with sensory disabilities, and the enforcement of access laws. They said that one indicator of the level of community integration would be the extent to which access laws are enforced. They stressed that community participation meant access to activities in facilities that were truly usable, for example, with bathrooms located close to the activity.

Participants stressed the importance of being able to do things spontaneously, on the spur of the moment, without having to plan in advance for transportation, personal assistance and accessibility. A man who is deaf wished he could pull off the road and use his cell phone to make last minute plans.

Group participants also talked about public attitudes and their impact upon community participation. They said that a measure of participation might be the extent to which people with disabilities believe that they are being treated with courtesy and respect by the general public.

Data and gaps: Most aspects of community participation identified as important by the focus groups are not covered in any existing surveys. However, existing surveys such as the NOD/Harris and the NHIS do provide some information on community participation of people with disabilities, for example:

6. Leisure and Recreation

Issues identified by the focus groups: When discussing recreation, participants raised many of the same issues as when discussing community participation, such as inclusion in activities not specifically designed for people with disabilities, positive and inclusive public attitudes, being able to participate spontaneously, with significantly less advance planning, and funding for personal assistance and transportation for recreational activities. One participant said, “So many people with disabilities go home and stay home;” she thought more funding for personal assistance for recreation could change that.

Data and gaps: Current data provide some information about the rates of participation in leisure and recreational activities of people with and without disabilities. However, there is very little information about the ability to participate in activities without significant advance planning.

7. Political Participation

Issues identified by the focus groups: Many participants believed that people with disabilities did not have a strong voice in politics—due in part to an absence of people with disabilities in elected offices, and in part to the lack of an organized, cohesive disability community. Participants also mentioned accessibility issues that posed barriers to political participation. Some participants simply found it difficult to get to the polls on Election Day, while others found some of the new identification requirements, such as requirements for ID cards, burdensome. Others mentioned access improvements, including new accessible voting machines that enabled blind people to vote independently. Some participants found voting by absentee ballot to be much easier than going to the polls. Again, choice in how to vote was the issue.

Data and gaps: Current data provides some limited information about voter registration and voter turnout for people with disabilities but little information about other aspects of political participation, such as participation in advocacy organizations and political party functions, is routinely collected.

8. Transportation

Issues identified by the focus groups: Participants who did not drive said that lack of public transportation severely hampered their ability to work, obtain an education, and participate in the community. They described an ideal transportation system as being one that is well-networked throughout the community, reliable, and affordable. They wanted a system with courteous staff, that did not require much advance planning to use, and that could be used for a multitude of trip purposes. Participants wanted accessible transportation that served outside the city limits—so that people with disabilities could live in rural communities or safer areas outside the central city. They wanted buses that ran more frequently and to more areas. Individuals with hearing impairments wanted to have the ability to request paratransit services via email. Many participants thought the best way to achieve the goal of an accessible public transportation system was to bring people with disabilities to the table—to let them take part in the planning and decision-making process.

Data and gaps: In response to the lack of information about transportation use among people with physical, mental, or emotional disabilities, the Bureau of Transportation Statistics (BTS) initiated the 2002 National Transportation Availability and Use Survey. This survey has not been repeated so it is not possible to measure progress in reducing transportation barriers. We know from existing data that:

9. Housing

Issues identified by the focus groups: Participants spoke of the lack of accessible housing in places people want to live. Often, accessible housing is located in unsafe areas, or in areas lacking nearby grocery stores or public transportation. Those who owned their own homes, or hoped to do so in the future, wanted financial assistance for accessibility modifications.

Other participants felt strongly that they should have a choice in their neighbors or roommates, a particular problem in supported living programs. They wanted more options in housing programs—such as being able to choose their roommates, live alone, have pets, and not be forced into “disabled/elderly housing” to obtain accessible, affordable housing.

Some participants said they still faced discrimination when seeking housing or loans to purchase homes. One participant said she was denied housing because she uses a dog guide, so she no longer brings her dog when looking for housing in order to avoid
such discrimination.

Data and gaps: There is limited information about the housing status of people with disabilities and no national data sources on housing accessibility. There are statistics on the number of complaints about fair housing discrimination, but it is difficult to identify if an increase in the number of complaints is a result of increased discrimination on the part of housing owners or increased awareness on the part of the consumer.

10. Personal Relationships

Issues identified by the focus groups: Participants agreed on the importance of personal relationships; as one participant said, “The value of personal relationships is so high—they are necessary for a good quality of life.” Another said a high quality of life was to, “get married, have an apartment, have kids, like everybody else.” They wanted choices as to whether to develop relationships with people with disabilities, people without disabilities, or both. Participants thought that agencies could facilitate this process by teaching social skills and offering opportunities to meet people. Group members also said that educating the public about disability could help people with disabilities foster relationships.

Data and gaps: The only information we have about personal relationships for people with disabilities is from the NOD/Harris survey. It is surprising that two data sources with great potential in this domain have not been analyzed—the National Survey of Family Growth and the National Survey of Families and Households. Both have disability indicators, both have sample sizes large enough to support an analysis of people with disabilities, and both have important measures of personal relationships.

In 2004, 79 percent of people with disabilities socialized with close friends, relatives, or neighbors at least once perweek (Harris Interactive 2004). The NOD/Harris survey has included this measure since 1986. In every year, people with disabilities were less likely than those without disabilities to have socialized once a week or more, with the difference between people with and without disabilities ranging from 10 to 15 percentage points. There is no apparent trend in either the percentage of people with disabilities socializing or in the disparity compared to people without disabilities.

11. Technology

Issues identified by the focus groups: Most participants viewed technology as vital to obtaining information, shopping on-line, or getting around in the community. Some participants saw a need for more funding for assistive technology for uses other than on the job, such as for independence, community participation, or education.

Other participants said that training in how to use technology was equally as important, so that they could obtain maximum benefit from their technology. Others, including those who used screen-readers, wished that off-the-shelf hardware and software were designed to be accessible to all.

Data and gaps: Much of the federal government data on the use of technology by people with disabilities comes from the 1994 National Health Interview Survey on Disability (NHIS-D) and is thus quite dated. Several one-time independent surveys (Carlson and Ehrlich 2005) look at access to off-the-shelf technology, often emphasizing the role that Universal Design can play regarding the accessibility of technology to individuals with various types of disabilities. The above surveys found that:

D. Conclusions and Recommendations

There are significant gaps in what the nation can measure using current data and the issues that focus group participants identified as important to the quality of life of people with disabilities. These gaps are caused by one or more of four types of limitations inherent in the national data collection system:

Recommendation 2: Promote a standard set of disability questions. As we have noted, some important surveys, such as the American Housing Survey, the Consumer Expenditure Survey and others, have no disability measures. When measures are included, they vary across surveys, often yielding inconsistent and confusing results. A common core of disability questions on all federal surveys would improve comparability and improve the national discourse about disability programs and policy.

Each of our national surveys is designed to meet a particular goal and each uses a definition of disability that is appropriate for that purpose. The surveys should be modified to add disability questions that would allow researchers to use the surveys for a common purpose—to assess the status and progress of people with disabilities across many domains.

Identifying an appropriate set of questions is challenging. First, stakeholders must agree on an appropriate conceptualization of disability. Second, survey developers need to address the technical challenge of operationalizing the definition in a way that will yield accurate and reliable results (McMenamin et al. 2005).

The Washington City Group on Disability Statistics, an international group of disability scholars and statisticians authorized by the UN Statistical Division, recommends that, in order to achieve this goal, the definition needs to identify people “who, because of their functional limitations, are at risk for the loss or restriction of opportunities to take part in the normal life of the community on an equal level with others” (Altman 2006).

Questions aiming to identify this group should characterize people by the difficulties they have in undertaking basic activities, like walking, seeing, and self-care—and not questions asking if they have a condition which limits their ability to participate in society, for example to work or attend school. The analysis can then determine to what extent people with functional limitations are excluded from society. For this reason, it is also important that the questions ask about functioning without assistive devices, since the presence of assistive devices is one of the mechanisms by which people with functional limitations gain access to society.

Recommendation 3: Fully disseminate disability data. Federal agencies and other organizations that conduct national surveys, such as the U.S. Census Bureau of Labor Statistics and the U.S. Department of Health and Human Services Administration on Healthcare Research and Quality, should provide comparisons of people with and without disabilities in their aggregated data reports and should, where sufficient data exists, offer comparisons of people with disability by gender, race, and other socio-demographic characteristics.

Chapter 5: Key Disability Indicators

A. Introduction

In this chapter, we propose a set of topical indicators intended to measure the progress of people with disabilities in important life domains over time. Data for the 18 indicators in the proposed topical indicator set can be produced relatively easily and quickly for NCD’s annual Progress Report, and will provide a foundation for the Key National Indicators Initiative (KNII) discussed in Chapter Three. Including a topical indicator set on people with disabilities in NCD’s Progress Report offers an annual opportunity to focus on the movement of people with disabilities in relation to their goals as well as their continuing needs. Whether viewed as “keeping score” or as a “report card,” such regular reporting of numbers that can be compared over time is a technique for accountability that is likely to stimulate interest among politicians, policy-makers, the press, and the public.

We begin this chapter by identifying critical features of social indicator systems and how the proposed set of indicators addresses each feature. We then present each of the 18 proposed indicators, along with a brief rationale for selecting the measure, and the strengths and weaknesses of each.

B. Desirable Features of the Disability Topical Indicators

A number of authors have listed critical features of social indicator systems (Hagerty et al. 2001; JCCI 2006; Lane 1992). Below, we present a composite list of desirable characteristics along with commonly accepted statistical principles. We also describe how we have used these characteristics to select the proposed set of indicators.

Defined by Stakeholders

In general—The indicator set must capture the concept of “well-being,” as defined by its potential users. The set of indicators should measure quality of life based on what a diverse group of community members agrees is important, in relation to the priorities in the community’s shared vision or goals. The indicators, taken together, should approximately encompass the totality of life experience, and each indicator should encompass a substantial but discrete portion of the aggregate (Hagerty et al. 2001).

In the proposed set—To obtain stakeholder input, we held a series of focus groups whose participants identified 11 domains and several dimensions within each domain they considered important to a high quality of life. Our goal was to select at least one indicator in each of the 11 domains, however, a lack of data in several domains made achieving this goal impossible.

Objective and Subjective Measures

In general—Both subjective and objective indicators are necessary, but neither is sufficient to encompass the totality of life experiences. Many researchers have demonstrated that objective indicators of QOL often do not correlate highly with their subjective counterparts (Hagerty et al. 2001). Lane (1992) argued that, because quality of life is a function of expectations, if we focus entirely on subjective measures we risk accepting the “wantlessness of the poor and the acquiescence of the exploited.” If
we focus entirely on objective measures we might overlook important benchmarks
of well-being.

In the proposed set—Although available data on subjective measures is extremely limited, we have included both objective measures (employment rate, educational level) and subjective measures (perception of whether physicians show respect for patient opinions, presence of emotional support) among our topical indicators. Data on additional subjective measures could be included as they become available.

Measurable, Reliable and Valid

In general—Data for the indicator must be available on a regular basis from a credible source. The data must be collected, compiled, and calculated in the same way each year. The metric must be able to function as a discriminative instrument (differentiating people who have a better quality of life from those who have a worse one) as well as an evaluative instrument (measuring how much the quality of life has changed over time or in response to a particular intervention). The indicator trend line should respond relatively quickly and noticeably to changes in the quality of life. Users must be able to identify differences in scores that correspond to trivial, small, moderate, and large differences. If the measure fluctuates over time, we must be relatively sure that this corresponds with a fluctuation in the quality of life and not be a statistical artifact.

In the proposed set—For the NCD indicator set we included data that meets the following criteria:

Understandable

In general—If the indicator trend line moves upward or downward, a diverse group of people in the community should agree on whether the quality of life is improving or declining. For an example that does not fulfill this feature, if the rate of enrollment in the food stamp program increases, it is not clear whether that indicates that the program is serving a greater portion of people in need (a positive) or that the need is increasing
(a negative).

In the proposed set—There does not seem to be any ambiguity in the interpretation of potential changes in the direction of each of the proposed set of indicators.

Comprehensive

In general—Together, the set of indicators should address as many domains as possible to paint a comprehensive picture of the quality of life for people with disabilities.

In the proposed set—We have included data from as many domains identified by the focus groups as possible, given the limitations of existing data noted in Chapter Four.

Leading and Lagging Indicators

In general—Some people argue that social indicator systems should be forward thinking (how do you feel about the future?) because leading indicators enable a proactive
policy response. Others suggest that lagging indicators, those that focus on current status or past trends, should be used so that responses reflect their current situation (JCCI 2006).

In the proposed set—There is little leading indicator data for people with disabilities
that meet the criteria we have established, so we have limited our indicator set to lagging indicators.

C. The Indicators

Exhibit 5.1 presents the proposed set of indicators for working-age people with and without disabilities for annual monitoring by NCD. In the following sections, we present a brief rationale for each indicator, and discuss their respective strengths and weaknesses.

Exhibit 5.1: Indicators for Working-Age People with and without Disabilities for Annual Monitoring by NCD
Employment
Indicator 1:	Employment rate
Indicator 2:	Employment rate by educational attainment
Indicator 3:	Median annual labor earnings of full-time/full-year workers
Indicator 4:	Median annual labor earnings of full-time/full-year workers by educational status
Education
Indicator 5:	Education less than a high school diploma
Indicator 6:	Education of at least a college degree
Health status and health care
Indicator 7:	Obesity
Indicator 8:	Smoking
Indicator 9:	Health insurance status
Indicator 10:	Failure to get needed care because of cost, by poverty status
Indicator 11:	Doctors or other health providers always show respect for what the patient has to say
Financial status and security
Indicator 12:	Median household income
Indicator 13:	Poverty status
Leisure and recreation
Indicator 14:	Participation in leisure time physical activity
Personal relationships
Indicator 15:	Social and emotional support always available
Indicator 16:	Marital status
Crime and safety
Indicator 17:	Violent crimes per 1,000 people
Indicator 18:	Property crimes per 1,000 people

1. Employment

Rationale for inclusion: Employment and earnings are the measures most commonly used to assess the economic productivity of society and the well-being of individuals. Employment can promote financial well-being, including escape from poverty, and integration. Employment was identified by our focus groups as a strong indicator of social acceptance and positive personal identity.

During the last 30 years, legislation and policies related to individuals with disabilities, including the Americans with Disabilities Act, the Ticket to Work and Work Incentives Improvement Act, and the New Freedom Initiative, have emphasized increased independence and economic self-sufficiency (Stapleton et al. 2006). Growing from the social/environmental model of disability that emphasizes the role of the environment in the disability experience, these laws and programs, along with technological and other innovations, have opened up new employment opportunities. The employment rate of working-age individuals is an important measure of our progress in meeting the goals of full opportunity and inclusion embodied in this legislation.

Employment and earnings are correlated with educational attainment (Becker 1993). Our focus groups expressed the desire to find employment appropriate to their level of experiential and educational qualifications and wages equivalent to their coworkers in similar positions. While we cannot compare wages by occupation between people with and without disabilities, we can compare annual labor earnings by education level between the two groups. Therefore, we have included the employment rate and mean annual earnings by education level for full-time, full-year workers.

Indicator 3: Median Annual Labor Earnings of Working-Age People Who Work Full-Time/Full-Year

Indicator 4: Median Annual Labor Earnings for Full-Time/Full-Year Workers by Educational Attainment

Strengths, weaknesses, and alternatives

The data are reliable. These data are from the ACS, which has a large sample size and disability questions that have been tested by the U.S. Census Bureau’s cognitive questionnaire lab. See Appendix D for a fuller description and strengths and weaknesses of the ACS and other surveys.

The ACS does not allow for reporting statistics over time prior to 2005: Although the ACS has been fielded annually since 1996, data prior to 2005 are based on samples designed for testing purposes rather than the full national sample of three million households. Thus, data prior to 2005 is not directly comparable to data from 2005 forward. We considered using the employment rate and earnings from the NHIS because it would allow NCD to develop a time series. However, we rejected this alternative because the ACS includes the employment rate of people with disabilities in its tables on its website and because the Statistics RRTC monitors the ACS value. Reporting the employment rate from the NHIS would require data programming expertise, making the ACS a much more promising data source going forward.

Employment versus unemployment rate: The employment rate is defined as the number of people who were working in the two weeks before the survey (or had a job but were on vacation) divided by the total population. The denominator includes people who did not want to work at that time, e.g., because they were attending school full-time or raising children, so it may not be an accurate measure of the percent of people who want to work or who are currently looking for work.

An alternative to the employment rate is the unemployment rate, which is calculated by dividing the number of people who are “unemployed” by the number “in the labor force.” In this calculation, only people who are actively looking for a job are included in either the numerator or denominator. Thus, it excludes people who are not looking for a job either because they are discouraged and have given up, or because of work disincentives in income support policies. There is a great deal of debate about whether the employment rate or the unemployment rate is the appropriate measure of employment outcomes for people with disabilities (Stapleton and Burkhauser 2003). We have chosen the employment rate in order to capture the status of the whole population with disabilities, not just those who are actively searching for employment.

In the employment rate calculation, an individual is considered “employed” if he or she worked at all, even a very minimal amount in the two weeks prior to the survey. We considered using a measure of the percent of people who were employed full time for the full year. However, given that the flexibility of part-time or intermittent work is important for many people with disabilities, we have opted to focus on the overall employment rate.

The employment rate in any particular survey may vary from other surveys because of differences in the definitions of disability and employment used. However, finding a rise or fall in the employment rate when the same definitions are used over time is generally indicative of a societal trend (Stapleton and Burkhauser 2003).

2. Education

Rationale for inclusion: Jobs and careers that provide a living wage, benefits, and meaningful upward mobility are generally predicated on high levels of educational attainment. Focus group participants mentioned other inherent values of education for improving quality of life, such as expanding learning and cultural experiences, expanding social contacts and promoting integration.

For persons with disabilities, a stronger positive correlation is demonstrated between level of education and rate of employment than is found in the general population (Stodden 2002). Lower educational attainment places students with disabilities at greater risk of unemployment or underemployment. Therefore, we include indicators to measure level of educational attainment.

Indicator 5: Percentage of Working-Age People with Less Than a High School Diploma

Indicator 6: Percentage of Working-Age People with at Least a College Degree

Strengths, weaknesses, and alternatives

We would like to include the high school graduation rate as an indicator. However, there are multiple methods used to calculate the graduation rate (Seastrom et al. 2006) and, as noted previously, it is not possible to compare graduation rates of students with and without disabilities.

The focus groups identified integration into mainstream educational opportunities as an important priority. Healthy People 2010 monitors progress toward this goal with the percent of special education students who are included in the regular classroom more than 80 percent of the day. We chose not to include this measure because it has several important limitations:

3. Health Status and Health Care

Rationale for inclusion: Americans are becoming increasingly concerned about their own health and the quality of the health care system that serves them. The health of people with disabilities can range from excellent to poor. The health status of people with disabilities in part reflects the quality of the health care system and health care prevention programs. We include obesity as an indicator not only because it is a leading cause of preventable chronic disease, but because it also has its own disabling effects and causes or contributes to other disabling conditions such as arthritis, diabetes, and cardiovascular disease. In addition to causing disability, obesity is also positively associated with having a disability. It is a leading cause of secondary disabling conditions (Weil et al. 2002) that can be more dangerous for people with disabilities than for others (Iezzoni and O’Day 2006). Obesity can interfere with work, maintaining independence and participating in community activities.

The dangers of smoking are also well established. People with disabilities have higher rates of smoking than the general public, yet doctors query people with disabilities less often about smoking and other risky behaviors (Iezzoni et al. 2001; Iezzoni and O’Day 2006).

Health insurance coverage is linked to the quality of care individuals receive. For those without health insurance, chronic conditions are often neglected or poorly managed medically, further worsening disease and disability (Institute of Medicine 2004). People with and without disabilities are insured at similar rates, however, people with disabilities tend to rely more heavily on the public health insurance programs (Medicare and Medicaid).

People with disabilities also report that affordability is a key component of quality health care, irrespective of whether they have insurance coverage, because their needs often outstrip the benefits allowed (Iezzoni and O’Day 2006). We therefore recommend an indicator comparing individuals above and below the federal poverty level who fail to get needed care because of cost.

Respect between health care provider and patient is crucial to the strong and trusting relationship necessary for high quality health care. Doctors are often the gatekeepers who provide access to specialty care, prescriptions for adaptive equipment, and document eligibility for social and financial services. Failure to take time to listen to patients’ concerns can damage the relationship and lead to poor clinical outcomes (Iezzoni and O’Day 2006). Because people with disabilities tend to use the health care system frequently, subjective measures as to whether doctors show respect and listen to what they have to say may be an important indicator of quality health care.

Indicator 10: People Who Failed to Get Needed Care Because of Cost, by Poverty Status

Indicator 11: Patients Who Report That Doctors or Other Health Providers Always Show Respect for What They Have to Say

Strengths, weaknesses, and alternatives

The data are valid and reliable: Data from the Behavioral Risk Factor Surveillance Survey (BRFSS) and the National Health Interview Survey (NHIS) have adequate sample sizes, strong disability measures, and are reported in Healthy People 2010. Medical Expenditure Panel Survey (MEPS) data drawn from a nationally representative subsample of households that participated in the prior year’s National Health Interview Survey. (See Appendix C for a fuller description of these surveys.)

Measures: The NHIS has three measures of lack of health insurance coverage: currently (uninsured at the time of interview), intermittent (uninsured at least part of the 12 months prior to interview), and long term (uninsured for more than a year at the time of interview). We chose the intermittent measure because the optimal situation is to have continuous coverage of the population. People who have only intermittent coverage are at risk for inconsistent and lower quality health care; those who do not currently have health care will be included in this measure.

The MEPS is the predominant source of data on health care costs (total and out-of-pocket). However, the Agency for Health Care Research and Quality (AHRQ) does not include disability status as a subgroup characteristic in either its regularly produced tables or as an option in its extensive MEPS data retrieval system. Unlike the data for indicator 10 (doctors or other health providers always show respect for what patients have to say), which is aggregated and reported regularly by Healthy People 2010, no entity regularly reports health care costs for people with and without disabilities using data from the MEPS. The public use micro data are difficult to use and analyses of those data are beyond the scope of this project.

4. Financial Status and Security

Rationale: It is often said that, “money does not buy happiness.” But few would dispute that there is a relationship between an adequate income and quality of life. Poverty rates for people with disabilities are at least twice as high as for people without disabilities (StatsRRTC 2005). Poverty rates are high despite the fact that almost nine million working-age adults with disabilities receive income support from the Social Security Disability Insurance (SSDI) and Supplemental Security Income (SSI) programs.

The presence of a disability can affect the financial status of the entire household, not just the person with a disability. The poverty rate measure is based on household income. We recommend indicators comparing the household incomes of people with and without disabilities and the percentage of households below the poverty level.

Strengths, weaknesses, and alternatives

The poverty rates reflect the official definition used to compute the federal poverty thresholds for families of various sizes and age composition. The official definition does not take into account the additional expenses many people with disabilities face, such as accessible housing or assistive technology (She and Livermore 2006).

5. Leisure and Recreation and Personal Relationships

Rationale: Focus group participants stressed the importance of participation in leisure and recreation activities, either as individuals on their own or with other members of their communities. However, there is little data on this topic that meets the criteria outlined in Section B above. The only relevant data we can identify is available from the NHIS. These data measure participation in leisure time physical activity. Because physical activity is so important to an individual’s overall health, we propose the NHIS measure as one measure of recreation.

Focus group participants stressed the importance of personal relationships in obtaining a high quality of life. We have selected two measures in this domain: marriage and obtaining needed social and emotional support.

Focus group participants said they aspired to strong, close, emotionally supportive relationships with people with and without disabilities, or said these relationships were essential for their well-being. Many also aspired to getting married and raising children. A person with a disability who is in a committed relationship is more likely to get informal support that is limited or unavailable from formal government programs. Such support can be more acceptable to the person with a disability, delivered more conveniently and efficiently, and can cost less, than formal support. Additionally, couples can share household and other chores based upon their abilities, e.g., a person who has difficulty going outside the home can handle cooking and cleaning, leaving the shopping to the partner. Economies of scale can also be achieved by living together—sharing rent and food—that make living together more economical than living separately.

Indicator 15: Always or Usually Get the Social and Emotional Support You Need

Strengths, weaknesses, and alternatives

Participation in leisure time physical activity measures participation in activities such as exercise, sports or physically active hobbies. It is not a strong indicator of participation in a wide range of leisure and recreation activities that may not involve physical exertion.

The marriage measure we have selected may not be a good proxy for strong personal relationships for two reasons: First, the strength of the spousal relationship varies among couples; some are strong and emotionally supportive and some are not. Second, the marriage indicator does not reflect relationships of couples who provide each other with significant emotional support but are not married. This issue may be particularly significant among people with disabilities who may choose not to marry to avoid losing SSI benefits. For example, two individuals living on SSI will receive a
lower payment as a married couple than their combined individual SSI payments. Individuals on SSI with Medicaid coverage are likely to lose these benefits if they marry a working spouse.

6. Crime and Safety

Rationale: Focus group participants mentioned personal safety as a pertinent dimension of several domains, including housing (living in a safe area), transportation (waiting for public transportation at night or in isolated areas), and personal assistance (being victimized by an assistant).

The National Crime Victimization Survey, conducted by the Department of Justice, Bureau of Justice Statistics, has included a disability indicator similar to the ACS indicator on their National Crime Victim Survey, as of January 2007. Data on this indicator will be released in late 2008, based on the data collected in 2007.

Indicator 17: Violent Crimes per 1,000 People With and Without Disabilities.
(Violent crime is the sum of rape, robbery, simple and aggregated assault)
Data Source: National Crime Victimization Survey.

Indicator 18: Property Crimes per 1,000 People With and Without Disabilities.
Data Source: National Crime Victimization Survey.

Strengths, weaknesses, and alternatives

The National Crime Victimization Survey has a large sample size and the new disability questions seem well designed. As a result, the survey has the potential to fill a gap in our knowledge of the prevalence of victimization among people with disabilities.

7. Domains for Which There Are No Indicators

Our focus groups identified other domains, including community participation, political participation, housing, transportation, and technology. We were unable to identify data in these domains that met the criteria we outlined in Section B above. We briefly summarize the data limitations in each of these domains below. Appendix D contains a fuller description of available data and limitations.

Community participation: The limited data we have on community participation is generally available from surveys that are conducted only sporadically or conducted by non-governmental entities such as NOD/Harris.

Political participation: Most of the information we have on political participation of people with disabilities comes from private, non-governmental surveys such as the NOD/Harris survey and the Pew Research Center for the People and the Press. The CPS is the only large, ongoing national survey that includes questions about political participation. The CPS has a core survey and monthly supplemental surveys. Each year, the November supplement queries about recent voting behavior. A different supplemental survey (the March supplement) asks whether the respondent has a “health problem or disability which prevents them from working or which limits the kind or amount of work they can do.” This question from the March supplement is the only way to define disability in the CPS. In order to calculate the voting behavior of people with disabilities compared to those without disabilities, an analyst needs to merge the two supplements. However, because of the sampling frames, only about one-third of the people in the March Supplement are asked the question from the November supplement. Thus, reporting voting behavior using the CPS violates two of the criteria identified above. First, disability is defined based on work limitation and second, retrieving the data is not easy. Both issues can be addressed by including additional questions about disability on the CPS and administering those questions in the core survey rather than in a supplement.

Transportation: There is no ongoing measure of transportation use by people with disabilities or their perceptions of available transportation. The only source of information on this topic, the Department of Transportation Bureau of Transportation Statistics 2002 National Transportation Availability and Use Survey, has not
been repeated.

Housing: We considered using the ACS measure of the number of individuals with disabilities living in congregate care facilities as a proxy measure of integration. However, we do not believe the measure provides unambiguous information about quality of life. Although most would agree that living in a large facility is undesirable, smaller group homes, also defined as congregate care facilities, often provide high quality care and may be more desirable than other options, such as living with aging parents. In addition, living in a congregate care facility is an improvement over homelessness. We also considered the ACS measure of home ownership, but found it to be problematic. The survey asks if the respondent lives in an owner-occupied dwelling, which does not accurately measure home ownership.

Technology: Every two years the CPS includes questions on one of the monthly supplements about computer ownership, internet access, and internet use. Using these data to track access to technology for people with disabilities has the same limitations as those noted above for voting behavior.

In this chapter, we have proposed and illustrated 18 societal indicators spanning seven domains of importance to people with disabilities—employment, education, health and health care, financial security, leisure and recreation, personal relationships, and crime and safety. While one domain (leisure and recreation) had only one indicator, all others have at least two. Our illustration of each indicator used only one recent year of data. For the future, at least two time points would be shown for each indicator, providing evidence of trends in direction.

The chapter also highlighted the strengths and weaknesses of the proposed measures; further discussion is provided in the Appendices to this report. Finally, we identified five important domains for which no adequate indicator can be proposed at this time. In each area, there is clear potential for development of one or more status indicators, given sufficient interest by policy-makers.

From this promising beginning toward a comprehensive set of indicators to monitor the status of people with disabilities, we turn next to focus on Program Performance Indicators.

Chapter 6: Program Performance Indicators

Most current program performance indicators measure the effect of programs on participants. They generally do not measure the number of people who could benefit from the program but do not participate either because they do not meet specific eligibility requirements, are unaware that the program exists, or do not use the program because of other entry barriers such as inaccessibility or excessive red tape. There are a few exceptions. The food stamp program monitors the participation rates of eligible households, and the chronic disease prevention programs monitor the national rate of chronic diseases.

A second limitation of program performance indicators is that they do not provide enough information to evaluate the impact of the program. They do not account for the complex interactions between the program and other factors such as the economy or other programs. For example, the percentage of vocational rehabilitation clients obtaining competitive jobs at closure partly depends on the quality of the services delivered, but also depends on the characteristics of the users, the availability of jobs in the economy, and the barriers and disincentives to work created by Social Security Disability Insurance and other cash and medical benefits.

Because of the limitations of performance measures for evaluation purposes, many agencies conduct specific program evaluations to identify opportunities to improve the program, ascertain the extent of the program’s effectiveness, increase the public’s general knowledge about the program or topic, or guide decisions on resource allocation within the program (GAO 1998). Performance measurement focuses on whether certain objectives have been achieved, but cannot determine the extent to which the program contributed to that achievement.

Program evaluations typically use comparison groups and empirical methods to assess the “counterfactual”—the extent to which the objectives would be achieved in the absence of the program or a program feature. A full evaluation requires an examination of a broader range of information on program performance and program context, as well as a comparison or control group, or some other method to assess the counterfactual. Comparison of actual to counterfactual performance measures can play a central role in a program evaluation, but actual performance measures on their own generally do not provide enough information to assess the value of the program itself. Program evaluations are generally conducted sporadically because they are too expensive and technically demanding to conduct on a routine basis (GAO 2005).

A. Existing Program Performance Measurement Systems

The government uses two performance measurement systems: The Government Performance and Results Act of 1993 (GPRA) and the Office of Management and Budget’s Program Assessment Rating Tool (PART). GPRA mandates that each federal agency develop a strategic plan with outcome-related goals and objectives; develop annual performance plans with output-and outcome-related goals; and report annually on progress toward achievement of these. GPRA was criticized for allowing agencies to focus on outputs (the quantity of the product or service provided) rather than outcomes (the results of providing the service), which led OMB to insist on outcome measures
for PART.

PART assesses whether agencies focus upon achievable outcomes based upon long-term goals and benchmarks and mandates that agencies develop an improvement plan to address deficits. The PART process requires agencies to answer 26 questions about the program purpose and design, strategic planning, program management, and program results and to develop and report quantifiable indicators. Generally, agencies report between one and twelve outcome measures, several efficiency measures, and occasionally several output measures. OMB makes these evaluations publicly available at http://www.whitehouse.gov/omb/expectmore/.

GAO recently conducted a review of 200 federal programs located in 20 agencies that serve individuals with disabilities. About half of these programs serve only people with disabilities, while the rest have eligibility criteria other than disability (GAO 2005). Focusing on the outcome measures in the PART evaluations of 40 programs (Appendix F), half of which serve only people with disabilities and half of which serve people with disabilities within a broader population, we found that the two types of programs had significant limitations in how outcomes for people with disabilities were measured.

Indicators for programs serving only people with disabilities: Outcomes of disability programs tend to be evaluated based on a narrow range of indicators that are generally inadequate to assess the true value of the program because they do not adequately capture the program’s impact on the quality of people’s lives. For example, employment and training programs use the following “job training common measures:”

While these measures are a great improvement over simply recording whether a program’s client entered employment at one point in time, they are limited. The measures do not capture dimensions of employment that the focus groups identified as important, such as whether the job is appropriate to their aspirations and long term goals or is considered to be worthwhile; whether it uses their full talents and abilities; whether the wages are appropriate given their qualifications; the extent to which they are satisfied with job conditions (including place, facility, co-workers, schedule requirements, accommodations, and opportunities for advancement); and whether they had meaningful opportunities to make choices about the conditions of their work.

Education programs tend to focus on rates of retention, graduation, and subsequent employment. Although those measures are a reasonable proxy for the quality of education, they do not address the issue of choice or affordability.

While these measures do not directly quantify any of the quality of life measures identified in the focus groups, the first two measure the extent to which the client is being treated with dignity and the third measures integration, all of which were identified as important dimensions by the focus groups.

One other program, the Assistive Technology Alternative Financing Program (AFP), has formally recognized the importance of measuring the program’s impact on quality of life. They have not yet established those measures, “…but a web-based outcomes reporting system is being developed through the AFP technical assistance grant. This system will collect information to address the program’s purpose, including data on how AFP loans have helped transform the lives of people with disabilities in employment, education and independent living” (OMB 2004). Although this is a laudable goal, outcome indicators alone will be insufficient to establish the impact of the program on people’s lives.
As described above, to do so requires some method for estimating the
counterfactual (OMB2004).

Indicators for programs that serve people with disabilities within a broader population: Outcome indicators tend to report the outcomes of all program enrollees rather than for different socio-demographic subgroups of enrollees. They do not allow a program to determine the specific impact on people with disabilities. While some means-test programs, such as Medicaid and the Food Stamp Program, have special eligibility categories for people with disabilities and monitor the enrollment of people in these categories, many programs do not include any measure of disability in their administrative records. For example, Temporary Assistance to Needy Families (TANF) does not track recipients with disabilities, even though there is extensive evidence that a large percentage of TANF families include parents and/or children with some disabilities (Zedlewski 1999; GAO 2002). Disability status might be an important determinant of TANF eligibility and enrollment. Disability status may also have an impact on ability to find work and transition out of the program. Although there have been non-agency efforts to look at disability prevalence among TANF recipients (Zedlewski 1999; GAO 2002), our understanding of the experiences of people with disabilities in TANF is limited because state programs and the federal Agency for Children and Families do not routinely monitor this subpopulation.

B. Integrating Quality of Life Concepts into Program Performance Measures

Current performance measures used for PART and GPRA do not address the dimensions identified by our focus groups such as choice, empowerment, and aspirations. However, there are several emerging initiatives that use some of these dimensions to monitor and improve the quality of providers and small programs. We mention them here because they provide a model of the type of issues that could be included in program performance measurement.

The most promising practice to integrate quality of life concepts into program performance measures is from the Centers for Medicare and Medicaid Services (CMS), in collaboration with the National Association of State Directors of Developmental Disabilities Services (NASDDDS) and the Human Services Research Institute (HSRI).

CMS developed the HCBS Quality Framework, which focuses attention on outcomes of home and community-based services that CMS believes are important, such as person- centered planning and delivery; supporting participants in the exercise of their rights; and participant access to services. The framework is not regulatory; states have the option to implement it as they wish.

NASDDDS, in conjunction with HSRI, has converted these concepts into National Core Indicators (NCI). The current set of performance indicators includes approximately 100 consumer, family, systemic, cost, and health and safety outcomes. Sources of information include consumer surveys (with questions addressing empowerment and choice issues) family surveys (with questions addressing satisfaction with supports), provider surveys (with questions addressing staff turnover), and state systems data (with questions addressing expenditures and mortality). Currently, 23 states and two sub-state developmental disability agencies are using the indicators. Using data from the consumer survey, the NCI coalition computes statistics for each state, which allows for a comparison among states (HSRI and NASDDS 2006). Outcome measures include the following: (A full list is shown in Appendix G.)

The major barrier to using outcome measures of this type for program monitoring is that the only way to collect this type of information is through lengthy surveys with program participants, which is usually prohibitively expensive to do on a regular basis and could impose a significant burden on respondents.

C. Conclusions and Recommendations

Recommendation 4: Administrative records of all means-tested programs should include a disability indicator. Programs that serve individuals with disabilities, such as One-Stop Employment Centers and TANF, should collect data on the number of individuals with disabilities who use their programs and compare outcomes between program users with and without disabilities. We recommend that the Interagency Committee on Disability Research (ICDR) develop a workgroup to establish criteria on which the indicator is based.

Chapter 7: Conclusions and Recommendations

This report describes what we know about the status of people with disabilities in our society and examines current data to ascertain whether it meaningfully measures their well-being. While much useful data on the status of people with disabilities exists, significant knowledge gaps hamper efforts to improve their well-being. Additionally, social indicator systems currently used or being developed do not provide specific information about people with disabilities; therefore, no comprehensive report exists on the status of people with disabilities.

This report proposes a set of statistical indicators to annually measure the status of working-age people with disabilities, and compare them to working-age people without disabilities in the United States. The topical indicator set, which can be produced relatively quickly, can provide a foundation for the social indicator systems being developed and stimulate public awareness about the status of people with disabilities and the need for improvement.

Recommendations fall into two categories: social indicators and program indicators.

Social Indicators: Social indicator systems are one of the chief mechanisms for assessing group status in our society. Although social data exist on many important aspects of daily life, there is no comprehensive indicator system at the federal level that coherently describes our social condition.

In 2003, the Government Accountability Office, in cooperation with the National Academies, began the Key National Indicators Initiative (KNII) to develop a comprehensive indicator system for the country as a whole. The goal of the initiative is to assess the progress of our nation in areas that are important to the citizenry. They are using the wealth of existing data from a variety of federal sources to develop a website with data on 30 “key” indicators and several hundred secondary indicators.

Unfortunately, many of the surveys the KNII will use to collect social data do not contain adequate measures to define disability or meaningful measures for program outcomes. Some existing surveys probe deeply into certain life domains and are, in principle, useful in measuring the well-being of people with disabilities. But they are constrained by the conceptual and methodological limitations of their disability questions and do not probe into areas of life important to focus group participants. Existing social measurement systems generally do not include measures that would enable us to identify people with disabilities in a consistent and meaningful fashion, or evaluate their quality of life. Current data cannot support a complete, holistic set of indicators. This lack of data makes monitoring and evaluating the impact of laws, policies and programs on people with disabilities impossible, which in turn undermines our ability to make informed programmatic and funding decisions.

Data on individuals with disabilities has various limitations that impact its usefulness in measuring quality of life. The definition of disability is quite complex and varies among different federal and state laws, public programs, insurance plans, and organization, which means that surveys do not all use the same definition. Some of the survey definitions are particularly problematic, most notably the “work limitation” definition. Answers are likely to be sensitive to characteristics of the economic or physical environment and are likely to differ among individuals with different circumstances and the same level of impairment.

Another problem is that, although people from diverse racial or ethnic backgrounds constitute a disproportionate share of the disability community and sometimes have a unique set of needs, they are often not adequately represented in surveys. Many large surveys, where it is likely to be possible to describe this subpopulation statistically, do not cross-tabulate race-ethnicity with disability status. A final problem is that there are few surveys with the same question, the same disability indicators and the same sampling methodology across time that do not use the problematic work definition discussed above, so it is difficult to track progress over time.

Program Performance Indicators: Programs that focus on individuals with disabilities, as well as programs that serve individuals with disabilities within a broader population, have significant limitations in how outcomes for people with disabilities are measured. In programs serving a broader population, outcome indicators tend to report the outcomes of all program enrollees rather than of different demographic subsets of enrollees. Thus, they do not allow a program to determine the specific impact on people with disabilities. Outcome indicators for programs focusing on people with disabilities, particularly employment programs, are evaluated based on a narrow range of indicators that do not adequately capture the program’s impact on the quality of people’s lives.

Recommendation 1: NCD recommends that the National Institute on Disability and Rehabilitation Research (NIDRR) establish and fund a coalition of disability policy-makers and advocates to: 1) develop a fuller set of indicators that are important to people with disabilities, building on the indicators outlined in this report (see Chapter 5); and 2) ensure that disability is included as a subgroup characteristic as the KNII is developed. The KNII offers an important opportunity to integrate disability into a larger national indicator system. When completed, the KNII will offer individuals who are looking for disability data relatively easy access to the data. It will highlight the importance of including disability as a subgroup in analyzing the relative status and progress of the population and highlight gaps in data about people with disabilities.

Recommendation 3: Fully disseminate disability data. Federal agencies and other organizations that conduct national surveys, such as the US Census Bureau of Labor Statistics and the U.S. Department of Health and Human Services Administration on Healthcare Research and Quality, should provide comparisons of people with and without disabilities in their aggregated data reports and should, where sufficient data exists, offer comparisons of people with disability by gender, race, and other socio-demographic characteristics.

Recommendation 4: Administrative records of all means-tested programs should include a disability indicator. Programs that serve individuals with disabilities, such as One-Stop Employment Centers, and TANF, should collect data on the number of individuals with disabilities who use their programs and compare outcomes between program users with and without disabilities. We recommend that the Interagency Committee on Disability Research (ICDR) develop a workgroup to establish criteria on which the indicator is based.

References

Altman, B (2006). Measuring disability in a survey or census context: Parallel work advancing the field. Presentation at2006 StatsRRTC State-of-the-Science Conference: The future of disability statistics: What we know and need to know. Washington, DC. October 5, 2006.

Becker, G.S. (1993). Human capital: A theoretical and empirical analysis, with special reference to education. University of Chicago Press.

Bennett, C., Lenihan, D.G., Williams, J. and Young, W. (2001). Measuring quality of life: The use of societal outcomes by parliamentarians. Ottawa, Canada: KTA Centre for Collaborative Government.

Burkhauser, R.V., Daly, M., Houtenville, A., and Nargis, N. (2002). Self-reported work limitation data: What they can and cannot tell us. Demography, 39(3), 541-555.

Carlson, D., and Ehrlich, N. (2005). Assistive technology and information technology use and need by persons with disabilities in the United States, 2001. U.S. Department of Education, National Institute on Disability and Rehabilitation Research. Retrieved May, 2007, from http://www.ed.gov/rschstat/research/pubs/at-use/at-use-2001.doc

Coutinho, M.J, Oswald, D.P., and Best, A.M. (2002, February). The influence of socio- demographics and gender on the disproportionate identification of minority students as having learning disabilities. Remedial and Special Education, 23(1), 49-59.

Cummins, R.A, Eckersley, R., Pallant, J., Van Vugt, J., Misajon, R. (2005). Developing a national index of subjective wellbeing: The Australian Unity Wellbeing Index. Social Indicators Research. 64(2),159-190.

Diener, E. and Suh, E. (1997). Measuring quality of life, economic, social, and subjective indicators. Social Indicators Research, 40, 189–216.

General Accounting Office (1998, April). Program evaluation, agencies challenged by new demand for information on program results (Publication No. GAO/GGD-98-53).

General Accounting Office (2002, July). Welfare reform, outcomes for TANF recipients with impairment (Publication No. GAO-02-884). Retrieved May, 2007, from http://www.gao.gov/new.items/d02884.pdf

General Accounting Office (2003, May). Forum on key national indicators (Publication No. GAO 03-672SP).

Government Accountability Office (2004, November). Informing our nation: Improving how to understand and assess the USA’s position and progress (Publication No.
GAO-05-01).

Government Accountability Office (2005). Federal Disability Assistance: Wide Array of Programs Needs to be Examined in Light of 21st Century Challenges (Publication No. GAO-05-626). Retrieved March, 25, 2007, from http://www.gao.gov/new.items/d05626.pdf

Government Accountability Office (2005b, October). Performance Budgeting, PART focuses attention on program performance but more can be done to engage Congress (Publication No. GAO-06-28).

Hagerty, M.R., Cummins, R.A., Ferriss, A.L., Land, K., Michalos, A.C., Peterson, M., et al. (2001). Quality of life indexes for national policy: review and agenda for research. Social Indicators Research, 55, 1–96.

Hale, T. W. (2001). The lack of a disability measure in today’s Current Population Survey. Monthly Labor Review, 124(6), 38–40.

Harris Interactive, Inc. (2000). 2000 N.O.D./Harris survey of Americans with disabilities. New York: Author.

Harris Interactive, Inc. (2004). 2004 N.O.D./Harris survey of Americans with disabilities. New York: Author.

Healthy People 2010 (2007, April). Data 2010...the Healthy People 2010 Database. Center for Disease Control and Prevention. Retrieved July, 2007, from http://wonder.cdc.gov/data2010/

Hoenig, C. (2007b, May 18th). Informing society in the 21st Century. Presentation to the National Academy of Public Administration. Washington, DC.

Hosp, J.L, Reschly, D. (2003). Referral rates for intervention or assessment: A meta-analysis of racial differences. Journal of Special Education, 37.

Human Services Research Institute (HSRI), National Association of State Directors of Developmental Disabilities Services (NASDDS). (2006, May). Consumer outcomes phase VII final report fiscal year 2004–2005 data. http://www.hsri.org/docs/786_P7_CS_05_final.pdf

Iezzoni, L.I., Davis, R.B., Soukup, and O’Day, B. (2003). Quality dimensions that most concern people with physical and sensory disabilities. Archives of Internal Medicine,163, 2085-2092.

Iezzoni, L.I., McCarthy, E.P., Davis, R.P., Harris-David, L., and O’Day, B.L. (2001). Use of screening and preventive services among women with disabilities. American Journal of Medical Quality, 16(4), 135-143.

Iezzoni, L.I., and O’Day, B.O. (2006). More than Ramps: A Guide to Improving Health Care Quality and Access for People with Disabilities. Oxford University Press.

Institute of Medicine. (2004, January). Insuring America’s Health: Principles and recommendations. Washington DC: IOM. Retrieved July 10, 2007 http://www.iom.edu/CMS/3809/4660/17632.aspx

Laidlaw, S. (2007, February 19). A richer way of measuring wealth: New well-being index would complement traditional GDP. Toronto Star. Retrieved May, 2007, from The Atkinson Charitable Foundation website, http://www.atkinsonfoundation.ca/updates/Document_1171911881843

Lane, R. E. (1992). The market experience. Cambridge, England: Cambridge University Press.

Loprest, P., and Wittenburg, D. (2005, May). Choices, challenges, and options: Child SSI recipients preparing for the transition to adult life. The Urban Institute.

Madans, J.H., Altman, B.M. and Rasch, E.K. Mbogoni, M., Synneborn, M., Banda, J. Me, A., and DePalma, E. (2004). Washington Group position paper: Proposed purpose of an internationally comparable general disability measure. Retrieved July 25, 2007 http://www.cdc.gov/nchs/about/otheract/citygroup/WG_purpose_paper.doc

McCormick, B.P. (2000). People with disabilities—National Survey of Recreation and the Environment. National Center on Accessibility. Retrieved April 30, 2007 from http://www.ncaonline.org/rec-leisure/nsre.shtml

McMenamin, T.M., Hale, T., Kruse, D., and Kim, H. (2005, August). Designing questions to identify people with disabilities in labor force surveys: The effort to measure the employment level of adults with disabilities in the CPS. Bureau of Labor Statistics. Retrieved July 25, 2007 from http://www.bls.gov/ore/pdf/st050190.pdf

Michalski, J.H. (2002). Quality of life in Canada: a citizen’s report card. Ottawa, ON: Canadian Policy Research Networks.

Miringoff, M., Miringoff, M.L., and Opdycke, S. (2003). Social indicators: What we need to make them count. U.S. General Accounting Office. Retrieved, May 2006, from http://www.gao.gov/npi/sicount.pdf

Mont, D. (2007). Measuring disability prevalence (Social protection discussion paper no. 0706). The World Bank.

National Council on Disability (1993, April 26). Meeting the unique needs of minorities with disabilities: A report to the president and the congress. Washington, DC. Retrieved May, 2006, from http://www.ncd.gov/newsroom/publications/1993/minority.htm

National Postsecondary Student Aid Study (2006). Profile of undergraduates in U.S. postsecondary education institutions: 2003-04. Retrieved May, 2007, from the National Center for Education Statistics website, http://nces.ed.gov/pubs2006/2006184.pdf

National Urban League (2007). The State of Black America 2007: Portrait of the Black Male. Silver Spring, MD: Beckham Publications Group.

Rehabilitation Research and Training Center on Disability Demographics and Statistics (StatsRRTC). (2005). 2005 Disability status reports. Ithaca, NY: Cornell University.

Russell, B. (1945). A history of western philosophy and its connection with political and social circumstances from the earliest times to the present day. New York: Simon and Schuster.

Russell, J.N., Hendershot, G.E., LeClere, F., and Howie, L.J. (1997, Nov 13). Trends and differential use of assistive technology (No. 292). Hyatsville, MD: Center for Disease Control and Prevention, National Center for Health Statistics.

Seastrom, M., Chapman, C., Stillwell, R., McGrath, D., Peltola, P., Dinkes, R., and Xu, Z. (2006). User’s guide to computing high school graduation rates, Volume 1: Review of current and proposed graduation indicators. National Center for Education Statistics.

She, P. and Livermore. G. (2006). Material hardship, disability, and poverty among working-age adults. Cornell University: Rehabilitation Research and Training Center for Economic Research on Employment Policy for Persons with Disabilities: Research Brief. Retrieved July 15, 2007 from http://digitalcommons.ilr.cornell.edu/edicollect/1223/

Silverstein, R., Julnes, G., and Nolan, R. (2005). What policymakers need and must demand from research regarding the employment rate of persons with disabilities. Behavioral Sciences and the Law, 23, 399-448.

Stapleton, D.C., and Burkhauser, R. (Eds.). (2003). The decline in employment of people with disabilities: A policy puzzle. Kalamazoo, MI : W.E. Upton Institute.

Stapleton, D.C., O’Day, B.L., Livermore, G.A., and Imparato, A.J. (2006) Dismantling the poverty trap: disability policy for the twenty-first century. The Milbank Quarterly, 84(4), 701-732.

Stapleton, D.C., She, P., and Livermore, G. (forthcoming). Options for improving the National Disability Data System. Working Paper. Ithaca, NY: Cornell University RRTC on Disability Statistics and Demographics.

Stodden, R.A. (2002, July 8). A framework for sharing research findings and implications. In National Summit Proceedings: Supporting individuals with disabilities across secondary, postsecondary education, and employment. Conference sponsored by the National Center for the Study of Postsecondary Educational University of Hawai`i at Manoa.

U.S. Census Bureau (2003). Characteristics of the Civilian Non-institutionalized Population by Age, Disability Status, and Type of Disability 2000.

U.S. Census Bureau (2006). American Housing Survey for the United States: 2005 (Report No. H150/05). Washington, DC: U.S. Government Printing Office. Retrieved May, 2007, from http://www.census.gov/prod/2006pubs/h150-05.pdf

Walker, D.M. (2007). Improving performance, transparency, and accountability through use of key national indicators. Presentation to OECD workshop on indicators for developing, monitoring, and analyzing agri-environmental policies. March 19, 2007. (GAP-07-642CG).

Weil, E., Wachterman, M., Davis, R.B., O’Day, B., Iezzoni, L.I., and Wee, C.C. (2002). Obesity among adults with disabling conditions. Journal of the American Medical Association, 288, 1265-1268.

Westat (2006a, October). OSEP data fact sheet: Environments. U.S. Department of Education: Office of Special Education Program. Retrieved July 26, 2007 from https://www.ideadata.org/docs/bfactsheetedenvirs.pdf

Westat (2006b, October). OSEP IDEA, Part B data collection. U.S. Department of Education: Office of Special Education Program. Retrieved July 26, 2007 from https://www.ideadata.org/docs/bdatahistory.pdf

Wilson, W. J. (2003). Jobless ghettos: The social implications of the disappearance of work in segregated neighborhoods. In T.E. Ore (Ed.) The social construction of difference and inequality: Race, class, gender, and sexuality (2nd Edition). McGraw-Hill.

Zedlewski, S.R. (1999). Work activity and obstacles to work among TANF Recipients. Washington DC: The Urban Institute. Accessed July 15, 2007 from http://www.urban.org/url.cfm?ID=309091

Appendix

Mission of the National Council on Disability

Overview and Purpose

The National Council on Disability (NCD) is an independent federal agency with 15 members appointed by the President of the United States and confirmed by the U.S. Senate. The purpose of NCD is to promote policies, programs, practices, and procedures that guarantee equal opportunity for all individuals with disabilities regardless of the nature or significance of the disability and to empower individuals with disabilities to achieve economic self-sufficiency, independent living, and inclusion and integration into all aspects of society.

Specific Duties

International

In 1995, NCD was designated by the Department of State to be the U.S. government’s official contact point for disability issues. Specifically, NCD interacts with the
special rapporteur of the United Nations Commission for Social Development on disability matters.

Consumers Served and Current Activities

Although many government agencies deal with issues and programs affecting people with disabilities, NCD is the only federal agency charged with addressing, analyzing, and making recommendations on issues of public policy that affect people with disabilities regardless of age, disability type, perceived employment potential, economic need, specific functional ability, veteran status, or other individual circumstance. NCD recognizes its unique opportunity to facilitate independent living, community integration, and employment opportunities for people with disabilities by ensuring an informed and coordinated approach to addressing the concerns of people with disabilities and eliminating barriers to their active participation in community and family life.

NCD plays a major role in developing disability policy in America. In fact, NCD originally proposed what eventually became ADA. NCD’s present list of key issues includes education, transportation, emergency preparedness, international disability rights, employment, foster youth with disabilities, vocational rehabilitation, livable communities, and crime victims with disabilities.

Statutory History

NCD was established in 1978 as an advisory board within the Department of Education (P.L. 95-602). The Rehabilitation Act Amendments of 1984 (P.L. 98-221) transformed NCD into an independent agency.


		Home \| FAQs \| Newsroom \| Site Map \| Federal Entities \| Resources
		Authorizing Statute \| Web Accessibility \| Information Quality \| Freedom of Information \| Research Opportunities
		Privacy Notice: The National Council on Disability (NCD) will collect no personal information about you when you visit its website unless you choose to provide that information. The only information NCD automatically collects is the visitor's Internet domain and Internet Protocol address, the type of browser and operating system used to access the site, the file visited and the time spent in each file, and the time and date of the visit.


	Enter a search topic:

Keeping Track: National Disability Status and Program Performance Indicators

Letter of Transmittal

National Council on Disability Members and Staff

Members

Staff

Acknowledgments

TABLE OF CONTENTS

Executive Summary

Chapter 1: Introduction

Chapter 2: Framework for Discussion of Social Reporting Systems

Chapter 3: Existing and Emerging Societal Indicators

Chapter 4: The Status of People with Disabilities

Chapter 5: Key Disability Indicators

Chapter 6: Program Performance Indicators

Chapter 7: Conclusions and Recommendations

References

Appendix

Executive Summary

Social Indicators

The Status of People with Disabilities

Program Performance Measures

Chapter 1: Introduction

A. Need for the National Disability Performance and Indicators Project

B. Current Measurement Systems

C. Organization of the Report

Chapter 2: Framework for Discussion of Social Reporting Systems

A. Concepts and Definitions

B. Brief History of Social Reporting Initiatives in the United States

C. Objectives of Societal Indicator Systems

D. Quality-of-Life Measures and Social Indicators

Chapter 3: Existing and Emerging Societal Indicators

A. Key National Indicators Initiative

B. The Canadian Experience

C. Topical Indicator Systems

D. Conclusions and Recommendations

Chapter 4: The Status of People with Disabilities

A. General Limitations of Existing Data

B. Identifying Concepts That Are Important to Measure

C. Domains and Data

1. Employment

2. Education

3. Health and Health Care

4. Financial Status and Security

5. Community Participation

6. Leisure and Recreation

7. Political Participation

8. Transportation

9. Housing

10. Personal Relationships

11. Technology

D. Conclusions and Recommendations

Chapter 5: Key Disability Indicators

A. Introduction

B. Desirable Features of the Disability Topical Indicators

Defined by Stakeholders

Objective and Subjective Measures

Measurable, Reliable and Valid

Understandable

Comprehensive

Leading and Lagging Indicators

C. The Indicators

1. Employment

Strengths, weaknesses, and alternatives

2. Education

Strengths, weaknesses, and alternatives

3. Health Status and Health Care

Strengths, weaknesses, and alternatives

4. Financial Status and Security

Strengths, weaknesses, and alternatives

5. Leisure and Recreation and Personal Relationships

Strengths, weaknesses, and alternatives

6. Crime and Safety

Strengths, weaknesses, and alternatives

7. Domains for Which There Are No Indicators

Chapter 6: Program Performance Indicators

A. Existing Program Performance Measurement Systems

B. Integrating Quality of Life Concepts into Program Performance Measures

C. Conclusions and Recommendations

Chapter 7: Conclusions and Recommendations

References