Jo Anne B. Barnhart
Commissioner

Inspector General

Performance Measure Review: Reliability of the Data Used to Measure Public Knowledge of the Social Security Administration (A-02-01-11015)

Following consultations with congressional committees, the Office of the Inspector General agreed to review the Social Security Administration’s (SSA) performance indicators over a continuous 3-year cycle. We recently completed our first 3-year cycle. In conducting this work, we used the services of an outside contractor, PricewaterhouseCoopers (PwC), LLP, to assist us in our efforts.

For this report, we used PwC to conduct the review of one of the Agency’s performance indicators related to the public’s knowledge of SSA. The objective of the review was to assess the reliability of the data used to measure the level of public knowledge of SSA.

Please comment within 60 days from the date of this memorandum on corrective action taken or planned on each recommendation. If you wish to discuss the final report, please call me or have your staff contact Steven L. Schaeffer, Assistant Inspector General for Audit, at (410) 965-9700.

James G. Huse, Jr.

To evaluate the 11 performance indicators identified by the Social Security Administration (SSA) in its Fiscal Year (FY) 2001 Annual Performance Plan (APP), PricewaterhouseCoopers, LLP (PwC) was contracted to determine whether:

• Performance data used to calculate measured values for comparison to SSA’s performance goals were complete, accurate, reasonable, consistent, and reliable;

• Performance data was properly calculated and reported;

• Key automated and manual internal controls related to systems and processes used to calculate and measure performance were adequate; and

• Performance measures were meaningful and in compliance with the Government Performance and Results Act of 1993 (GPRA).

This report is one of five separate stand-alone reports, corresponding to the following SSA process and performance measure (PM):

• Percent of public who are knowledgeable about Social Security programs (PM #11)

This report reflects our understanding and evaluation of the process related to PM #11. To achieve its strategic goal "To strengthen public understanding of Social Security programs" SSA has developed several strategic objectives. One of these objectives is, "By 2005, nine out of ten Americans will be knowledgeable about the Social Security programs in five important areas:"

• Basic program facts;
• Financial value of programs to individuals;
• Economic and social impact of the programs;
• How the programs are financed today; and
• Financing issues.

We performed our testing from September 21, 2000 through February 15, 2001. Our engagement was limited to testing at SSA’s headquarters in Woodlawn, Maryland. The procedures that we performed were in accordance with the American Institute of Certified Public Accountants’ Statement on Standards for Consulting Services, and are consistent with appropriate standards for performance audit engagements in Government Auditing Standards (Yellow Book, 1994 version). However, we were not engaged to and did not conduct an audit, the objective of which would be the expression of an opinion on the reliability or accuracy of the reported results of the performance measures evaluated. Accordingly, we do not express such an opinion. Had we performed additional audit procedures, other matters might have come to our attention that would have been reported to you.

This indicator has been created to measure the percent of the public who are knowledgeable about Social Security programs. The goal during FY 2000 is to have 65 percent of the public knowledgeable about SSA programs. SSA measures public understanding by conducting an annual survey. Below is an overview of the Public Understanding Measurement System (PUMS) survey.

PUMS is an annual survey conducted by SSA and its contractor, designed to determine what percent of the public is knowledgeable about SSA and the services it provides. The first year of the study established a baseline knowledge indicator that has been used in following years to track changes in the public’s knowledge of SSA and its programs. Specifically, this study aims to measure:

Although not the data source for this Performance Measure, an additional study, Moving the Needle (MTN), was performed to further evaluate the effectiveness of various forms of public education and outreach efforts in raising public awareness and knowledge of Social Security. SSA anticipates using this information to design annual public education programs, which would target specific knowledge or performance gaps. Ultimately, it is expected that results from this survey will assist SSA in achieving the target goal that at least 90 percent of the public will be knowledgeable about SSA and its services by 2005.

The PUMS surveys have employed a stratified probability sampling design, where strata are defined based on SSA regions. Specifically, list-assisted Random Digit Dialing (RDD) samples are selected to secure a minimum of 400 interviews in each of the 10 SSA regions.

The sampling design for the MTN involved a 2-stage process. In the first stage, 32 geographic areas were selected, reflecting population groups with the greatest potential for improvement with respect to knowledge of SSA services. These 32 areas were then paired based on demographic information to form 16 pairs of control-treatment groups. Upon consultation with SSA, eight of these pairs were retained, from which RDD samples of households were selected for participation in the survey. Prior to the survey administration, various methodologies were used to increase the knowledge of the public regarding the SSA services in each of the eight test areas.

The PUMS questionnaire was developed with collaborative efforts from SSA, SSA’s contractor, and other experts. The survey instrument consisted of questions dealing with the following three areas of inquiry:

• Public knowledge
• Sources of social security information
• Demographic information

In designing the questionnaire, SSA proposed a set of knowledge metrics that was tested in nine focus groups, which consisted of three age, income, and geographical groups. The Office of Communications (OComm) and Office of External Affairs conducted these focus groups in April of 1998. Consistent with Office of Management and Budget (OMB) restrictions, fewer than 10 respondents participated in each focus group. Following the focus groups, SSA provided all related materials to its contractor for their review. Senior researchers at the National Academy of Social Insurance also reviewed the measures.

Once essential knowledge indicators were identified, a scoring process was established to capture the trend in the public’s knowledge. Specifically, it was decided to assign an equal measure of importance to each of the 23 awareness questions, with anyone scoring a 70 percent or higher identified as "knowledgeable." It should be noted that 1 of these 23 questions could secure 4 possible points, making the total possible points

be 26. Ultimately, however, only 19 of the questions were used to measure knowledge, as 4 questions, including the 1 with a 4-point possible score were eliminated from the calculations. The following table, reproduced from the contractor’s report, summarizes the evolution of the knowledge metric. Note that the contractor report did not explicitly define the question types shown below.

For the most part, the questionnaires for PUMS-I and -II have remained unchanged, with the exception of a new section that was added to the PUMS-II survey. Starting with PUMS-III, however, a number of major changes were introduced for various policy and research issues. For example, 2 of the 19 questions contributing to the knowledge score were eliminated from the questionnaire, while another 3 questions were modified and demoted from the set of questions contributing to the knowledge metric.

The questionnaire used for the MTN survey is very similar to that used for PUMS-II. It contained all of the 19 questions related to the knowledge score, however, a number of other questions have been deleted. In addition, new questions inquiring about different public education and outreach programs by SSA were added to allow assessment of their effectiveness. The following table from the contractor’s 1999 report provides a summary of the composition of these questionnaires.

The PUMS-I survey used a total of 19,283 telephone numbers to secure 4,009 completed interviews. While an 80 percent response rate was targeted, a response rate of only 33 percent was achieved for this survey. Upon conducting a nonresponse analysis, it was concluded that the resulting survey data were not subject to any nonresponse bias. No information has been provided regarding the number of telephone numbers used in each of the other studies, nor have we received any disposition reports that could be used to develop independent estimates of response rates for PUMS-II, III, and MTN.

Due to unavailability of technical reports for PUMS-II or PUMS-III, we cannot comment on whether changes have been introduced with respect to the administration of these surveys. Starting in November 1999 and ending in January 2000, the MTN study was administered quarterly in 16 communities in Philadelphia, Atlanta, Chicago, and

San Francisco. Each quarter 3,000 surveys were conducted, resulting in 12,000 completed interviews. All surveys were conducted in English and Spanish, with field periods as summarized in the following table.

The following table provides a summary of the data collection activities, reflecting the extent of undisclosed information.

Prior to data analysis, survey data were weighted to project the findings to the population of interest. While improving the demographic representation of the resulting samples, the weighting process has made it possible to develop national estimates by compensating for the different regional sampling rates. As stated earlier, the PUMS-I knowledge metric began with 23 questions (26 possible knowledge points), however, it was reduced to only 19 questions with 19 possible points. The resulting survey data were analyzed using a 2-tiered approach: conceptual and factual knowledge.

Overall, the public performed well on the conceptual measure and poorly on the factual knowledge. Those respondents who could correctly answer 13 of the 19 aided questions were deemed knowledgeable about SSA. Accordingly, 55 percent of the public were estimated to be knowledgeable with the SSA services. Moreover, the following are some of the highlights of the analyses that were performed on the PUMS-I survey data.

• Education, age, income, and receiving a Personal Earnings and Benefits Estimate Statement were the four most important factors determining respondent’s knowledge.
• There were no regional differences.
• Minorities, including women, indicated less knowledge of SSA programs compared to other groups.

Again, because there are no technical reports available for PUMS-II or PUMS-III, we cannot comment on whether the knowledge rating for these were calculated in a similar manner. Based on the PUMS-II survey, 57 percent of the public were estimated to be knowledgeable, which is the same rating obtained for the prior year via the PUMS-I survey. At the present time, the results from PUMS-III are not available.

Analogous to the PUMS surveys; the data from the MTN survey have been weighted to represent the demographic composition of the surveyed areas. Subsequently, quarterly estimates of knowledge ratings were calculated for each area and the Nation. According to the MTN survey results, it has been concluded that the public in the test areas has a significantly higher level of knowledge as compared to those in the control areas. This indicates that the additional efforts by SSA positively affect the public’s knowledge. The following table summarizes the MTN estimates that have been obtained from the documents available for our evaluation.

During the period of September 21, 2000 to February 15, 2001, we evaluated the current processes, systems and controls, which support the FY 2000 SSA performance measurement process. In addition, we determined the accuracy of the underlying performance measure data. Our evaluation of the information provided by SSA management and its contractor allowed us to determine that the preliminary reported

FY 2000 results of the performance measure tested (shown below) was reasonably stated based on the methodology used by SSA.

However, we did note the following four opportunities for improvement is SSA methodology:

1. Currently, there are no formal procedures in place to properly reflect the variance inflation due to weighting
2. Multicollinearity among the predictor variables may lead to incorrect results
3. The questionnaire design needs improvement
4. The survey results may be biased due to a significant rate of nonresponse

These items were noted as a result of our testing. We performed an evaluation of the survey methodology, including the sampling and questionnaire designs, data collection procedures, and data analysis and reporting. Because this performance measure is conducted each year, specific attention was given to each annual administration.

Based on our evaluation of the available questionnaires, we have identified a number of potential issues with the structure and wording of the questions, as follows:

• For the MTN questionnaire, the Question 19 series inquires about familiarity with three separate sources of receiving information about the SSA programs: news, public education, and campaigns. Throughout this series, however, inconsistent references are made to these three sources.

• The question series used within the PUMS and MTN questionnaire dealing with usefulness and impact of the SSA statement is based on a screening question (Q24) that simply asks respondents whether they have received the statement or not. That is, the existing sequence of questions does not determine if they have actually read the statement. It is preferable to establish prior reading of the statement, before inquiring about its usefulness.
  
  
• The answer categories used within the PUMS and MTN questionnaire to the demographic question series can be improved. For example, the employment status question does not provide an answer category for students and singles out maternity leave from all other types of leaves. This abbreviated set of employment categories can also confuse others, such as those disabled individuals who do not consider themselves unemployed. The marital status question, on the other hand, has a negative connotation to the order of its answer categories: divorced, separated, widowed, currently married, and single and never married.

1. The survey results may be biased due to a significant rate of nonresponse

Initially, it was anticipated that the response rate to these surveys would be at least 80 percent. However, the secured response rates ranged between 25 and 34 percent. In light of such high rates of nonresponse, it is important to take remedial measures that would increase the response rate to these surveys. A response rate of 25 percent means that 75 percent of the targeted individuals have remained uncovered by the survey. If the group who decided to answer the survey differed, in some substantial way, from those who did not choose to respond, the results obtained could be far from the true proportion in the population as a whole. This uncertainty raises doubts about the credibility or usefulness of the findings of the survey. By weighting the data, SSA has attempted to remove some of the potential bias due to undercoverage; however, the employed methodology should be more outreaching. For example, in residential studies, typically the weighting procedure involves adjustment of the survey data along demographic and socioeconomic dimensions to make the respondent population match more closely these demographic or socioeconomic characteristics in the population as a whole.

According to the Paperwork Reduction Act of 1995 (PRA), Implementing Guide, Chapter VI Section E, samples that suffer from significant nonresponse cannot support valid statistical inferences.

The employed weighting process does not adjust the data with respect to any socioeconomic indicator. It is typical to use income or education as part of the weighting process.

Our evaluation found that the reported FY 2000 results of the performance measure tested were reasonably stated. However, our evaluation noted various issues with the 2000 survey. We recommend that SSA take the following corrective actions:

As part of this engagement, we evaluated the appropriateness of each of the performance measures with respect to GPRA compliance and SSA’s APP. We determined whether the specific indicators and goals corresponded to the strategic goals identified in SSA’s APP, determined whether each of these indicators accurately measure performance, and determined their compliance with GPRA requirements.

The relationship between PM #11 and the applicable SSA Strategic Goal is depicted in the following figure:

The SSA mission is supported by five strategic goals, including Goal 5, "To strengthen public understanding of Social Security programs." Goal 5, in turn, is supported by the single strategic objective, "By 2005, nine out of ten Americans will be knowledgeable about the Social Security programs in five important areas." PM #11 characterizes the public’s level of knowledge about SSA programs. Assuming that the metric has strong performance measurement attributes, the diagram indicates that PM #11 logically aligns with SSA’s strategic planning process.

Based on the taxonomy of performance measures included in Appendix F, PM #11 is a measure of accomplishment because it reports on a result (public awareness) achieved with SSA resources. It is further categorized as an outcome measure because it indicates the accomplishments or results (level of public awareness) that occur because of the services (public relations) provided. Furthermore, this measure of public awareness is similar to a measure of "public perceptions." As shown in Appendix F, measures of public perceptions are considered as outcome measures.

Within the framework of GPRA, Performance Measure #11 fits the intent of an outcome measure because it is "…a description of the intended result, effect, or consequence that will occur from carrying out a program or activity." The intent of this performance measurement is to gauge public awareness (i.e., the effect) for the activity of providing information to the public. A survey-based measurement of this type can be costly and takes time to implement. Nevertheless PM #11 is an appropriate and worthwhile GPRA performance indicator. It can be useful to both management and external stakeholders, as encouraged by OMB Circular A-11. However, there are a few inherent deficiencies in the current design of Performance Measure #11 that are worth noting:

Ideally, a performance metric should help the agency take action to affect the performance of the indicator being measured. In this case, the measurement system will not provide a clear indication of necessary action; this is because the analytical component of the public awareness survey is weak, since it utilizes simple statistical procedures. The performed analyses do not "read between the data lines." For the results to be more actionable, more advanced data analysis methodologies should be used to extract as much intelligence from the data as possible.

Performance Based Budgeting, the ultimate intent of GPRA, is an approach that relates budgets to outputs, and/or resources to performance. At a high level, this metric is well suited for performance based budgeting because stakeholders can evaluate the change in public awareness as a result of changing the total dollars spent on public relations. Where the current measurement system may fall short, however, is in indicating how dollars spent on specific types of educational programs or media impact public awareness. The MTN survey was intended to help clarify this. It is hoped that MTN can ultimately achieve this objective or that SSA can develop an alternative method for measuring the effectiveness of specific educational programs or media.

8. For the results to be more actionable, more advanced data analysis methodologies should be used to extract as much intelligence from the data as possible.
9. SSA should also work toward the successful implementation of the MTN survey or develop an alternative method for measuring the success of specific educational programs and/or media.

As part of this evaluation, we identified an issue that is peripheral to the engagement but, we believe, warrants SSA’s attention. This point is discussed below.

While the idea of switching components of a knowledge indicator is not to be encouraged, it is understandable that, because of changes in policy, the definition of knowledge can change, requiring modifications to the questionnaire. However, it is notable that because of the introduced changes (e.g., use of 19 question in PUMS I & II instead of 14 questions in PUMS III) the knowledge scores have increased significantly, both for PUMS-I and II. These results are summarized in the following table.

Upon further evaluation, it appears that the above increase is partially due to elimination of questions that respondents have commonly scored low. The following table provides a summary of these questions.

8. If SSA plans to report the recalculated results for FY 1999 or FY 2000, it should ensure that the reported results include a description of the change in the knowledge calculation for these 2 years.

SSA agreed with 9 of the 10 recommendations contained in this report. While agreeing with recommendation number eight, "…use more advanced data analysis methodologies to extract as much intelligence from the data as possible…." SSA stated that a lack of available resources has prevented it from completing more advanced analysis of the data collected in the PUMS survey. It noted that it is currently recruiting a staff person to help make such work possible in the future.

In disagreeing with recommendation number seven, "adjust the data with respect to any socioeconomic indicator, and use income or education as part of the weighting process," SSA stated that it was satisfied with the procedures it has used throughout the PUMS survey. It believed that the changes suggested would not be an improvement and would make comparisons to previous years’ data difficult. The full text of the Agency’s comments is in Appendix C.

We appreciate the Agency’s comments to this report. The implementation of the recommendations will help to ensure for the efficient collection and use of the PUMS survey data.

We believe the data collected through PUMS would be more precise if SSA changed its current weighting methodology. There is a non-uniform response pattern across different demographic backgrounds in virtually all surveys. For instance, in household surveys, there are different response rates when comparing higher educated (more affluent) individuals with those at lower levels of education. The primary objective of weighting is to realign the composition of respondents so that they mimic that of the target population. Knowing that the socioeconomic composition of respondents (e.g., income or education) is almost always different from that of the target universe, it would benefit SSA to adjust (weight) the data along such indicators to reduce the skew that will otherwise bias the results. Key outcome measures of this survey are highly correlated with income and education. This further argues for adjusting the data with respect to these indicators, otherwise, the resulting data will be at the mercy of the mix of respondents they manage to contact. It is understandable that changing (improving) the weighting process will introduce some difficulties when it comes to comparing historical data. However, throughout the history of this survey significant changes have been introduced as deemed necessary; this change could be considered as yet another necessary adjustment.

APPENDIX A – Scope and Methodology

APPENDIX B – Acronyms

APPENDIX C – Agency Comments

APPENDIX D – Performance Measure Summary Sheets

APPENDIX E – Performance Measure Process Maps

APPENDIX F – Performance Measure Taxonomy

The Social Security Administration (SSA) Office of the Inspector General (OIG) contracted PricewaterhouseCoopers to evaluate 11 SSA performance indicators identified in its Fiscal Year (FY) 2001 Annual Performance Plan (APP). We performed our testing from September 21, 2000 through February 15, 2001. Since FY 2001 performance results were not yet available as of the date of our evaluation, we performed tests of the performance data and related internal controls surrounding the maintenance and reporting of the results for FY 2000. Specifically, we performed the following:

1. Obtained an understanding of the Public Understanding Measurement System (PUMS) surveys.
2. Tested the reasonableness of the survey data.
3. Determined whether performance measures were meaningful and in compliance with the Government Performance and Results Act of 1993 (GPRA).
4. Identified findings relative to the above procedures and provided recommendations for improvement.

Our engagement was limited to testing at SSA’s headquarters in Woodlawn, Maryland. The procedures that we performed were in accordance with the American Institute of Certified Public Accountants’ Statement on Standards for Consulting Services, and are consistent with appropriate standards for performance audit engagements in Government Auditing Standards (Yellow Book, 1994 version). However, we were not engaged to and did not conduct an audit, the objective of which would be the expression of an opinion on the reliability or accuracy of the reported results of the performance measures evaluated. Accordingly, we do not express such an opinion. Had we performed additional audit procedures, other matters might have come to our attention that would have been reported to you.

• PUMS-I:

• National Report, 1998
• Focus Group report, April 1998
• Technical Report
• Statement of Work

• PUMS-II:

• Contract Requirements, May 1999
• SSA’s contractor Technical Proposal, June 1999
• Survey Results
• Summary of National Results
• National Findings, April 2000
• SSA Knowledge Indicators

• Moving the Needle:

• Questionnaire
• Regional Knowledge Tracking Survey: First Quarter (Nov 1999 – Sep 2000)
• Knowledge Tracking Scorecard

• PUMS-III:

• National Survey – Talking Points
• Questionnaire
• 14 Point Knowledge Point Discussion

To ensure the reasonableness of the number reported in the FY 2000 GPRA section of the SSA Annual Performance and Accountability Report, we evaluated the survey data for PUMS-I (FY 1999 survey) and PUMS-II (FY 2000 survey). Please note that the

FY 2000 GPRA section of the SSA Annual Performance and Accountability Report only includes the results of the FY 1999 survey. Our evaluation included replicating the calculation of the knowledge score based on 26 and then 19 point scales. Once knowledge scores were calculated for each respondent, we calculated the overall percent of the population that are considered "knowledgeable."

As a result of this process, we were able to match those percents reported for PUMS-I as part the GPRA section, and for PUMS-II on SSA’s internal reports. Moreover, survey weights were evaluated to ensure proper calculation of various weighting factors.

As part of this engagement, we evaluated the appropriateness of each of the performance measures with respect to GPRA compliance and SSA’s APP. We determined whether the specific indicators and goals corresponded to the strategic goals identified in SSA’s APP, determined whether each of these indicators accurately measure performance, and determined their compliance with GPRA requirements.

APP Annual Performance Plan
FY Fiscal Year
GPRA Government Performance and Results Act
MTN Moving the Needle
OMB Office of Management and Budget
PM Performance Measure
PUMS Public Understanding Measurement System
PwC PricewaterhouseCoopers LLP
RDD Random Digit Dialog
SSA Social Security Administration
SSI Supplemental Security Income