Jo Anne B. Barnhart
Commissioner

Inspector General

Performance Measure Review: Reliability of the Data Used to Measure the Quality of the Social Security Administration’s Research (A-02-01-11011)

Following consultations with congressional committees, the Office of the Inspector General agreed to review the Social Security Administration’s (SSA) performance indicators over a continuous 3-year cycle. We recently completed our first 3-year cycle. In conducting this work, we used the services of an outside contractor, PricewaterhouseCoopers, LLP, (PwC), to assist us in our efforts.

For this report, we used PwC to conduct the review of two of the Agency’s performance indicators related to the quality of its research publications. The objective of the review was to assess the reliability of the data used to measure the quality of SSA’s research.

Please comment within 60 days from the date of this memorandum on corrective action taken or planned on each recommendation. If you wish to discuss the final report, please call me or have your staff contact Steven L. Schaeffer, Assistant Inspector General for Audit, at (410) 965-9700.

James G. Huse, Jr.

This report is one of five separate, stand-alone reports, corresponding to the following Social Security Administration’s (SSA) process and performance measures (PM):

• Percent of Major Statistical Products that are timely. (PM #2)

This report reflects our understanding and evaluation of the process related to PMs #1

and #2. To achieve its strategic goal, "To promote valued, strong, and responsive social security programs and conduct efficient policy development, research, and program evaluation," SSA has developed several strategic objectives. One of these objectives is "…to provide information for decision makers and others on the Social Security and Supplemental Security Income programs through objective and responsive research, evaluation, and policy development." SSA’s FY 2001 Annual Performance Plan (APP) contains two performance indicators developed to meet this objective as follows:

• Percent of customers assigning a high rating to the quality of SSA’s research and analysis products in terms of accuracy, reliability, comprehensiveness, and responsiveness - This indicator will be considered achieved if SSA develops a customer survey and data collection mechanism.

• Percent of major statistical products that are timely - This second indicator will be considered achieved if SSA establishes a baseline for percent of major statistical projects that are produced on schedule.

We performed our testing from September 21, 2000 through February 15, 2001. Our engagement was limited to testing at SSA’s headquarters in Woodlawn, Maryland. The procedures that we performed were in accordance with the American Institute of Certified Public Accountants’ Statement on Standards for Consulting Services, and are consistent with appropriate standards for performance audit engagements in Government Auditing Standards (Yellow Book, 1994 version). However, we were not engaged to and did not conduct an audit, the objective of which would be the expression of an opinion on the reliability or accuracy of the reported results of the performance measures evaluated. Accordingly, we do not express such an opinion.

Performance Measure # 1 - Percent of customers assigning a high rating to the quality of SSA’s research and analysis products in terms of accuracy, reliability, comprehensiveness, and responsiveness.

This indicator has been created to measure the percent of customers assigning a high rating to the quality of SSA’s research and analysis products in terms of accuracy, reliability, comprehensiveness, and responsiveness. The goal during FY 2000 was to develop a customer survey and data collection mechanism. After the first survey is administered SSA plans to make updates based on the results, as appropriate. Below is an overview of SSA’s customer satisfaction survey and data collection mechanism.

This initiative seeks to better understand the information needs of decision-makers and other customers of SSA research, statistics, and policy analysis products and services. Moreover, SSA seeks to determine how effectively it is meeting those needs. Specifically, this study is to determine:

• To what extent SSA’s work is reaching a broad audience in the research and policy community.

• To what degree this work is perceived to be of high quality.

• How satisfied the SSA customers are with the information they receive.

The survey is expected to employ a stratified probability sampling design, where the strata are defined based on the following customer characteristics:

1. Decision-maker;
2. Subscriber;
3. Non-subscribers who are active in the research and policy community; and
4. Non-subscribers who are interested.

The first stratum, Decision-makers, consists of 59 individuals selected by SSA based on the assumption that they have significant potential for influencing policy decisions. Those in the second stratum, nearly 1,000, are individual subscribers to various SSA publications. The third stratum is comprised of different individuals, including attendees of various SSA conferences and members of the National Academy of Social Insurance (NASI). This stratum includes about 1,500 individuals. The fourth stratum includes approximately 800 non-subscribers who are interested in receiving SSA statistical publications. While individuals from the third and fourth strata originate from separate sources, they will be combined to create an "active non-subscriber" group. The following table provides a summary of the sampling design for this survey.

Because the list of decision-makers is relatively small, the sample design calls for a census of all these individuals. Further, because the subscriber group is expected to contain the largest concentration of "users," the entire list will also be included in the sample. For the remaining two strata, however, it is not clear at this point what methods will be used to select the needed samples. It appears that all lists have been examined to remove individuals with multiple entries, though additional processing is required to identify addresses and or telephone numbers where needed information is missing.

SSA personnel stated that the target response rate to this survey is expected to be at least 80 percent. Consequently, SSA anticipates needing a sample of about 1,250 to secure the required 1,000 responses. It is expected that this sample size will be sufficient for various statistical analyses. It should be noted that SSA’s contractor deems the 80 percent response rate unachievable and is designing the samples based on a lower rate of 50 percent. However, the total sample of 1,800 will not be adequate to obtain the desired 1,000 responses should the response rate be as low as 50 percent.

The questionnaire for this survey is designed with the intent that respondents can complete it within 15 minutes. Considering that the length of a survey can impact the response rate, it is important to keep the response time within the specified time limit, even in a survey like this where some respondents might have a vested interest.

Since there will be one single questionnaire for all sample types, a few skip patterns are required to administer appropriate questions to corresponding respondents. For instance, respondents who identify themselves in the survey as recent users will be asked about the type, frequency, topics, quality, usefulness, and satisfaction with SSA materials. Non-users will be surveyed about their Social Security related informational needs. Lastly, users and non-users will also be asked about the most important research/policy issues that SSA should focus on during the next few years.

Currently, the questionnaire is being tested and fine-tuned via focus groups and cognitive interviews. The focus groups were conducted in December 2000 with a sample of users with various informational needs. SSA personnel stated that, consistent with Office of Management Budget (OMB) restrictions, fewer than 10 respondents will participate in each focus group. Following the focus groups, cognitive interviews will be conducted to further test questionnaire content. These interviews are planned to be conducted over the telephone with a combination of users and non-users once the questionnaire is finalized; the date of which had not been determined at the time of our evaluation.

Since the survey had not been administered at the time of our evaluation, this evaluation will be limited to those procedures that have been proposed. According to the proposed plan, the survey administration will consist of the following steps:

• All sampled individuals will be mailed a pre-notification letter. This letter will provide a brief description of the purpose of the study and inform the individual that the questionnaire will be arriving shortly.

• One week later, the survey packet will be mailed. This packet will consist of a copy of the questionnaire, a return envelope, and a letter regarding the purpose of the study. This letter will also assure confidentiality and specify the web-site address where respondents can visit to complete the survey on-line.

• One week later, a reminder postcard will be sent to all sampled individuals.

• Two weeks later, non-responder telephone calls will be initiated, at which point individuals will be given the option to complete the questionnaire over the phone, as well as on the web, or by mail.

It is anticipated that multiple data collection modes will help secure the required 80 percent response rate. Meanwhile, inactive non-subscribers will be dropped from the survey and excluded from the number of completes and the response rate.

SSA’s contractor anticipates the first step in the data analysis plan to be combining the data gathered via the three modes of data collection into one file. This data file will then be cleansed of duplicate entries. Next, the contractor will remove responses where over half of the items are missing. Checks will also be performed to clean inconsistent data.

With the "cleansed" data, the contractor will conduct an analysis within the self-identified users group. The users will be divided into two sub-groups – subscriber or non-subscriber, based on their sample frame origin. Comparisons between the subscriber and non-subscriber sub-groups are expected to be evaluated to determine any differences in the way these groups perceive the quality of SSA’s work.

During the period of September 21, 2000 to February 15, 2001, we evaluated the current processes and controls, which support the FY 2000 SSA performance measurement process. Our evaluation of the information provided to us by SSA management and its contractor as of February 15, 2001, allowed us to determine that SSA has made significant progress to date in the implementation process, and it has the capacity to effectively complete the implementation process.

Although not directly related to GPRA compliance, we noted the following three opportunities for improvement:

1. Sampling frames do not provide a comprehensive listing of all users and non-users of SSA research products.
2. Limited information has been collected by SSA from non-users of SSA research products.
3. The questionnaire design could be improved by SSA.

These items were noted as a result of our testing of the SSA survey instrument, as it currently exists, as well as the focus group documentation, sampling parameters, and discussions with SSA regarding future plans.

This survey has not been conducted yet. As a result, our evaluation will be limited to the steps that have been completed to date and the planning that has been done for future steps. After the survey has been administered and the corresponding data has been analyzed, it is critical to apply a similar evaluation to these future components of research. The following is a listing of potential issues that deserve attention:

SSA is making progress in preparing for the implementation of this survey. Per conversations with SSA, the contractor has revised the questionnaire to eliminate some of the earlier issues. This process of questionnaire refinement is still in progress. It appears that SSA, with its contractor support, has the capacity to implement the process.

It is important to note that even if everything proceeds according to the plans, this survey will continue to have issues related to the extrapolation of results. Since the target universe for this survey is not determinable, the results of the survey cannot be generalized to a known universe of potential users and non-users, with any degree of validity at this time.

We recommend:

1. SSA and its contractor should investigate alternative sources for a sampling frame, so they can better target the universe. For the current year, steps should be taken to refine the universe by eliminating "ineligible" individuals.
2. Since users cannot comment on why they do not use SSA’s services, SSA should conduct focus groups that also include non-users of SSA’s defined statistical products.
3. In order to increase response rate to this question and improve the quality of the resulting data, SSA should rephrase the question to non-users of SSA’s statistical products, regarding the reason why they have not used SSA’s information. The question could be a "check all" type question, with reasons why non-users have not used SSA information as response categories. In addition, SSA should include "Other:_________" as a response category.

The indicator created by SSA is to measure the "Percent of major statistical products that are timely," with the goal during FY 2000 to establish a baseline for timeliness. This indicator is linked to SSA’s strategic objective to "Provide information for decision makers and others on the Social Security and Supplemental Security Income programs through objective and responsive research, evaluation, and policy development."

The SSA APP for FY 2001 states that "Responsive research, evaluation, and policy development means providing relevant information in the clearest possible manner with an efficient use of resources." The Office of Research, Evaluation and Statistics (ORES) has selected five of their publications to measure.

1. Fast Facts (annual)

On Internet—10 weeks from release of Trustees Report
In print—11 weeks from release of Trustees Report

2. Income of the Population 55 or Older (biannual, in even years)

On Internet—January 31
In print-March 15

3. Income of the Aged Chartbook (biannual, in even years)

On Internet—March 31
In print—April 7

4. Annual Statistical Supplement (annual)

On Internet—in phases between August and December (still finalizing dates)
In print—December 31

5. SSI Annual Statistical Report (annual)

On Internet—July 10
In print—July 3

The publications selected by ORES were chosen based on their volume of usage. ORES arrived at the above dates through: 1) detailed discussions with the division directors or analysts involved in developing the publications and with staff of the Division of Information Resources (DIR) in ORES, and 2) historical data from DIR for the last several years on dates when documents were sent to the Office of Publications Management (OPUM), when OPUM cleared the documents to be sent to the printer, and when documents were received from the printer. These dates have been established as the measure of a report being timely.

ORES receives the information to be included in the publications from several sources both within and outside SSA. The measure begins when ORES receives the data and ends when the information is published on the Internet and in hard copy. The ORES milestones in the publication process are as follows:

• Data availability;

• ORES analysts/programmer production of all text, tables and charts;

• DIR completion of editing and production of charts;

• For 3 publications, receipt of blue lines (a copy of the print format) from the printer;

• DIR posting of document on Internet in PDF format; and

• Receipt of publication from printer.

During the period of September 21, 2000 to February 15, 2001, we evaluated the current processes, systems and controls, which support the FY 2000 SSA performance measurement process. Our evaluation of the information provided by SSA management as of February 15, 2001, allowed us to determine that SSA has made significant progress to date in the implementation process, and it has the capacity to effectively complete the implementation process.

However, we did note the following two opportunities for improvement:

1. SSA should incorporate other publications as part of this measure.
2. SSA is not fully evaluating the currency of the data.

These items were noted as a result of our discussions with SSA regarding current and future plans.

After evaluating SSA’s documentation and speaking with the Deputy Associate Commissioner of ORES it is apparent that SSA has put significant effort into deciding what publications are to be measured and in creating a timetable for publication. The timelines were established based on when ORES has historically received the information. Since part of the information included in the publications comes from outside SSA, the established deadlines may need to be adjusted in the future due to circumstances beyond SSA’s control.

The following is a listing of potential issues that deserve attention.

1. SSA should incorporate other publications as part of this measure.

The strategic objective linked to this measure is for the entire agency while only one part of the agency is being measured. Currently SSA is only measuring five statistical publications that ORES produces. Other divisions of SSA produce material "…for decision makers and others…" about the Social Security and Supplemental Security Income programs. Furthermore, this indicator only measures statistical publications, instead of statistical and non-statistical publications about these programs. By including major publications produced by other SSA divisions and major non-statistical products, the performance measure would better support the strategic objective and allow management to see the timelines throughout the agency rather than within one division. Thus, this would provide management with a tool to measure the performance of divisions throughout the agency.

2. SSA is not fully evaluating the currency of the data.

The strategic objective linked to this measure requires the information to be relevant. Currently SSA only measures the production cycle of the publication process. The collection cycle is not included in the measure. By not including the collection process, the metric does not yield a complete or full measurement of the time involved. If information entering the production cycle is already outdated, it will not be relevant to users, and therefore the measure does not evaluate whether the data was collected in a timely manner. By measuring the entire publishing cycle from data collection through release to users, management will be better able to evaluate if the information provided is relevant pertaining to timeliness.

SSA has started the process to develop a good measure to support it strategic objective; however, the measure could be improved to better support the strategic objective. SSA is only measuring the timeliness of publication of one division and only five publications, which represent only a portion of the information provided to decision-makers for these programs. The strategic goal is for all information used by SSA, not just that published by ORES.

While not measuring inputs versus outputs, the measure does fall within the intent of GPRA. The measure, which is being used to establish whether information is being provided in a timely manner, will provide management the ability to evaluate their processes and improve internal management.

We recommend:

The Agency agreed with eight of the nine recommendations within this report. It disagreed with the recommendation to "…include additional types of reports and include publications produced by divisions other than the Office of Research, Evaluations, and Statistics" in the performance measure, "Percent of major statistical products that are timely." SSA determined that there are no other statistical publications that are produced regularly, have a statutory deadline, or are considered major in terms of usage.

In agreeing with the recommendation that SSA should evaluate the entire publication cycle, not just the production cycle, when measuring the timeliness of its statistical products, SSA stated that it was already doing what was recommended.

We also continue to recommend that SSA evaluate the entire publication cycle, and not just the production cycle, when measuring the timeliness of its statistical products. SSA currently does not measure the collection cycle of the process for all of the publications being measured. By measuring the entire publishing cycle, from data collection through release to users, SSA will be better able to evaluate if the information provided is relevant to its users.

APPENDIX A – Scope and Methodology

APPENDIX B – Acronyms

APPENDIX C – Agency Comments

APPENDIX D – Performance Measure Summary Sheets

APPENDIX E – Performance Measure Process Maps

APPENDIX F – Performance Measure Taxonomy

The Social Security Administration (SSA) Office of the Inspector General contracted PricewaterhouseCoopers (PwC) to evaluate 11 SSA performance indicators identified in its FY 2001 APP. We performed our testing from September 21, 2000 through February 15, 2001.

Our engagement was limited to testing at SSA’s headquarters in Woodlawn, Maryland. The procedures that we performed were in accordance with the American Institute of Certified Public Accountants’ Statement on Standards for Consulting Services, and are consistent with appropriate standards for performance audit engagements in Government Auditing Standards (Yellow Book, 1994 version). However, we were not engaged to and did not conduct an audit, the objective of which would be the expression of an opinion on the reliability or accuracy of the reported results of the performance measures evaluated. Accordingly, we do not express such an opinion.

For these two performance indicators, PwC was asked to assess SSA’s progress to date in the implementation process, and evaluate if SSA has the capacity to effectively complete the implementation process.

We performed an evaluation of the survey methodology used for this performance measure, to assess SSA’s progress in implementation of the survey and ability to complete the implementation process. Specifically, we assessed completed steps associated with the sampling frame, the sample design, and the draft of the questionnaire. This included evaluating survey objectives, sample and questionnaire designs, data collection, and analysis and report generation.

We obtained an understanding of the underlying process and procedures surrounding the implementation of the measure through interviews and meetings with the appropriate SSA and contractor personnel and by reviewing and evaluating the following documentation:

• SSA’s Statement of Work, dated July 2000;

• Most recent draft of the questionnaire, dated January 29, 2001;
• Contractor’s status report to SSA, Office of Policy, dated October 27, 2000, regarding: "Sample frame, initial recommendations for data collection, and plans for pretest and focus group;"

• Focus Group report, dated December 21, 2000;

• Contractor’s memo to SSA, Office of Policy, dated January 12, 2001, regarding "Proposed sampling plan, data collection plan, and initial data analysis plan." This memo provided a summary of recent developments and modifications to the original plan

• General Accounting Office (GAO) reports on SSA FY 2001 Performance Plan.

In order to assess SSA’s progress to date in the implementation process and evaluate if SSA has the capacity to effectively complete the implementation, we obtained an understanding of the underlying process and procedures surrounding the implementation of the measure through interviews and meetings with the appropriate SSA personnel. In addition, we evaluated the following:

• Memo from ORES regarding publication process and method used to established timelines for publication;
• Process/cycle memos for the publications to obtain an understanding of the publication process; and

• GAO reports on SSA FY 2001 Performance Plan.

As part of this engagement, we evaluated the appropriateness of each of the performance measures with respect to GPRA compliance and SSA’s APP. To do so, we determined whether the specific indicators and goals corresponded to the strategic goals identified in SSA’s APP, determined whether each of these indicators accurately measure performance, and determined their compliance with GPRA requirements.

APP Annual Performance Plan
DIR Division of Information Resources
FY Fiscal Year
GAO General Accounting Office
GASB Government Accounting Standards Board
GPRA Government Performance Results Act
NASI National Academy of Social Insurance
OASDI Old Age, Survivors, and Disability Insurance
OIG Office of the Inspector General
OMB Office of Management and Budget
OP Office Policy
ORES Office of Research, Evaluation and Statistics
PM Performance Measure
PwC PricewaterhouseCoopers LLP
SSA Social Security Administration
SSI Supplemental Security Income
OPUM Office of Publications Management