Jo Anne B. Barnhart
Commissioner
Inspector General
Performance Measure Review: Reliability of the Data Used to Measure the Quality of the Social Security Administration’s Research (A-02-01-11011)
Following consultations with congressional committees, the Office of the Inspector General agreed to review the Social Security Administration’s (SSA) performance indicators over a continuous 3-year cycle. We recently completed our first 3-year cycle. In conducting this work, we used the services of an outside contractor, PricewaterhouseCoopers, LLP, (PwC), to assist us in our efforts.
For this report, we used PwC to conduct the review of two of the Agency’s performance indicators related to the quality of its research publications. The objective of the review was to assess the reliability of the data used to measure the quality of SSA’s research.
Please comment within 60
days from the date of this memorandum on corrective action taken or planned
on each recommendation. If you wish to discuss the final report, please call
me or have your staff contact Steven L. Schaeffer, Assistant Inspector General
for Audit, at (410) 965-9700.
James G. Huse, Jr.
OFFICE OF
THE INSPECTOR GENERAL
SOCIAL SECURITY ADMINISTRATION
PERFORMANCE MEASURE REVIEW:
RELIABILITY OF THE DATA USED TO
MEASURE THE QUALITY OF THE
SOCIAL SECURITY ADMINISTRATION’S
RESEARCH
March 2002
A-02-01-11011
EVALUATION REPORT
INTRODUCTION
This report is one of five separate, stand-alone reports, corresponding to the following Social Security Administration’s (SSA) process and performance measures (PM):
Fiscal Year (FY) 2000 Goal: Develop customer survey and data collection mechanism.
FY 2001 Goal: Establish a baseline for percent of major statistical projects that are produced on schedule.
This report reflects our understanding and evaluation of the process related to PMs #1
and #2. To achieve its strategic goal, "To promote valued, strong, and responsive social security programs and conduct efficient policy development, research, and program evaluation," SSA has developed several strategic objectives. One of these objectives is "…to provide information for decision makers and others on the Social Security and Supplemental Security Income programs through objective and responsive research, evaluation, and policy development." SSA’s FY 2001 Annual Performance Plan (APP) contains two performance indicators developed to meet this objective as follows:
We performed our testing from September 21, 2000 through February 15, 2001. Our engagement was limited to testing at SSA’s headquarters in Woodlawn, Maryland. The procedures that we performed were in accordance with the American Institute of Certified Public Accountants’ Statement on Standards for Consulting Services, and are consistent with appropriate standards for performance audit engagements in Government Auditing Standards (Yellow Book, 1994 version). However, we were not engaged to and did not conduct an audit, the objective of which would be the expression of an opinion on the reliability or accuracy of the reported results of the performance measures evaluated. Accordingly, we do not express such an opinion.
Performance Measure # 1 - Percent of customers assigning a high rating to the quality of SSA’s research and analysis products in terms of accuracy, reliability, comprehensiveness, and responsiveness.
BACKGROUND
This indicator has been created to measure the percent of customers assigning a high rating to the quality of SSA’s research and analysis products in terms of accuracy, reliability, comprehensiveness, and responsiveness. The goal during FY 2000 was to develop a customer survey and data collection mechanism. After the first survey is administered SSA plans to make updates based on the results, as appropriate. Below is an overview of SSA’s customer satisfaction survey and data collection mechanism.
This initiative seeks to better understand the information needs of decision-makers and other customers of SSA research, statistics, and policy analysis products and services. Moreover, SSA seeks to determine how effectively it is meeting those needs. Specifically, this study is to determine:
It also aims to better understand the needs of active participants in the research and policy community who are currently non-users of SSA informational services.
The survey is expected to employ a stratified probability sampling design, where the strata are defined based on the following customer characteristics:
The first stratum, Decision-makers, consists of 59 individuals selected by SSA based on the assumption that they have significant potential for influencing policy decisions. Those in the second stratum, nearly 1,000, are individual subscribers to various SSA publications. The third stratum is comprised of different individuals, including attendees of various SSA conferences and members of the National Academy of Social Insurance (NASI). This stratum includes about 1,500 individuals. The fourth stratum includes approximately 800 non-subscribers who are interested in receiving SSA statistical publications. While individuals from the third and fourth strata originate from separate sources, they will be combined to create an "active non-subscriber" group. The following table provides a summary of the sampling design for this survey.
Table 1. Sampling Design and Universe Counts
Strata |
Source |
Population |
Sample |
Decision-makers |
Individually selected by SSA Office of Policy |
59 |
59 |
Subscribers |
Subscribers to various SSA publications |
978 |
978 |
Active, Non-Subscribers |
Non-subscribers, attendees of Retirement Research Consortium and other conferences, as well as members of NASI |
1,422 |
485 |
Non-subscribers |
Non-subscribers, generated from database of stakeholders, maintained by SSA’s Office of Communication |
814 |
278 |
Total |
3,273 |
1,800 |
Because the list of decision-makers is relatively small, the sample design calls for a census of all these individuals. Further, because the subscriber group is expected to contain the largest concentration of "users," the entire list will also be included in the sample. For the remaining two strata, however, it is not clear at this point what methods will be used to select the needed samples. It appears that all lists have been examined to remove individuals with multiple entries, though additional processing is required to identify addresses and or telephone numbers where needed information is missing.
SSA personnel stated that the target response rate to this survey is expected to be at least 80 percent. Consequently, SSA anticipates needing a sample of about 1,250 to secure the required 1,000 responses. It is expected that this sample size will be sufficient for various statistical analyses. It should be noted that SSA’s contractor deems the 80 percent response rate unachievable and is designing the samples based on a lower rate of 50 percent. However, the total sample of 1,800 will not be adequate to obtain the desired 1,000 responses should the response rate be as low as 50 percent.
The questionnaire for this survey is designed with the intent that respondents can complete it within 15 minutes. Considering that the length of a survey can impact the response rate, it is important to keep the response time within the specified time limit, even in a survey like this where some respondents might have a vested interest.
Since there will be one single questionnaire for all sample types, a few skip patterns are required to administer appropriate questions to corresponding respondents. For instance, respondents who identify themselves in the survey as recent users will be asked about the type, frequency, topics, quality, usefulness, and satisfaction with SSA materials. Non-users will be surveyed about their Social Security related informational needs. Lastly, users and non-users will also be asked about the most important research/policy issues that SSA should focus on during the next few years.
Currently, the questionnaire is being tested and fine-tuned via focus groups and cognitive interviews. The focus groups were conducted in December 2000 with a sample of users with various informational needs. SSA personnel stated that, consistent with Office of Management Budget (OMB) restrictions, fewer than 10 respondents will participate in each focus group. Following the focus groups, cognitive interviews will be conducted to further test questionnaire content. These interviews are planned to be conducted over the telephone with a combination of users and non-users once the questionnaire is finalized; the date of which had not been determined at the time of our evaluation.
Administration/Data Collection (Anticipated)
Since the survey had not been administered at the time of our evaluation, this evaluation will be limited to those procedures that have been proposed. According to the proposed plan, the survey administration will consist of the following steps:
It is anticipated that multiple data collection modes will help secure the required 80 percent response rate. Meanwhile, inactive non-subscribers will be dropped from the survey and excluded from the number of completes and the response rate.
Analysis and Report Generation (Anticipated)
SSA’s contractor anticipates the first step in the data analysis plan to be combining the data gathered via the three modes of data collection into one file. This data file will then be cleansed of duplicate entries. Next, the contractor will remove responses where over half of the items are missing. Checks will also be performed to clean inconsistent data.
With the "cleansed" data, the contractor will conduct an analysis within the self-identified users group. The users will be divided into two sub-groups – subscriber or non-subscriber, based on their sample frame origin. Comparisons between the subscriber and non-subscriber sub-groups are expected to be evaluated to determine any differences in the way these groups perceive the quality of SSA’s work.
The primary analysis will focus around the subscriber and non-subscriber groups. In addition to simple descriptive statistics, SSA is expecting analyses that will effectively measure respondents’ evaluations and levels of satisfaction with the services they receive. SSA’s contractor will also perform a leverage analysis to determine the most critical characteristics in driving overall product and service satisfaction. This method uses regression techniques to identify the most important factors that can predict the outcome measure of interest. In the final report, SSA is expecting documentation regarding the methodology used, and conclusions and recommendations for further study.
RESULTS OF EVALUATION
During the period of September 21, 2000 to February 15, 2001, we evaluated the current processes and controls, which support the FY 2000 SSA performance measurement process. Our evaluation of the information provided to us by SSA management and its contractor as of February 15, 2001, allowed us to determine that SSA has made significant progress to date in the implementation process, and it has the capacity to effectively complete the implementation process.
Performance Measure
|
Reported Result
|
1. Percent of customers assigning a high rating to the quality of SSA’s research and analysis products in terms of accuracy, reliability, comprehensiveness, and responsiveness. |
In the process of developing customer survey and data collection mechanism. |
Although not directly related to GPRA compliance, we noted the following three opportunities for improvement:
These items were noted as a result of our testing of the SSA survey instrument, as it currently exists, as well as the focus group documentation, sampling parameters, and discussions with SSA regarding future plans.
This survey has not been conducted yet. As a result, our evaluation will be limited to the steps that have been completed to date and the planning that has been done for future steps. After the survey has been administered and the corresponding data has been analyzed, it is critical to apply a similar evaluation to these future components of research. The following is a listing of potential issues that deserve attention:
Based on the very limited information provided by SSA’s contractor it appears that the sampling frames that support the nonsubscriber stratum may not provide a complete listing. In addition, these lists include ineligible (non-active) individuals, as well as inaccurate contact information for a significant number of listed individuals. Consequently, the target universe for this survey is not determinable at this time, and therefore the results of the survey may not be generalized.
The Paperwork Reduction Act of 1995, Implementing Guide, Chapter 6, E1 states that statistical laws that permit inference from a sample to a population assume complete coverage, complete response, and random selection. If any of these conditions are not met, then inferences cannot be demonstrated to be valid (i.e. cannot produce results that can be generalized to the universe of study). The OMB Resource Manual for Customer Survey states, "The ability to make general statements about your agency’s customers based on the survey results is linked to how complete and accurate your frame is."
Furthermore, based on the contractor’s report from the focus groups, we also are aware of inconsistencies in SSA’s current lists. Therefore, we suggest that SSA carefully reexamine the sampling frame used to support the survey.
Limited information has and will be collected from the active non-user group. Focus groups have been conducted to improve the effectiveness and quality of the questionnaire. However, it appears that only the users of SSA services were included in the process and focus groups, leaving out non-users, a group that can provide invaluable insight regarding the available services. For example, users cannot comment on why they do not use SSA’s services.
The OMB Resource Manual for Customer Survey, Chapter 3 Defining Customer Survey, states, "It is important to talk to as diverse a group of customers as possible, so that no major perspective or point of view is omitted."
The most recent draft of the questionnaire we have been provided, dated January 29, 2001, asks non-users via an open-ended question, if there is any particular reason why they have not used SSA’s information. The content of this question is a valuable addition to the questionnaire design. However, because it is an open-ended question, fewer respondents are likely to answer. Closed-ended questions receive better response rates than open-ended questions, because they are less burdensome for respondents than open-ended questions. The use of closed-ended questions will secure a higher response rate and will improve the quality of the resulting data.
CONCLUSIONS AND RECOMMENDATIONS
SSA is making progress in preparing for the implementation of this survey. Per conversations with SSA, the contractor has revised the questionnaire to eliminate some of the earlier issues. This process of questionnaire refinement is still in progress. It appears that SSA, with its contractor support, has the capacity to implement the process.
It is important to note that even if everything proceeds according to the plans, this survey will continue to have issues related to the extrapolation of results. Since the target universe for this survey is not determinable, the results of the survey cannot be generalized to a known universe of potential users and non-users, with any degree of validity at this time.
We recommend:
Performance Measure # 2 - Percent of Major Statistical Products that are timely.
BACKGROUND
The indicator created by SSA is to measure the "Percent of major statistical products that are timely," with the goal during FY 2000 to establish a baseline for timeliness. This indicator is linked to SSA’s strategic objective to "Provide information for decision makers and others on the Social Security and Supplemental Security Income programs through objective and responsive research, evaluation, and policy development."
The SSA APP for FY 2001 states that "Responsive research, evaluation, and policy development means providing relevant information in the clearest possible manner with an efficient use of resources." The Office of Research, Evaluation and Statistics (ORES) has selected five of their publications to measure.
On Internet—10 weeks from
release of Trustees Report
In print—11 weeks
from release of Trustees Report
On Internet—January 31
In print-March 15
On Internet—March 31
In print—April 7
On Internet—in phases
between August and December (still finalizing dates)
In print—December
31
On Internet—July 10
In print—July 3
The publications selected by ORES were chosen based on their volume of usage. ORES arrived at the above dates through: 1) detailed discussions with the division directors or analysts involved in developing the publications and with staff of the Division of Information Resources (DIR) in ORES, and 2) historical data from DIR for the last several years on dates when documents were sent to the Office of Publications Management (OPUM), when OPUM cleared the documents to be sent to the printer, and when documents were received from the printer. These dates have been established as the measure of a report being timely.
ORES receives the information to be included in the publications from several sources both within and outside SSA. The measure begins when ORES receives the data and ends when the information is published on the Internet and in hard copy. The ORES milestones in the publication process are as follows:
RESULTS OF EVALUATION
During the period of September 21, 2000 to February 15, 2001, we evaluated the current processes, systems and controls, which support the FY 2000 SSA performance measurement process. Our evaluation of the information provided by SSA management as of February 15, 2001, allowed us to determine that SSA has made significant progress to date in the implementation process, and it has the capacity to effectively complete the implementation process.
Performance Measure
|
Reported Result |
2. Percent of major statistical products that are timely |
Established baseline schedule |
However, we did note the following two opportunities for improvement:
These items were noted as a result of our discussions with SSA regarding current and future plans.
After evaluating SSA’s documentation and speaking with the Deputy Associate Commissioner of ORES it is apparent that SSA has put significant effort into deciding what publications are to be measured and in creating a timetable for publication. The timelines were established based on when ORES has historically received the information. Since part of the information included in the publications comes from outside SSA, the established deadlines may need to be adjusted in the future due to circumstances beyond SSA’s control.
The following is a listing of potential issues that deserve attention.
The strategic objective linked to this measure is for the entire agency while only one part of the agency is being measured. Currently SSA is only measuring five statistical publications that ORES produces. Other divisions of SSA produce material "…for decision makers and others…" about the Social Security and Supplemental Security Income programs. Furthermore, this indicator only measures statistical publications, instead of statistical and non-statistical publications about these programs. By including major publications produced by other SSA divisions and major non-statistical products, the performance measure would better support the strategic objective and allow management to see the timelines throughout the agency rather than within one division. Thus, this would provide management with a tool to measure the performance of divisions throughout the agency.
The strategic objective linked to this measure requires the information to be relevant. Currently SSA only measures the production cycle of the publication process. The collection cycle is not included in the measure. By not including the collection process, the metric does not yield a complete or full measurement of the time involved. If information entering the production cycle is already outdated, it will not be relevant to users, and therefore the measure does not evaluate whether the data was collected in a timely manner. By measuring the entire publishing cycle from data collection through release to users, management will be better able to evaluate if the information provided is relevant pertaining to timeliness.
CONCLUSIONS AND RECOMMENDATIONS
SSA has started the process to develop a good measure to support it strategic objective; however, the measure could be improved to better support the strategic objective. SSA is only measuring the timeliness of publication of one division and only five publications, which represent only a portion of the information provided to decision-makers for these programs. The strategic goal is for all information used by SSA, not just that published by ORES.
While not measuring inputs versus outputs, the measure does fall within the intent of GPRA. The measure, which is being used to establish whether information is being provided in a timely manner, will provide management the ability to evaluate their processes and improve internal management.
We recommend:
OTHER MATTERS
SSA has not quantified the timeliness goal.
While SSA has established a timeline and cycle for the production of the publications, the actual goal of how many of the publications have to be produced within their timeline in order for SSA to meet its goal has not been set. SSA has not established the target goal, i.e., what would be considered a success for this indicator. Do all of the publications have to be produced by their established publication date or only 60 percent for the Agency to be successful? GPRA states in section 1115 (a)(2), "…express such goals in an objective, quantifiable, and measurable form unless authorized to be in an alternative form under subsection (b)." Subsection (b) does not apply to this goal because it is feasible to meet the criteria stated in section 1115 (a)(2).
Recommendation
By establishing what percentage of publication cycles has to be completed within their established timeframe, management will be establishing target goals for the performance indicator. This would allow management to assert whether they have met their goal and allow it to be evaluated by outside observers.
APPROPRIATENESS OF PERFORMANCE MEASURES
As part of this engagement, we evaluated the appropriateness of each of the performance measures with respect to GPRA compliance and SSA’s APP. We determined whether the specific indicators and goals corresponded to the strategic goals identified in SSA’s APP, determined whether each of these indicators accurately measures performance, and determined their compliance with GPRA requirements.
Performance Measure #1 aligns logically with the SSA Strategic Plan but still needs some improvement.
The relationship between PM #1 and the applicable SSA Strategic Goal is depicted in the following figure:
The SSA mission is supported by five strategic goals, including Goal #1, "to promote valued, strong and responsive social security programs and conduct effective policy development, research and program evaluation." Goal #1, in turn, is supported by several strategic objectives, including the relevant objective that deals with the provision of SSA information to decision makers, "Provide information for decision-makers and others on the Social Security and Supplemental Security Income programs through objective and responsive research, evaluation, and policy development." Performance Measure #1 depicts the customer satisfaction with SSA research and analysis products. Assuming that the metric has strong performance measurement attributes, the diagram indicates that PM #1 logically aligns with SSA’s strategic planning process.
Based on the taxonomy of performance measures included in Appendix F, PM #1 is a measure of accomplishment because it reports on a result (customer satisfaction) achieved with SSA resources. It is further categorized as an outcome measure because it indicates the accomplishments or results (customer satisfaction) that occur because of the services (SSA research products) provided. As shown in Appendix F, outcome measures include public perceptions of outcomes, such as a customer satisfaction index.
Within the framework of GPRA, Performance Measure #1 fits the intent of an outcome measure because it is "…a description of the intended result, effect, or consequence that will occur from carrying out a program or activity." The intent of this performance measurement is to gauge customer satisfaction (i.e., the effect) for the activity of providing research and analysis products. QAAssuming it is accurate, it can be useful to both management and external stakeholders, as encouraged by OMB Circular A-11. However, we have identified a few underlying deficiencies that indicate it may not be a reliable GPRA performance indicator.
First, the metric may not promote an adequate and correct understanding of the subject area. Outcome measures such as customer satisfaction indices are complex multivariate metrics that depend on a host of attributes. Defining a critical performance indicator simply based on a single percentage will fail to provide a comprehensive metric. Such simplistic measures may not provide sufficient intelligence for taking appropriate action and subsequently improving performance. Due to the sampling frame issue, there is no guarantee the metric will provide a true picture of the value of SSA research data.
The data is partially subjective because it reflects the opinion of the respondents at the time they answer the survey. Furthermore, use of the data to measure progress from year-to-year is reliable only for the specific sample. Since the employed sampling method is not probability based, the extrapolative power of the resulting survey data will remain undeterminable. In addition, a performance metric should ideally allow SSA to track progress over time. In this case, consistency may suffer due to the nature of the sampling frame. If the survey is repeated from year to year, statistical comparisons between the various years are likely to be unreliable.
At the time of this evaluation, SSA and their contractor had not yet determined a method for analyzing and processing the survey data in order to obtain the desired baseline metric. Furthermore, they had not yet established a method for evaluating progress over a multi-year horizon. These shortcomings pose legitimate concerns, since such analyses are essential for the metric to be useful.
Based on their assigned task, Office of Policy and their contractor have done well in establishing the activities required for obtaining PM #1. Furthermore, the customer satisfaction survey will be a useful product because SSA can use the data to gain valuable insight into stakeholders’ needs with respect to the research and analysis products. However, as a GPRA performance metric, PM #1 has underlying deficiencies and its reliability is in question.
Recommendation
Combined survey results can be misleading because the utility of the SSA research and analysis products varies for each user.
At the time of this evaluation, neither ORES nor their contractor had established a plan for computing the customer satisfaction results from the survey data. When this is determined, ORES should carefully consider the manner in which they combine the responses from the various user types. The utility of SSA research products is not equal for each user. A few significant respondents may rely extensively on the SSA research and analysis products as crucial inputs to their own analytical and decision making products, which, in turn, are important national policy tools in their own right. In such cases, research information used by one user may have a significant benefit to society. On the other hand, many respondents may utilize these products to a far lesser degree and may not rely on their timeliness and accuracy to as great an extent. Unfortunately, these two types of customers carry the same weight in the customer satisfaction survey results. This is a ramification of the survey’s sampling frame. For example, assume that all of the significant users have responded and that any additional responses will come from "occasional" users. As additional responses are incrementally added to survey, the results may become skewed. The occasional users may have either a higher or a lower opinion of the products relative to the significant users. Regardless, they can change the apparent outcome of the customer satisfaction survey, even though the overall value of the SSA research products to the public have changed very little.
Recommendation
Performance Measure #2 aligns logically with the SSA Strategic Plan and can potentially provide numerous management benefits.
The relationship between PM #2 and the applicable SSA Strategic Goal is depicted in the following figure:
The SSA mission is supported by five strategic goals, including Goal #1, "to promote valued, strong and responsive social security programs and conduct effective policy development, research and program evaluation." Goal #1, in turn, is supported by several strategic objectives, including the relevant objective that deals with the provision of SSA information to decision makers, "Provide information for decision-makers and others on the Social Security and Supplemental Security Income programs through objective and responsive research, evaluation, and policy development." Performance Measure #2 characterizes the timeliness of the SSA statistical products. Assuming that the metric has strong performance measurement attributes, the diagram indicates that PM #2 logically aligns with SSA’s strategic planning process.
Based on the taxonomy of performance measures included in Appendix F, PM #2 is a measure of accomplishment because it reports on a result (timeliness) achieved with SSA resources. It is further categorized as an output measure because it indicates the quantity of services provided. As shown in Appendix F, output measures include indications of the quantity of a service provided that meets a certain quality requirement, such as on-time standards of achievement.
Within the framework of GPRA, PM #2 fits the intent of an output measure because it is "…a description of the level of activity or effort that will be produced or provided over a period of time or by a specified date, including a description of the characteristics and attributes (e.g., timeliness) established as standards in the course of conducting the activity or effort." While this performance measure may, in theory, represent a good measure of timeliness, it also has some areas for improvement.
The metric is narrow in focus. Specifically, it counts the timeliness of publications from only ORES, thus representing only one office within the agency. However, by its very name and description, stakeholders may be led to believe that it measures the timeliness of all of SSA’s major statistical products. Furthermore, the metric only includes five products in its calculation. In addition, the measure does not distinguish between large and small-scale efforts. A publication that theoretically requires 80 percent of the resources will impact the measure in the same way as a publication that requires negligible resources. Finally, there is no policy as of yet for determining the circumstances that allow for a change in the target date without being penalized for the lack of timeliness.
Despite these issues, the metric has strong performance measure attributes and can be used as an excellent project management tool. In fact, SSA is encouraged to leverage the additional benefits that can be obtained with this type of project management tool. For example, they could focus beyond the literal description of the metric and utilize the data itself as a management tool. As a result, real time decisions and corrections can be made during the publication process to help improve the on-time delivery of the products. It is also noteworthy that the measure addresses only the steps of the publishing process that are completed after ORES receives the relevant information.
Recommendation
Agency Comments
The Agency agreed with eight of the nine recommendations within this report. It disagreed with the recommendation to "…include additional types of reports and include publications produced by divisions other than the Office of Research, Evaluations, and Statistics" in the performance measure, "Percent of major statistical products that are timely." SSA determined that there are no other statistical publications that are produced regularly, have a statutory deadline, or are considered major in terms of usage.
In agreeing with the recommendation that SSA should evaluate the entire publication cycle, not just the production cycle, when measuring the timeliness of its statistical products, SSA stated that it was already doing what was recommended.
OIG Response
We appreciate the Agency’s response to this report. The implementation of the recommendations will help to ensure for the better measurement and reporting of the quality of SSA’s research. We continue to recommend that SSA consider the inclusion of additional types of reports and publications in its measurement of the timeliness of its research. While the Agency may have determined that it currently includes all publications that are regularly produced, have a statutory deadline, or are considered major in terms of usage, we believe there are other research related publications produced by the Agency that are used by decisionmakers. While these publications may not be regularly produced or mandated by statute, their timely delivery to decisionmakers is nonetheless important. The measurement of the timeliness of all research products produced in a given year is in line with the strategic objective related to the timeliness performance measure--Provide information for decision-makers and others on the Social Security and Supplemental Security Income programs through objective and responsive research, evaluation, and policy development.
We also continue to recommend that SSA evaluate the entire publication cycle, and not just the production cycle, when measuring the timeliness of its statistical products. SSA currently does not measure the collection cycle of the process for all of the publications being measured. By measuring the entire publishing cycle, from data collection through release to users, SSA will be better able to evaluate if the information provided is relevant to its users.
APPENDICES
APPENDIX A – Scope and Methodology
APPENDIX B – Acronyms
APPENDIX C – Agency Comments
APPENDIX D – Performance Measure Summary Sheets
APPENDIX E – Performance Measure Process Maps
APPENDIX F – Performance Measure Taxonomy
Scope and Methodology
The Social Security Administration (SSA) Office of the Inspector General contracted PricewaterhouseCoopers (PwC) to evaluate 11 SSA performance indicators identified in its FY 2001 APP. We performed our testing from September 21, 2000 through February 15, 2001.
Our engagement was limited to testing at SSA’s headquarters in Woodlawn, Maryland. The procedures that we performed were in accordance with the American Institute of Certified Public Accountants’ Statement on Standards for Consulting Services, and are consistent with appropriate standards for performance audit engagements in Government Auditing Standards (Yellow Book, 1994 version). However, we were not engaged to and did not conduct an audit, the objective of which would be the expression of an opinion on the reliability or accuracy of the reported results of the performance measures evaluated. Accordingly, we do not express such an opinion.
For these two performance indicators, PwC was asked to assess SSA’s progress to date in the implementation process, and evaluate if SSA has the capacity to effectively complete the implementation process.
Percent of customers assigning a high rating to the quality of SSA’s research and analysis products in terms of accuracy, reliability, comprehensiveness, and responsiveness. (PM #1)
We performed an evaluation of the survey methodology used for this performance measure, to assess SSA’s progress in implementation of the survey and ability to complete the implementation process. Specifically, we assessed completed steps associated with the sampling frame, the sample design, and the draft of the questionnaire. This included evaluating survey objectives, sample and questionnaire designs, data collection, and analysis and report generation.
We obtained an understanding of the underlying process and procedures surrounding the implementation of the measure through interviews and meetings with the appropriate SSA and contractor personnel and by reviewing and evaluating the following documentation:
Percent of Major Statistical Products that are timely. (PM #2)
In order to assess SSA’s progress to date in the implementation process and evaluate if SSA has the capacity to effectively complete the implementation, we obtained an understanding of the underlying process and procedures surrounding the implementation of the measure through interviews and meetings with the appropriate SSA personnel. In addition, we evaluated the following:
Determined whether both performance measures were meaningful and in compliance with GPRA.
As part of this engagement, we evaluated the appropriateness of each of the performance measures with respect to GPRA compliance and SSA’s APP. To do so, we determined whether the specific indicators and goals corresponded to the strategic goals identified in SSA’s APP, determined whether each of these indicators accurately measure performance, and determined their compliance with GPRA requirements.
ACRONYMS
APP Annual Performance
Plan
DIR Division
of Information Resources
FY Fiscal Year
GAO General
Accounting Office
GASB Government
Accounting Standards Board
GPRA Government
Performance Results Act
NASI National
Academy of Social Insurance
OASDI Old Age,
Survivors, and Disability Insurance
OIG Office
of the Inspector General
OMB Office
of Management and Budget
OP Office Policy
ORES Office
of Research, Evaluation and Statistics
PM Performance
Measure
PwC PricewaterhouseCoopers
LLP
SSA Social
Security Administration
SSI Supplemental
Security Income
OPUM Office of
Publications Management
Agency Comments
COMMENTS ON THE OFFICE OF THE INSPECTOR GENERAL (OIG) DRAFT REPORT, "PERFORMANCE MEASURE REVIEW: RELIABILITY OF THE DATA USED TO MEASURE THE QUALITY OF THE SOCIAL SECURITY ADMINISTRATION’S RESEARCH" (A-02-01-11011)
Thank you for the opportunity to review and comment on this draft report. We are pleased to note that OIG acknowledges that SSA has developed useful performance indicators in the spirit of the Government Performance and Results Act (GPRA) and has discussed them in detail in the Fiscal Year (FY) 2001 Performance Plan.
Our comments on the report recommendations are detailed below.
Performance Measure Number 1 - Percent of Customers assigning a high rating to the quality of SSA’s research and analysis products in terms of accuracy, reliability, comprehensiveness, and responsiveness.
Recommendation 1
SSA and its contractor should investigate alternative sources for a sampling frame, so they can better target the universe. For the current year, steps should be taken to refine the universe by eliminating "ineligible" individuals.
Comment
We agree. SSA has long recognized that measuring satisfaction with policy analysis products is inherently difficult, especially in identifying actual and potential customers. SSA is the only agency that is attempting to include such a measure in its performance plan. In order to meet our goal of conducting an initial survey in FY 2001, we used the best information that could be developed, while recognizing that we did not have a fully satisfactory sampling frame. However, OIG has offered no possible solutions. We believe that attempting to measure customer satisfaction is important, even if it cannot be done with complete precision, and we will continue to do so periodically. Consequently, if we cannot develop a measure that is statistically representative and can be tracked over time, we will drop this item from the performance plan.
Recommendation 2
Since users cannot comment on why they do not use SSA’s services, SSA should conduct focus groups that also include non-users of SSA’s defined statistical products.
Comment
We agree. We will implement this recommendation before conducting the next survey in FY 2004.
Recommendation 3
In order to increase response rate to this question and improve the quality of the resulting data, SSA should rephrase the question to non-users of SSA’s statistical products, regarding the reason why they have not used SSA’s information. The question could be a "check all" type question, with reasons why non-users have not used SSA information as response categories. In addition, SSA should include "Other:_________" as a response category.
Comment
We agree and will implement this recommendation before conducting the next survey in FY 2004.
Performance Measure Number 2 (PM #2) -Percent of Major Statistical Products that are timely.
Recommendation 4
SSA should expand PM #2 to include additional types of reports and include publications produced by divisions other than the Office of Research, Evaluation, and Statistics (ORES).
Comment
In the report, OIG acknowledges the Agency made a significant effort in identifying publications for this measure. We have determined that no other SSA statistical publications are produced regularly, have a statutory deadline, or are considered major
in terms of usage. Therefore, we disagree with this recommendation.
Recommendation 5
For the timeliness to be measured and to ensure that data being published is current, SSA should evaluate the entire publication cycle, not just the production cycle.
Comment
We agree and believe that we are already doing what is recommended, with the exception of benchmarking against other publishing operations. The current timetable lists the reference period of the data, when the data will become available, when analysts will complete a draft, when editing and production will be completed, when the document will be posted on the Web and when we will receive the published document from the printer. Most of these publications are based on SSA program data and appear within months of the end of the reference year. Two of the publications are based on a Census Bureau survey that becomes available within six months of data collection. The schedule also includes target dates for each major function in the production process, so that we can track the source of any problems in meeting the ultimate deadline. Finally, the schedule is updated annually to allow for changes in target dates. In FY 2002, target dates were modified to reflect the need for all publications to be section 508 compliant. Some dates are specified in terms of elapsed time from the reference period of the data, others are fixed dates. However, no meaningful comparison can be made between our schedules and those of other agencies without conducting a detailed analysis of the complexity of the information being released and the resources available to produce it. Comparisons with commercial publishing operations would need to consider that commercial entities are not required to use the Government Printing Office or to be section 508 compliant. This performance measure is already resulting in shorter timeframes and reliable schedules for users. We will continue to look for ways to improve our processes.
Other Matters
Recommendation 6
Starting in FY 2002, SSA should consider establishing a quantitative goal (e.g.,
95 percent of all publications should be produced within their established timetable) in order to evaluate whether the timeliness measure has been met.
Comment
This has now been done. SSA’s goal for FY 2001 was to develop a baseline for this measure. SSA’s goal for FY 2002 is to produce all publications on time.
Recommendation 7
SSA should put this metric on hold until the contractor develops a method for analyzing the survey data, developing the baseline metric, and using the metric to manage performance over a multi-year horizon. If these objectives cannot be satisfactorily completed, the metric should not be included in the GPRA measurement system.
Comment
See response to recommendation 1.
Recommendation 8
The Office of Research, Evaluation and Statistics (ORES) should pay careful attention to the manner in which survey results are combined.
Comment
We agree with the intent of this recommendation but not with trying to differentiate "significant" users from others. The contractor’s report on the results of the customer satisfaction survey provides not only overall results but also results by four groups of respondents (decisionmakers, subscribers, nonsubscribers, and stakeholders). The results show very little difference in outcomes across the four categories of respondents.
Recommendation 9
To further leverage the project management characteristics of this metric, it is recommended that ORES develop methods for obtaining status throughout the publishing process. Finally, it would be relatively easy to compare the timeliness of SSA statistical products with the on-time delivery of publications from other agencies. It is recommended that ORES decompose their publication process into major subtasks, as necessary, so that they can benchmark against other government and commercial publishing operations. In addition, ORES should expand the metric to include other documents. They should also develop a policy to address the change of target dates.
Comment
We have taken steps to address this recommendation. See our response to recommendation 5.
PERFORMANCE MEASURE SUMMARY SHEETS
Name of Measure |
Measure Type |
Strategic Goal/Objective |
|||
1) Percent of Customers assigning a high rating to the quality of SSA’s research and analysis products in terms of accuracy, reliability, comprehensiveness and responsiveness. |
Percentage |
Goal: To promote valued, strong, and responsive social security programs and conduct effective policy development, research, and program evaluation. Objective: Provide information for decision makers and other on the Social Security and Supplemental Security Income programs through objective and responsive research, evaluation and policy development. |
|||
Purpose |
Survey Frequency |
||||
To assess the percent of customers assigning a high rating to the quality of SSA’s research and analysis products in terms of accuracy, reliability, comprehensiveness and responsiveness; SSA will develop a preliminary questionnaire to gather information from customers about the quality of research, analysis and evaluation products. SSA will also identify customers from whom feedback will be elicited. |
The survey has not been developed at this point. SSA has not yet determined the frequency in which the survey will be administered. |
||||
Target Goal |
How Computed |
Data Source |
|||
To develop a customer survey and data collection mechanism.
|
This goal will be considered achieved if SSA:
|
SSA’s Contractor |
|||
Designated Staff Members |
Division |
||||
Michael A. Cronin Susan Grad John Woods |
Office Policy (OP) |
||||
Testing and Results |
|||||
We obtained an understanding of the underlying process and procedures surrounding the implementation of the measure through interviews and meetings with the appropriate SSA and its contractor personnel and by evaluating the following documentation:
Refer to "Results of Evaluation" for a description of the findings. |
Name of Measure |
Measure Type |
Strategic Goal/Objective |
|
2) Percent of major statistical products that are timely. |
Percentage |
Goal: To promote valued, strong, and responsive social security programs and conduct effective policy development, research, and program evaluation. Objective: Provide information for decision makers and others on the Social Security and Supplemental Security Income programs through objective and responsive research, evaluation and policy development. |
|
Purpose |
Frequency |
||
Measure of the extent to which SSA’s major statistical products are produced in a timely manner. |
Annual Indicator |
||
Target Goal |
Data Source |
||
This goal will be considered achieved if SSA establishes a baseline for percent of major statistical projects that are produced on schedule. |
Bureau of the Census SSA Office of Systems, Division of Program Studies |
||
Designated Staff Members |
Division |
||
Susan Grad Alexander Estrin |
OP ORES |
||
Testing and Results |
|||
We obtained an understanding of the underlying process and procedures surrounding the implementation of the measure through interviews and meetings with the appropriate SSA personnel. In addition, we evaluated the following:
Refer to "Results of Evaluation" for a description of the findings. |