James G. Huse, Jr.

OFFICE OF

This report reflects our understanding and evaluation of the process related to PMs #3 and #4. To achieve its strategic goal "To Deliver Customer-Responsive World-Class Service," SSA has developed several strategic objectives. One of these objectives is to have 9 out of 10 customers rating SSA’s service as "good," "very good," or "excellent," by 2002 with most rating it "excellent." SSA’s FY 2001 Annual Performance Plan (APP) contains two performance indicators developed to meet this objective as follows:

• Percent of employers rating SSA’s overall service as "excellent," "very good" or "good" - This indicator will be considered achieved if 93 percent of employers surveyed rate SSA’s overall service as "excellent," "very good" or "good."

One of the main highlights of this survey was that a notable number of respondents expressed unfamiliarity with the SSA services, as their wage reports were prepared by a payroll service or by someone outside of the company.

Performance Measure	Reported Result
Percent of employers rating SSA’s overall service as "excellent," "very good," or "good."	82 percent
Percent of employers rating SSA’s overall service as "excellent."	6 percent

Survey methodology may have led to a large non-response rate;
Survey results cannot be extrapolated to the target population;
Survey results may be upwardly biased;
Survey design needs improvement; and
Errors were identified in FY 2000 Survey Draft Report.

Survey methodology may have led to a high non-response rate.

Overall 40 percent of employers surveyed responded to SSA. Although it is not uncommon for mail surveys to experience response rates that are of this level or lower, it is generally accepted, for example by the Office of Management and Budget (OMB), that response rates of at least 70 percent are needed to provide credible results. This response rate was partially due to the fact that a large number of employers had their wage reports prepared by third parties and therefore experienced difficulties in evaluating SSA effectively on this aspect. In addition, a significant number of employers have minimal or no direct contact with SSA or knowledge of the type of services SSA provides. This has led to a high non-response rate of 60 percent of employers surveyed. For this research to serve as an employer satisfaction survey, it is critical for the sampling frame to include only "eligible" employers. Eligible employers would be those who have experience with the SSA-provided services.

The OMB Resource Manual for Customer Survey states, "The ability to make general statements about your agency’s customers based on the survey results is linked to how complete and accurate your frame is."

The data collection methodology should be improved to ensure a better than 40 percent response rate. SSA should consider the use of a Computer Assisted Telephone Interviewing (CATI) method. By using a CATI method it would be possible to screen out those employers who have had no direct contact with SSA. This will effectively eliminate the need for forwarding the survey to third parties, which in turn will eliminate the compounding effects due to responses from inappropriate respondents, and, based on experience with such surveys, should increase the response rate. The Paperwork Reduction Act of 1995, Implementing Guide, Chapter 6, F2 states, "Agencies shall use electronic collection techniques where such techniques reduce burden on the public, increase efficiency of government programs, reduce cost to the government and the public, and/or provide better service to the public." While mail surveys are sometimes less expensive to administer, by utilizing the CATI method SSA could increase the quality of the resulting data, and therefore increase the efficiency and effectiveness of the survey.

If a CATI approach is not feasible, a second suggestion is to include a simple "screening" section at the top of the questionnaire that asks whether respondents ever used or attempted to use SSA services. If the answer is "NO," the text of the questionnaire would instruct the response to put the form in the return envelope and return it, without further effort. This should tend to increase the response rate (by adding those who are out of scope), to allow for a count of those using versus not using SSA services, and to avoid the confusion of a handoff between the employer and the third-party service providers. SSA could then send a separate survey to a sample of third-party service providers, which is more controlled.

No matter what mode of data collection is employed, it is important to conduct non-response analysis after the survey has been administered. This way, it would be possible to design and implement a non-response adjustment step in the weighting process, which can reduce some of the potential bias due to differential non-response.

Survey results cannot be extrapolated to the target population.

In order to minimize non-response, the current survey has instructed employers to forward the questionnaire to the third party they use for wage reporting. This snowballing feature has made it impossible to extrapolate the results to the target population. Also, this approach mixes together the awareness results of the employers and third party providers. Since these two groups are likely to have different levels of awareness, it is best to produce separate awareness estimates for them.

The OMB Resource Manual for Customer Survey states, "The ability to make general statements about your agency’s customers based on the survey results is linked to how complete and accurate your frame is."

Survey results may be upwardly biased.

The above issues have impacted the analysis and reporting of the survey data as well. For instance, when calculating the percent of employers satisfied with the SSA services, a large number of respondents who have expressed no opinion are excluded from calculations. Because of this exclusion, the estimate of the percent of employers satisfied with the SSA service changes from 48 percent (of the entire set of respondents) to 82 percent (excluding those who said they had "no opinion"). It can be argued that such an estimate is upwardly biased. Instead, the report should reflect the percents for each of the answer categories, including no opinion, as shown in Table 3. The "no opinion" response, as it is currently implemented, does not clearly distinguish between those in the population who do not have an opinion because they did not use the services from those who did use the service, but just do not have an opinion.

The survey design needs improvement.

Section 1 of the 2000 survey (page A2) includes a battery of awareness questions with simple Yes/No type answer categories. This is not an efficient use of valuable space on the questionnaire. Questions regarding awareness with simple answer categories (yes or no) have limited informational value. Instead of allocating a significant part of the survey to such questions, it would be advisable to change the structure of these questions so that both awareness and degree to which respondents find the given service useful can be captured. An example would be:

On a scale of 1 to 5, where 1 means very useless and 5 means very useful, please rate the usefulness of each of the following services provided by the SSA. If you are unaware of the given service, please mark the "Don’t know" box.

Changing the wording of these questions and their corresponding answer categories would allow SSA to obtain the awareness measures, while determining how useful respondents find such services at the same time. In addition, the 2000 questionnaire did not use question numbers, making it difficult for the respondent to delineate individual questions, as some questions wrap to the next line. Numbering the questions will make it easier to identify where questions begin and end.

Errors were identified in FY 2000 Survey Draft Report.

1. SSA should enhance the sampling frame to include a larger percentage of respondents and only those which are "eligible" employers (i.e., those employers who use SSA’s services and have direct contact with SSA). SSA should consider the use of a more effective data collection methodology, such as a CATI method. It is likely that SSA can obtain as many respondents as they did for the most recent survey (approximately 1,300) at a cost comparable to the mailed out protocol most recently employed. If SSA continues the mail protocol, telephone follow-ups should be done for employers who, after the second follow-up, do not respond.
   
   
2. In addition, SSA should conduct a non-response analysis after the survey has been administered and develop non-response adjustment factors for the survey data.
   
   
3. If a CATI approach is not feasible, SSA should consider including a simple "screening" section at the top of the questionnaire that asks whether respondents ever used or attempted to use SSA services.
4. SSA should change the structure of the yes/no questions so that both awareness and degree to which respondents find the given service useful can be captured.
   
   
5. In addition, SSA should number the questions to make it easier to identify where questions begin and end.
   
   
6. When calculating the percent of respondents providing a rating of Good, Very Good, or Excellent, the denominator should include all respondents, regardless of their ratings.

This will provide an unbiased estimate of percent satisfied.

7. In future surveys, SSA should ask respondents whether they use SSA services or not, and exclude them from the computation as non-customers.



APPROPRIATENESS OF THE PERFORMANCE MEASURES

As part of this engagement, we evaluated the appropriateness of each of the performance measures with respect to the Government Performance and Results Act of 1993 (GPRA) compliance and SSA’s APP. To do so, we determined whether the specific indicators and goals corresponded to the strategic goals identified in SSA’s APP, determined whether each of these indicators accurately measure performance, and determined their compliance with GPRA requirements.

Performance Measures #3 and #4 align logically with the SSA Strategic Plan but still need improvement

The relationships between PMs #3 and #4 and the applicable SSA Strategic Goal are depicted in the following figure:



The SSA mission is supported by five strategic goals, including Goal #2, "To deliver customer-responsive world-class service." Goal #2, in turn, is supported by several strategic objectives, including the relevant objective dealing with customer perceptions, "By 2002, to have 9 out of 10 customers rate SSA’s service as ‘good,’ ‘very good,’ or ‘excellent,’ with most rating it ‘excellent’." Performance Measures #3 and #4 characterize employer satisfaction with SSA services. Both PM #3 and PM #4 logically align with SSA’s strategic planning process.

Based on the taxonomy of performance measures included in Appendix F, both PM #3 and PM #4 are measures of accomplishment because they report on a result (employer satisfaction) achieved with SSA resources. They are further categorized as outcome measures because they indicate the accomplishments or results (employer satisfaction) that occur because of the SSA services provided. As shown in Appendix F, outcome measures include public perceptions of outcomes, such as employer satisfaction.

Within the framework of GPRA, Performance Measures #3 and #4 fit the intent of an outcome measure because they provide "…a description of the intended result, effect, or consequence that will occur from carrying out a program or activity."³ The intent of these two performance measurements is to gauge employer satisfaction (i.e., the effect) for the activity of providing services. TATTTrrTTTThey can both be useful to management and external stakeholders, as encouraged by OMB’s Circular A-11 Preparation and Submission of Budget Estimates. Given the duplication of the measures, SSA should eliminate one of the measures, unless it finds utility in keeping both of them. Nevertheless, these measures could be improved, as discussed below.

In its current form, the metric may not provide reliable results. The survey methodology might not be appropriate for the underlying objectives. The results cannot be extrapolated to the target population and may be upwardly bias. Furthermore, our evaluation of the draft report indicated that there were errors in the reported data. In addition, the survey design was confusing for employers and the corresponding results are skewed. For example, almost 42 percent of respondents stated they had no opinion on the services provided by SSA.

Ideally, a performance metric should help the agency to take action and affect the performance of the indicator being measured. However, due to the presence of numerous third parties, the measurement becomes clouded and is, therefore, less credible and less useful.

The metric readily facilitates comparisons between geographic regions and business segments. With some enhancements, the measurement system could also be used to benchmark against other agencies with employer relationships, such as the IRS, the Health Care Financing Administration, the Pension Benefit Guaranty Corporation, and the Office of Personnel Management.

Recommendations

8. SSA should enhance its measurement system to ensure greater accuracy.
9. They should also take steps to make the metric actionable so that it will provide greater utility for the agency.
10. In addition, SSA should consider using the measurement to benchmark against other agencies that interface with employers.
11. Finally, SSA should eliminate one of the two measures, unless SSA finds a utility in keeping both measures.

Obtained an understanding and documented the FY 2000 Employer Survey Methodology.

Determined whether performance measures were meaningful and in compliance with GPRA

Enhance the sampling frame to include a larger percentage of respondents and only those which are "eligible" employers, i.e., those employers who use the Social Security Administration’s (SSA) services and have direct contact with SSA.

We agree. We have discontinued the generalized sampling frame used in the survey OIG reviewed. We have already implemented a survey in FY 2002 targeted to employers who have received direct service from SSA; i.e., employers who called the Employer Reporting Services Center (ERSC) for assistance. This sampling frame includes specific contact information that will greatly enhance our ability to achieve an acceptable response rate.

Conduct a non-response analysis after the survey has been administered and develop non-response adjustment factors for the survey data.

We will conduct the suggested analysis if the response rate for the survey falls below acceptable levels. We anticipate that our revised methodology, which includes use of Computer Assisted Telephone Interviewing (CATI) on a more targeted sample, will result in a much higher response rate.

If a Computer Assisted Telephone Interview approach is not feasible, consider including a simple "screening" section at the top of the questionnaire that asks whether respondents ever used or attempted to use SSA services.

We have adopted a CATI for the current survey, which also utilizes a completely redesigned questionnaire developed to assess satisfaction with service provided by the ERSC.

Change the structure of the yes/no questions so that both awareness and degree to which respondents find the given service useful can be captured.

Our current survey does not include yes/no questions.

Number the questions to make it easier to identify where questions begin and end.

We agree. The CATI software being used for the current survey eliminates this problem.

When calculating the percent of respondents providing a rating of Good, Very Good, or Excellent, the denominator should include all respondents, regardless of their ratings.

We disagree. Basing ratings on substantive responses is acceptable practice, as long as accompanying discussion indicates that the percentages reflect the opinions of those who provided a rating, as was done in the report prepared by the Office of Quality Assurance and Performance Assessment. It should also be noted that the rating questions did not include a choice of "no opinion" as indicated in OIG's report; the choice was "not applicable/service not used," which served as a screening device that fit into the format of the questionnaire.

In future surveys, ask respondents whether they use SSA services or not, and exclude them from the computation as non-customers.

Our FY 2002 survey specifically targets employers who have received direct service from SSA (i.e., employers who call the ERSC for assistance).

Enhance the measurement system to ensure greater accuracy.

We agree. The revised methodology for the employer survey should produce this result.

Take steps to make the metric actionable so that it will provide greater utility for the Agency.

We agree. We have modified the employer performance indicator to "Percent of employers rating SSA's overall service during interactions with SSA as excellent, very good or good." This performance measure will be based solely on results of a survey of employers who have called the ERSC. Therefore, the performance measure will indicate satisfaction specifically with service from the ERSC. The survey results will identify specific areas where improvements can be made in the Center's service in order to increase satisfaction as captured by the performance measure. In the future, we intend to include additional types of employer interactions in the survey. At that time, we will determine if and how the employer performance measures should be modified.

Consider using the measurement to benchmark against other agencies that interface with employers.

We believe the modification to the indicator described above and changes to the survey instrument address the need for benchmarking at this time. In addition, our day-to-day interactions with the States and the Internal Revenue Service provide us with an awareness of "best practices" used by those entities. Due consideration is given to incorporating those practices into our own service delivery.

Eliminate one of the two measures, unless SSA finds a utility in keeping both measures.

At this time, we intend to keep both measures. We developed the two performance measures to capture how well the Agency was achieving both parts of the customer satisfaction strategic objective--"...9 out of 10 customers rate SSA's service as good, very good or excellent" and "with most rating it excellent." It is consistent with the two measures assessing the satisfaction of our core business customers. In both instances, the first measure identifies the universe of satisfied employers/customers. The second measure serves to capture how well the Agency is achieving best in business service.