OFFICE OF
THE INSPECTOR GENERAL
SOCIAL SECURITY ADMINISTRATION
PERFORMANCE MEASURE REVIEW:
RELIABILITY OF THE DATA USED TO
MEASURE ELECTRONIC SERVICE DELIVERY
July
2002
A-14-01-11032
AUDIT REPORT
We improve SSA programs and operations and protect them against fraud, waste, and abuse by conducting independent and objective audits, evaluations, and investigations. We provide timely, useful, and reliable information and advice to Administration officials, the Congress, and the public.
Authority
The Inspector General Act created independent audit and investigative units, called the Office of Inspector General (OIG). The mission of the OIG, as spelled out in the Act, is to:
Conduct and supervise independent and objective audits and investigations relating to agency programs and operations.
Promote economy, effectiveness, and efficiency within the agency.
Prevent and detect fraud, waste, and abuse in agency programs and operations.
Review and make recommendations regarding existing and proposed legislation and regulations relating to agency programs and operations.
Keep the agency head and the Congress fully and currently informed of problems in agency programs and operations.
To ensure objectivity, the IG Act empowers the IG with:
Independence to determine what reviews to perform.
Access to all information necessary for the reviews.
Authority to publish findings and recommendations based on the reviews.
Vision
By conducting independent and objective audits, investigations, and evaluations, we are agents of positive change striving for continuous improvement in the Social Security Administration's programs, operations, and management and in our own office.
SOCIAL SECURITY
MEMORANDUM
Date: July 30, 2002
To: The Commissioner
From: Inspector General
Subject: Performance Measure Review: Reliability of the Data Used to Measure Electronic Service Delivery (A-14-01-11032)
The Government Performance and Results Act (GPRA) of 1993 requires the Social Security Administration (SSA) to develop performance indicators that assess the relevant service levels and outcomes of each program activity. GPRA also requires disclosure of the processes used to verify and validate the measured values used to report on program performance. The Office of the Inspector General will conduct reviews to determine the reliability of SSA’s reported performance data. One of SSA’s key initiatives in its Performance Plan for Fiscal Year 2000 calls for increasing the range of program and information services available to its customers over the phone (800-number) or electronically. The objective of this audit was to determine the reliability of the data SSA reported for the following GPRA performance indicator and its relationship to the strategic objective of the Agency.
Indicator FY 2000 Goal FY 2000 Actual Percent increase in the number of automated SSA-initiated Personal 50 percent 55 percent Earnings and Benefit Estimate Statement (PEBES) inquiries processed on the Internet.
Baseline Data: Actual percent of Social Security Statement inquiries processed on the Internet during Fiscal Year (FY) 1999 was 23 percent.
BACKGROUND
In accordance with the Social Security Act, SSA provides wage earners information maintained in SSA’s system of records. This information includes reported wages and self-employment income, the amount of social security taxes paid by the requestor and his or her employer, and estimated future benefits.
In FY 1988, SSA began issuing this information in PEBES upon request to eligible individuals. SSA would only issue the PEBES statement if the requestor’s Social Security number (SSN) and identity successfully matched SSA enumeration records.
In March 1997, because of customer demand and technological advances, SSA began testing a fully interactive Internet PEBES. Using an Internet application, requestors could enter their SSN and identifying information directly to SSA systems and receive their PEBES statement electronically while sitting at a terminal or personal computer. This initiative, however, caused public and congressional privacy concerns, and SSA suspended interactive PEBES testing in April 1997.
These privacy concerns caused SSA to change its strategy for using the Internet to provide PEBES information. Since FY 1998, requestors have been able to go to SSA’s Internet web site and either complete and submit a PEBES application on-line or have SSA mail them a PEBES request form. However, the PEBES statement itself is not provided electronically.
Since FY 1999, PEBES has been called the Social Security Statement. Applications to receive Social Security Statements can be obtained from a variety of sources such as senior citizen centers, retirement seminars, local SSA field offices (FO) or by calling SSA’s 800-number telephone service. When calling SSA’s 800-number, individuals can request a form from a TeleService Representative or through an automated response unit (ARU). The ARU requires requestors to use their touch-tone telephone to enter the necessary information.
When individuals request a form via the Internet or telephone, SSA will process the request and mail the form to the requestor. For individuals to obtain a Social Security Statement, they must complete and mail the form to SSA’s Data Operations Center (DOC) in Wilkes-Barre, Pennsylvania. Staff at the DOC will enter information from the form to SSA’s automated Social Security Statement System. If the essential information (such as name, date of birth and SSN) is consistent with SSA’s data base, SSA will create a Social Security Statement and mail it to the requestor. Requestors who have not earned enough Social Security credits or supplied incomplete or inaccurate information will receive a letter of explanation instead of a benefit statement.
For those who complete and submit Social Security Statement request forms on-line, SSA mails them their Social Security Statements or letters of explanation. This method enables SSA to avoid the cost of handling and mailing request forms to customers, and having to enter data from completed forms. In addition, customers do not have to manually complete and return the form, which saves them time and money.
RESULTS OF REVIEW
We were unable to determine the reliability of the data SSA used for its FY 2000 performance indicator because SSA did not retain adequate documentation. In addition, because the indicator measured the use of only one Internet application, it did not support its strategic objective of expanding the number of program informational services available to customers over the telephone or through the Internet.
Although we could not determine the reliability of the data SSA reported, we reviewed SSA’s revised performance plan for FY 2001 to see if the Agency had ever modified its performance indicators to better support its efforts to expand the number of services available via the phone or Internet. In doing this, we found a number of issues that management should consider.
In its revised performance plan for FY 2001, SSA established new Electronic Service Delivery performance indicators. While the new indicators track SSA’s success in establishing more customer-oriented services on-line, they no longer emphasize the importance of encouraging more people to use existing on-line or Internet services. Customers, if not informed that on-line services are available, will not use them, which will reduce the economic benefit that SSA receives from providing Internet services. As such, we believe that SSA could save nearly $5 million in a 5-year period by encouraging more customers to use Internet-linked devices in its FOs instead of more costly labor-intensive services (see Appendix B for details).
NO SUPPORTING DOCUMENTATION
We could not determine the reliability of the data that SSA used in measuring the increase in Internet Social Security Statement inquiries. SSA did not retain documentation of the data it used in counting inquiries during October, November and December 1999, and it under-reported the number of inquiries for January and August 2000. Consequently, we could not verify whether SSA exceeded its FY 2000 goal by 5 percent as reported in its Performance and Accountability Report for Fiscal Year 2000. SSA staff responsible for collecting and maintaining the data told us that they had not been instructed to retain supporting documentation.
PERFORMANCE INDICATOR DOES NOT SUPPORT STRATEGIC OBJECTIVE
GPRA requires the head of each agency to submit to the Director of the Office of Management and Budget and to Congress a strategic plan for program activities. This includes an explanation of how performance indicators relate to strategic plan objectives.
SSA’s FY 2000 performance indicator did not support its strategic objective of increasing the range of program and information services available to customers over the telephone or electronically. Instead of measuring the number of additional on-line and ARU services made available to customers, the performance indicator measured the number of persons accessing one Internet service, Social Security Statement inquiries.
ENCOURAGING GREATER USE OF ELECTRONIC SERVICES
In its revised performance plan for FY 2001, SSA modified the previous year’s strategic objective to read: "By 2005, make 60 percent of SSA’s customer-initiated services available to customers either electronically via the Internet or through automated telephone service…." The performance indicator for the FY 2001 objective is the percent of SSA customer-initiated services available via the Internet or ARU. The goal is to have approximately 15 of 73 possible customer-initiated services available through the Internet or ARU.
The revised FY 2001 performance indicator complements the strategic objective of expanding the number of customer-initiated services available through the Internet or ARU. However, in establishing that performance measure, SSA dropped the FY 2000 indicator focusing on the volume of Social Security Statement request inquiries initiated through the Internet. Although that indicator did not complement its strategic objective, it did reflect the importance of promoting the greater use of customer services available through the Internet and ARU. It is the secure and effective use of those services that will reduce the Agency’s dependence on the more expensive and labor-intensive customer services such as face-to-face conversations with an SSA Service Representative, or using the 800-number to speak with a TeleService Representative.
As indicated in an internal October 2000 report, SSA needs to promote easy-to-use, self-help applications accessible from Internet-linked devices located in SSA FOs. Instead of waiting for assistance from an SSA Service Representative, applicants for Social Security Statements and other automated services could follow simple instructions highlighted on a kiosk-like computer screen and initiate their Internet requests in the FO reception area.
SSA has already assessed the cost-benefit of implementing an in-house, on-line kiosk system and making it available in high-volume FOs for three customer-initiated services. The services include applications for new and replacement SSN cards, verifying benefit amounts and applying for retirement benefits.
Based on a proof of concept project conducted between July 1999 and January 2000, SSA prepared a preliminary cost-benefit analysis for piloting and implementing the kiosk service. For a 5-year period, SSA estimated a total cost of $6.1 million and total savings of over $11 million realizing a net savings of nearly $5 million that SSA could put to better use (see Appendix B for details). By adding an existing Internet service—Social Security Statement requests—to the kiosk project, SSA should increase estimated project savings. As of June 2002, SSA decided not to fund a pilot of the kiosk proposal.
CONCLUSIONS AND RECOMMENDATIONS
In our review, we were unable to determine the reliability of data used to measure electronic service delivery, and we found that the performance indicator in SSA’s Performance Plan for Fiscal Year 2000 did not support its strategic objective. In addition, we identified a number of issues SSA management should address as it expands electronic service delivery.
SSA’s efforts to expand services available to customers via the telephone and Internet should focus on the potential economic benefits of increasing the use of customer-initiated services that are available electronically. SSA should establish a strategic objective for increasing customer use of secure Internet services. It should also re-establish a performance indicator to assess its success in achieving that objective. Because of the potential savings of doing business over the Internet, SSA should pilot customer use of easy-to-use, self-help applications accessible from Internet-linked devices located in SSA FOs. As SSA increases the use of Internet services, it should also assess inherent security and privacy risks. By encouraging the use of available and secure, customer-initiated Internet services; SSA will save money, reduce demands on human capital, diversify customer services available to the public, and reduce customer wait time and postage expenses.
We recommend SSA:
AGENCY COMMENTS
SSA agreed with Recommendations 1 through 3. SSA agreed to establish automated procedures to help employees retain documentation of the data sources and values used in reporting program performance, and to emphasize the importance of retaining the documentation. In the future, SSA will establish data bases or a data warehouse to help employees create the reports needed to retrieve historical performance indicator data.
In addition, SSA agreed to establish a strategic objective of increasing the use of secure on-line or Internet services along with a performance indicator to measure the volume usage of Internet and electronic services.
SSA disagreed with our fourth and fifth recommendations to: (1) pilot customer use of easy-to-use, self-help applications accessible from Internet-linked devices located in SSA FOs, including the Internet Social Security Statement request service; and (2) expand use of the Internet-linked application if the pilot proves it to be secure and cost beneficial. SSA decided that although the pilot has merit, it would not spend scarce resources on it. Rather than finding alternative ways to access Internet services, SSA will concentrate on increasing the breadth and usage of Internet services. Also, SSA conducted a pilot of customer use of Internet-linked personal computers in FO lobbies and found that existing Internet services did not address the needs of FO clients (see Appendix C for details).
OFFICE OF THE INSPECTOR GENERAL RESPONSE
One way for SSA to accomplish its major objective of increasing "the breadth and usage of Internet services" is to continue searching for alternative ways for clients to access those services. We believe that many people do not use SSA Internet services because they do not have access to Internet-linked personal computers. Making this type of access available in SSA FOs, along with simple instructions, will provide people with an alternative to waiting for an SSA representative and ultimately increase Internet usage.
Because
secure access to Internet services does not have to be through a kiosk, we have
removed that word "kiosk" from Recommendations 4 and 5 in our report. However,
Internet applications are rapidly evolving to meet user needs, including secure
electronically-transmitted reports to users. If SSA does not evaluate the use
of secure Internet-linked devices in SSA FO lobbies before these developments
occur, it will not be prepared to fully exploit the new software and security
advances.
Appendices
APPENDIX A — Scope and Methodology
APPENDIX B — Office of Automation Support Access Project
APPENDIX C — Agency Comments
APPENDIX D — OIG Contacts and Staff Acknowledgments
Appendix A
Scope and Methodology
Our audit methodology included reviewing Social Security Administration (SSA) reports and policies and procedures pertaining to the Government Performance and Results Act of 19931. We also examined an SSA study, which discussed the cost-effective use and increasing availability of automated program services initiated over the phone or via the Internet.2
We reviewed detailed documentation of the procedures SSA uses to collect, count and process Social Security Statement requests. To acquire this documentation and learn more about the Social Security Statement process, we interviewed SSA personnel at Headquarters in the Office of Systems, the Office of Information Management and the Office of Workload Analysis. We also interviewed personnel at the SSA Data Operations Center in Wilkes-Barre, Pennsylvania.
Our audit work was performed in Baltimore, Maryland, between April and June 2001 and was conducted in accordance with generally accepted government auditing standards as they pertain to performance audits.
Appendix B
AUTOMATED CUSTOMER COMMON ENTRY SUPPORT SYSTEM – KIOSK PILOT PROPOSAL
AUTOMATED CUSTOMER COMMON ENTRY SUPPORT SYSTEM – KIOSK PILOT PROPOSAL1
Pilot for 26 Social Security Administration (SSA) Field Offices (FO) for Fiscal Year (FY) 2002:
ACTION Kiosks Unit Cost Total
Purchase 43 $4,000 $172,000
Software $100,000
Site Preparation 43 $1,000 $43,000
Total Pilot Cost $315,000
If successful, roll-out funding in FY 2003:
Purchase 1,038 $2,4002 $2,491,200
Software $100,000
Maintenance 43 $ 1203 $5,160
Total FY 2003 Costs $2,596,360
Rollout Funding in FY 2004:
ACTION Kiosks Unit Cost Total
Purchase 1,038 $2,400 $2,491,200
Software $100,000
Maintenance 1,081 $ 120 $129,720
Total FY 2004 Costs $2,720,920
Future Maintenance
FY 2005 2119 $ 120 $254,280
FY 2006 2119 $ 120 $254,280
Total future maintenance $508,560
Total Costs for First 5 Years $6,140,840ESTIMATED COST SAVINGS ON THE SOCIAL SECURITY ADMINISTRATION’S KEY INITIATIVE PLAN AND SCHEDULE, SECTION IV, BUSINESS CASE/RETURN ON INVESTMENT
Annual Annual
Salary Cost
Action Workyears With Benefits Savings
Benefit Verification 3 $43,476 $ 130,428
Social Security Number 24 $43,476 $ 1,043,424
Retirement Insurance Benefits 18 $58,104 $ 1,045,872
Total Projected Annual Savings $ 2,219,724
Total Projected 5-Year Savings $11,098,620
Total Estimated 5-Year Costs ($ 6,140,840)
Estimated Net Savings over the 5-Year period4 $ 4,957,780
Appendix C
Agency Comments
SOCIAL SECURITY
MEMORANDUM |
Date: |
May 31, 2002 |
|
To: |
James
G. Huse, Jr. |
From: |
Larry
Dye |
Subject: |
Office of the Inspector General Draft Report, "Performance Measure Review: Reliability of the Data Used to Measure Electronic Service Delivery" (A-14-01-11032)—INFORMATION |
We appreciate
OIG's efforts in conducting this review. Our comments on the report content
and recommendations are attached.
Please
let us know if we can be of further assistance. Staff questions can be referred
to Dan Sweeney on extension 51957.
COMMENTS ON THE OFFICE OF THE INSPECTOR GENERAL (OIG) DRAFT REPORT, "PERFORMANCE MEASURE REVIEW: RELIABILITY OF THE DATA USED TO MEASURE ELECTRONIC SERVICE DELIVERY" (A-14-01-11032)
Recommendation 1
Establish procedures requiring staff to retain documentation of the data sources and values used in reporting program performance and make sure that staff is aware of the procedures.
Comment
We agree. Since there is no central repository to store archived information, employees manually maintain historic and trend analysis on spreadsheets. With changing indicators and numerous key staff changes, we were unable to retrieve all of these manually maintained counts. We recognize the importance of data retrieval and will emphasize this point with employees providing data for the Performance Plan indicators. Our long-term goal is to establish databases or a data warehouse that will enable employees to create the necessary reports for retrieving historical data.
Recommendations 2 and 3
Establish an additional strategic objective of increasing the use of secure on-line or Internet services.
Develop performance indicators consistent with the strategic objective of increasing customer use of secure on-line or Internet services.
Comment
We agree. We plan to address OIG’s concerns by updating the objective to include both the breadth and usage of services in the FY 2004 Performance Plan. Included in our plans is the development of a new performance indicator to focus on the volume usage of Internet and electronic services.
Recommendations 4 and 5
Pilot customer use of easy-to-use, self-help applications accessible from Internet-linked kiosk devices located in SSA field offices (FO). Expand the use of the kiosk applications if the pilot proves to be secure and cost beneficial.
Include the Internet Social Security Statement request service in the kiosk pilot.
Comment
We do not agree with either recommendation.
Following receipt of the draft report, the Electronic Service Delivery Associate Commissioner Steering Committee reviewed the kiosk project at its May 7, 2002, meeting to consider the OIG recommendations. Although the proof of concept data from the kiosk study supports additional piloting of kiosks, the Committee’s consensus was that, in the current environment of limited resources for competing Information Technology initiatives, further piloting should not be done at this time. The Agency’s major focus now is on increasing the breadth and usage of Internet services, not finding alternate ways to access existing services.
In addition to SSA’s consideration of kiosk technologies, we conducted a pilot under the Techway umbrella that tested providing Internet-linked personal computers in FO lobbies. In that pilot, we found that the available Internet applications do not address the needs of FO visitors. For example, the Internet benefit verification application results in a letter mailed to the beneficiary’s home and does not provide a printed notice for immediate pick up.
Appendix D
OIG Contacts and Staff Acknowledgments
OIG Contacts
Kitt Winter,
Director, Data Analysis and Technology Audit Division (410) 965-9702
Patrick
Kennedy, Audit Manager, Mainframe Controls and Advanced Techniques (410) 965-9724
Acknowledgments
In addition to those named above:
Thomas P.
Tennant, Auditor-in-Charge
Annette
DeRito, Writer-Editor, Policy, Planning and Technical Services Division
For additional copies of this report, please visit our web site at www.socialsecurity.gov/oig or contact the Office of the Inspector General’s Public Affairs Officer at 410-966-1375.
Please refer to Common Identification Number A-14-01-11032.
Overview of the Office of the Inspector General
Office of Audit
The Office of Audit (OA) conducts comprehensive financial and performance audits of the Social Security Administration’s (SSA) programs and makes recommendations to ensure that program objectives are achieved effectively and efficiently. Financial audits, required by the Chief Financial Officers' Act of 1990, assess whether SSA’s financial statements fairly present the Agency’s financial position, results of operations and cash flow. Performance audits review the economy, efficiency and effectiveness of SSA’s programs. OA also conducts short-term management and program evaluations focused on issues of concern to SSA, Congress and the general public. Evaluations often focus on identifying and recommending ways to prevent and minimize program fraud and inefficiency, rather than detecting problems after they occur.
Office of Executive Operations
The Office of Executive Operations (OEO) supports the Office of the Inspector General (OIG) by providing information resource management; systems security; and the coordination of budget, procurement, telecommunications, facilities and equipment, and human resources. In addition, this office is the focal point for the OIG’s strategic planning function and the development and implementation of performance measures required by the Government Performance and Results Act. OEO is also responsible for performing internal reviews to ensure that OIG offices nationwide hold themselves to the same rigorous standards that we expect from SSA, as well as conducting investigations of OIG employees, when necessary. Finally, OEO administers OIG’s public affairs, media, and interagency activities, coordinates responses to Congressional requests for information, and also communicates OIG’s planned and current activities and their results to the Commissioner and Congress.
Office of Investigations
The Office of Investigations (OI) conducts and coordinates investigative activity related to fraud, waste, abuse, and mismanagement of SSA programs and operations. This includes wrongdoing by applicants, beneficiaries, contractors, physicians, interpreters, representative payees, third parties, and by SSA employees in the performance of their duties. OI also conducts joint investigations with other Federal, State, and local law enforcement agencies.
Counsel to the Inspector General
The Counsel to the Inspector General provides legal advice and counsel to the Inspector General on various matters, including: 1) statutes, regulations, legislation, and policy directives governing the administration of SSA’s programs; 2) investigative procedures and techniques; and 3) legal implications and conclusions to be drawn from audit and investigative material produced by the OIG. The Counsel’s office also administers the civil monetary penalty program.