OFFICE OF
THE INSPECTOR GENERAL

SOCIAL SECURITY ADMINISTRATION

THE EFFECTIVENESS OF THE
SOCIAL SECURITY
ADMINISTRATION'S 800-NUMBER
AUTOMATION SERVICE

October 2007

A-02-07-17049

AUDIT REPORT

Mission

By conducting independent and objective audits, evaluations and investigations, we inspire public confidence in the integrity and security of SSA's programs and operations and protect them against fraud, waste and abuse. We provide timely, useful and reliable information and advice to Administration officials, Congress and the public.

Authority

The Inspector General Act created independent audit and investigative units, called the Office of Inspector General (OIG). The mission of the OIG, as spelled out in the Act, is to:

Conduct and supervise independent and objective audits and investigations relating to agency programs and operations.
Promote economy, effectiveness, and efficiency within the agency.
Prevent and detect fraud, waste, and abuse in agency programs and operations.
Review and make recommendations regarding existing and proposed legislation and regulations relating to agency programs and operations.
Keep the agency head and the Congress fully and currently informed of problems in agency programs and operations.

To ensure objectivity, the IG Act empowers the IG with:

Independence to determine what reviews to perform.
Access to all information necessary for the reviews.
Authority to publish findings and recommendations based on the reviews.

Vision

We strive for continual improvement in SSA's programs, operations and management by proactively seeking new ways to prevent and deter fraud, waste and abuse. We commit to integrity and excellence by supporting an environment that provides a valuable public service while encouraging employee development and retention and fostering diversity and innovation.

MEMORANDUM

Date: October 19, 2007

To: The Commissioner

From: Inspector General

Subject: The Effectiveness of the Social Security Administration's 800-Number Automation Service (A-02-07-17049)

OBJECTIVE

Our objective was to review the effectiveness of the Social Security Administration's (SSA) 800-number automated service system.

BACKGROUND

SSA maintains a national 800-number for individuals to file claims, update records and request information about SSA's programs. Since becoming available nationwide in 1989, SSA's 800-number has become a principal contact point for individuals seeking Agency services. Although the public has a variety of other options (Internet, field offices [FO], etc.) to obtain information or conduct business with SSA, most customers conduct their business with SSA by telephone. In Fiscal Year (FY) 2006, SSA reported that there were 59,475,747 transactions handled by the national 800-number network. The 800-number's telecommunications services are currently provided under contract by Verizon Business.

To keep pace with the growing demand for its 800-number service, the Agency expanded the automated service options to reduce the number of calls going to agents and time spent waiting to speak to an agent. To that end, callers to the 800-number may use a self-service menu of automated services or request to speak with an agent. The 800-number automated system offers a variety of services, in English or Spanish, 24 hours a day, 7 days a week. Callers can request pamphlets and forms in addition to obtaining information, such as business hours and locations of nearby FOs.

An August 2005 Government Accountability Office (GAO) report stated "since 2002, SSA's 800-number automated menus have received progressively higher call volumes but handled fewer calls to completion." GAO further reported that "the number of calls being abandoned without completing a transaction in the automated menus has steadily increased, culminating in 2004, when nearly half of calls to automation were abandoned." The GAO report indicated that, although SSA offered a number of reasons why fewer calls were being handled to completion, it was unable to say with certainty why calls continued to be abandoned.

To meet our audit objective, we analyzed and compared the 800-number's automated service system data for March 2006 and 2007 to identify trends in caller outcomes and differences in performance of individual service options. We also interviewed
800-number officials to obtain an understanding of the Agency's data collection methodology used to track, monitor, and report on the automated system. Lastly, we reviewed the Agency's contracts for 800-number services to determine the contractual obligations of its telecommunications provider for 800-number related management information. See Appendix B for additional background, information and details of our audit scope and methodology.

RESULTS OF REVIEW

We found the service provided by SSA's 800-number automated system has improved. However, SSA can further improve the automated system to better meet the needs of its customers. While automated services were introduced to lessen the call burden on live agents and provide additional services to SSA's customers, a similar percentage of calls to the 800-number were placed with live agents over the last 2 FYs, and many callers who used the 800-number automated services did not complete their calls in automation. These callers either hung up or were re-routed to live agents. The automated services are not cost-effective when calls are not completed because SSA pays for customers' use of the automated services whether they use it successfully or not. Also, unsuccessful use of the automated services can lead to additional charges since customers are often routed to, or call back to speak with, a live agent. In addition, Spanish-language callers had a far higher rate of hanging up in the automated system than English-language callers. We attempted to determine the reasons for the high rate of incomplete calls but found that SSA's management information on callers did not provide detailed caller data to facilitate follow up. Lastly, we found SSA did not have performance measures and goals to determine the effectiveness of its automated
800-number system.

AUTOMATED SYSTEM OUTCOMES

The percentage of 800-number callers who chose to use the automated system was 33.3 percent in March 2006 and 35.3 percent in March 2007. In each month,
22.4 percent and 21 percent, respectively, of the callers who used the automated services were re-routed to a live agent because of problems in completing the automated transaction or requests to speak to a live agent before completing the automated transaction. In addition, 22.6 percent and 16.4 percent of those who used automated services hung up, that is, abandoned the automated system before completing their transaction (see table below).

800-Number Automation Outcomes March 2006 March 2007
Total Calls Placed to the 800-Number 7,396,073 6,662,739
800-Number Callers Selecting the Automated System 2,461,102 2,352,578
Percent of Total Calls to Automated System 33.3 35.3
Percent of Calls Completed in Automation 51.5 60.4
Percent of Calls Abandoned in Automation 22.6 16.4
Percent of Calls in Automation Rerouted to Agent 22.4 21
Percent of Calls to After Hours Message 0.3 0.6
Percent of Calls Returned to Main Menu Incomplete 3.1 1.6

As the table shows, SSA realized some notable improvements in the performance of its automated system. For example, the percentage of calls completed in automation increased from 51.5 percent in March 2006 to 60.4 percent in March 2007. Additionally, the percent of calls abandoned in automation declined from 22.6 percent in March 2006 to 16.4 percent in March 2007.

We reviewed automation outcome data for FYs 2005 and 2006 and found that the percentage of calls abandoned in automation improved from one year to the next. However, the percentage of calls extended to a live agent increased from FY 2005 to FY 2006. Fifty-two percent of the calls placed in automation were completed in automation in both FYs 2005 and 2006. In FY 2005, 36 percent of the calls placed in automation were abandoned, and 11 percent were extended to live agents. In
FY 2006, 22 percent of the calls placed in automation were abandoned, and 23 percent were extended to live agents.

In terms of automation lessening the burden on live agents, the percentage of calls to the 800-number that were placed with live agents was relatively the same-63.9 percent in FY 2005 to 62.6 percent in FY 2006. While the 1.3 percent drop between the FYs is an improvement, the overall volume of calls to the 800-number rose between the FYs, from over 82.6 million calls in FY 2005 to over 88.4 million in FY 2006. Accordingly, the number of calls to live agents rose from 52.8 million in
FY 2005 to 55.3 million in FY 2006.

Automated Services

SSA's 800-number automated system offers 11 automated services to callers, including several with multiple subsets containing numerous data. Some subsets are caller decision points while others are data elements that highlight, for example, language differences (English versus Spanish) or technical differences (touch-tone versus speech-enabled). Callers to the automated menu can request pamphlets on programs or a Social Security Statement of earnings and benefits, listen to informational messages, create passwords, switch to Direct Deposit for Social Security payments, or notify the Agency of a change of address. As an example, initially a caller to the automated system is prompted to conduct the call in English or Spanish. Then, the caller selects 1 of the 11 menu service options (requests a pamphlet, for example) and then is guided through multiple subsets to complete their transaction (which pamphlet, where to mail the pamphlet, etc.).

To determine the effectiveness of the automated services, we reviewed the detailed management information available for each service. We found that different services had varying completion rates. In fact, an average of nearly 46 percent and 42 percent of the automated services had a 50 percent or less completion rate in March 2006 and in March 2007 (for additional details, see Appendices C and D). In addition, some automated services had completion rates of 20 percent or less (22 percent of the services in March 2006 and 17 percent of the services in March 2007).

We found the automated services callers selected most frequently had completion rates ranging from about 1 to 75 percent. The 20 automated services for March 2006 and 2007 with the heaviest transaction volume are detailed in Appendices E and F. We identified the 20 most frequently accessed services and found that, in March 2006, 16 of the 20 heaviest used services had completion rates of 50 percent or less, while, in March 2007, 14 of the 20 most frequently used services had completion rates of 50 percent or less. (See the table below for the distribution of the 20 busiest services by completion rates).

20 Most Frequently Used Automated Services
Completion Rate March 2006 March 2007
Services Volume Services Volume
1 to 25 percent 5 244,915 5 122,477
26 to 50 percent 11 617,419 9 646,584
51 to 75 percent 4 388,502 6 292,783
Total 20 1,250,836 20 1,061,844

We also found some of the automated services were used less than others, with some services rarely being selected. While over 2 million callers used SSA's automated services in March 2006, about 41 percent of SSA's automated services had 100 or fewer transactions, and 21 percent had 10 transactions or less. Similarly, while over 2 million callers used SSA's automated services in March 2007, over 42.7 percent of the automated services had 100 or fewer transactions, and 16.9 percent had
10 transactions or less.

Cost-effectiveness

Verizon charges SSA a basic usage fee for every call that is connected to the 800-number network (see Appendix G). The basic usage fee consists of flat rate fees, as well as a charge for each minute the call is connected. Additionally, the Agency is charged a flat rate fee for each automated system transaction selected by callers. The fees for automated transactions are charged even if the caller hangs up before completing their business.

SSA incurs additional costs if a caller is re-routed from the automated system to an agent or requests to speak to an agent before completing the automated transaction. In such instances, the Agency incurs the cost for the automated system transaction the caller selected as well as the additional costs related to remaining on the telephone longer to speak to an agent. Additional costs can be incurred when a caller abandons their call in automation and re-dials the 800-number to speak to an agent to complete their business. SSA incurs the cost of the unsuccessful use of the automated system, including the basic usage fee, as well as the additional basic usage fee and time charge for the additional call to the agent.

Spanish-language Callers

An average of 92 percent of callers who used the automated system in FYs 2006 and 2007 chose to use the English-language option, and 8 percent chose to use the Spanish-language option. While Spanish-language callers used the automated system far less, they had a much lower completion rate than English-language callers. In March 2006, there were 103,994 Spanish-language transactions on the 800-number automated services, and 54.5 percent of the callers hung up before completing their business. Similarly, in March 2007, there were 101,710 Spanish-language transactions on the 800-number, and approximately 54.9 percent of the callers hung up before completing their business. English-language callers had 31.21 and 28.3 percent
hang-up rates for March 2006 and March 2007, respectively.

Spanish-language callers do not have as many service options available as English-language callers. If Spanish-language callers encounter problems while in an automated option, they must either complete the option or hang up because they cannot opt out or be rerouted to speak to an agent or leave a callback message during
non-business hours.

MANAGEMENT INFORMATION

We attempted to determine the reasons for the high rate of incomplete calls in the 800-number automated system but found SSA did not collect caller-level data, which linked callers to their specific outcomes in the automated system. SSA did conduct 800-number caller contact surveys periodically. However, the surveys were based on call data that did not specifically target 800-number callers who had negative outcomes in the automated system. We wanted to use that type of specific information to contact callers to find out why they did not successfully complete their calls in automation. While Verizon supplies SSA with caller-level data for callers who choose to use the automated services, the caller-level data do not link callers to outcomes and do not distinguish whether a caller had a positive or negative outcome in the automated system.

We asked SSA whether data matching individual telephone numbers to specific outcomes were available. SSA informed us that the contract for 800-number services with Verizon did not require that the contractor supply such data. On our behalf, the Agency asked Verizon if such data were available and was told Verizon could provide the data at an estimated cost of $57,780 for 1 day's data. SSA reported that it did not have the funding to acquire the data on our behalf.

Without these data, we could not effectively identify and contact callers who had a negative outcome with the automated system. In our review of the two contracts to which SSA is a party related to its 800-number, we did not identify any language that would prevent SSA from receiving this type of data at no additional costs. In addition, a General Services Administration (GSA) contract specialist informed us that it appeared most of the data elements we believed were needed to identify and contact callers with negative outcomes in the automated system should be available at no additional costs under the GSA Federal Technology Service 2001 contract. The Federal Technology Service contract provides telecommunications service to Federal agencies. SSA is a party to this contract, which is managed by GSA.

PERFORMANCE MEASURES AND GOALS

Although one of SSA's automation goals is to reduce the burden on live agents, it does not have predetermined and quantitative performance measures and goals to measure the effectiveness of the 800-number automated system. We were unable to identify any such goals in SSA's Annual Performance Plan, Performance and Accountability Report, or Office of Telephone Service's Strategic Plan: FY 2005 - FY 2010. Without quantitative performance measures and goals, SSA lacks an important tool to monitor and determine whether the automated system is effectively serving the needs of callers.

CONCLUSION AND RECOMMENDATIONS

Based on our review, we believe SSA has been able to realize improvements in the effectiveness of its 800-number automated service system. However, there are still opportunities to improve the automated system to better serve the needs of its customers. To ensure the 800-number automated service system serves SSA customers in a cost-effective manner, we recommend SSA:

1. Have its Office of General Counsel determine whether SSA's current 800-number service related contract(s) allow for the provision of management information that identifies 800-number callers with unsuccessful outcomes in the automated system at no additional cost to the Agency.

2. Ensure future contracts routinely provide management information needed to identify 800-number callers with unsuccessful outcomes in the automated system at no additional cost to the Agency.

3. Determine, once the necessary call data are obtained, why any of its automated options have low completion rates and take necessary steps to improve the service provided.

4. Develop quantitative performance measures and goals to measure the effectiveness of the automated service system.

AGENCY COMMENTS

SSA agreed with our recommendations. See Appendix H for the full text of SSA's comments.

Patrick P. O'Carroll, Jr.

Appendices
APPENDIX A - Acronyms
APPENDIX B - Scope and Methodology
APPENDIX C - Automated Services with Completion Rates of 50 Percent or Less: March 2006
APPENDIX D - Automated Services with Completion Rates of 50 Percent or Less: March 2007
APPENDIX E - Analysis of the 20 Most Frequently Used Automated Services: March 2006
APPENDIX F - Analysis of the 20 Most Frequently Used Automated Services: March 2007
APPENDIX G - Cost Data for 800-Number Network Transaction Charges
APPENDIX H - Agency Comments
APPENDIX I - OIG Contacts and Staff Acknowledgments

Appendix A
Acronyms
ASR Automated Services Report
FO Field Office
FY Fiscal Year
GAO Government Accountability Office
GSA General Services Administration
OIG Office of the Inspector General
OTS Office of Telephone Services
OTSO Office of Telecommunications and Systems Operations
SSA Social Security Administration

Appendix B
Scope and Methodology

To accomplish our objectives, we reviewed:

The Office of Quality Assurance and Performance Assessment's 800-number Caller Survey Results for Fiscal Year (FY) 2005 that, in part, dealt with
800-number callers' experiences with the automated service system and difficulties reported.

The Office of Telephone Services (OTS) Strategic Plan FY 2005-FY 2010 -
May 2006 to obtain an understanding of its plans and goals for using automation to improve public service and network efficiency of its 800-number telephone services.

The Social Security Administration's (SSA) Office of the Inspector General Report, Performance Indicator Audit: 800-Number Access, September 2006.

A Government Accountability Office (GAO) report on Social Security Administration: Additional Actions Needed in Ongoing Efforts to Improve 800-Number Service, GAO-05-735, August 2005.

Documentation detailing the organization, service functions, and configuration of the 800-numbers automated system, including the network's new speech-recognition technology.

Documentation describing the data elements used for maintaining statistical data, as well as management information reports.

Documentation describing the data elements used to monitor the performance of the automated service system, as well as management information reports.

The Federal Technology Service 2001 contract between the General Services Administration and Verizon Business for telecommunications services for federal agencies (Contract No. GS00T99NRD2002) to determine the contractual obligations of the vendor to provide toll-free services management information.

SSA's contract with Verizon Business (Contract CCNS 0600-00-40649) to determine the vendor's contractual obligations to provide 800-number related management information to the Agency.

We interviewed OTS and Office of Telecommunications and Systems Operations staff to gain an understanding of the management information used to track how callers' experiences on the automated system are recorded, analyzed, and reported. We analyzed the 800-number transaction data in the Automated Services Reports, Password Services Reports, and Voice Portal Reports for the month of March 2006 and the month of March 2007 to identify and compare trends in caller outcomes, and performance of individual Automated Options. We also analyzed 800-number data for FYs 2005 and 2006.

We attempted to obtain caller-level data to identify callers with unsuccessful outcomes within SSA's 800-number automated system to determine why some callers did not successfully use the automated system. However, we found that SSA did not maintain such data on callers who used the automated service system. As a result, we were not able to perform our caller contact survey as originally planned.

We performed our audit from December 2006 through May 2007. We were unable to independently test the reliability of the data we used in our analyses because it was maintained by an outside contractor (Verizon), and we were not provided access to its source. Though, SSA reported to us that both Verizon and OTS test the reliability of data. Accordingly, we determined it was reasonable to use this data to meet our audit objectives.

The entities audited were the OTS under the Deputy Commissioner for Operations and the Office of Telecommunications and Systems Operations under the Deputy Commissioner for Systems. We conducted this audit in accordance with generally accepted government auditing standards.

Appendix C
Automated Services with Completion Rates of 50 Percent or Less: March 2006

The table below contains data on the volume of transactions for the Social Security Administration's (SSA) 800-number automated services, which had completion rates of 50 percent or less in March 2006. The automated services are maintained in one of three different management information databases: Automated Services Report (ASR), Voice Portal Report, or Password Services Report. The Automated Service field in the table lists the database that maintains the service, as well as the name of the transaction (for example, "Pamphlets-Understand Facts" or "Password Services-Title 2"). The field also indicates whether the item was the first automated transaction or a subsequent automated transaction selected by a caller. Lastly, it indicates the technological platform of the item, that is, whether the caller selected to use the Touchtone or Speak Freely Menus. The transactions selected refer to the number of times the automated service was selected in the month and the completion rate refers to the percentage rate of transactions that were completed.

Automated Service Language of Caller Transactions
Selected Completion Rate
1. ASR: Pamphlets-Understand Facts-Subsequent Selections-Touchtone English 3 0.00%
2. Password Services-Title 2: Overpayment Information-Touchtone Spanish 0 0.00%
3. Password Services-Title 2: Overpay Last-Touchtone Spanish 0 0.00%
4. Password Services-Title 2: Overpay Next-Touchtone Spanish 0 0.00%
5. Password Services-Password Change-Touchtone Spanish 61 1.64%
6. Password Services-Password Use-Touchtone Spanish 2,540 2.40%
7. Password Services-Password Block-Touchtone Spanish 75 2.67%
8. Password Services-Password Block-Touchtone English 559 4.47%
9. Password Services-Password Change-Touchtone English 177 6.21%
10. Password Services-Password Registration-Touchtone Spanish 258 7.75%
11. Password Services-Password Change-Speech English 86 8.14%
12. ASR: Mainframe-Change of Address/Telephone Number-Knowledge Based Authentication-Primary Selections-Speech English 41,901 9.49%
13. ASR: Mainframe-Change of Address-Knowledge-Based Authentication-Subsequent Selection English 63,110 10.24%
14. ASR: Mainframe-Change of Address/Telephone Number-Knowledge Based Authentication-Primary Selections-Speech English 2,684 12.63%
15. ASR: Mainframe - Speech Application for Direct Deposit-Password-Subsequent Selections-Speech English 78 12.82%
16. ASR: Mainframe-Change of Address-Speech Application for Direct Deposit English 872 12.84%
17. ASR: Mainframe-Change of Address-Knowledge Based Authentication-Subsequent Selection-Touchtone English 2,418 13.28%
18. ASR: Mainframe-Speech Application for Change of Address/Telephone Number-Primary Selections-Touchtone English 597 14.07%
19. Password Services: Get Password Request Code-Touchtone Spanish 1,259 15.49%
20. Password Services: Password Block Access-Speech English 115 15.65%
21. Password Services: Password Block Access-Speech English 690 15.94%
22. ASR: Mainframe-Speech Application for Direct Deposit Application-Password Authentication-Primary Selections-Touchtone English 31,491 16.16%
23. Title 16-Payment-Touchtone Spanish 11 18.18%
24. ASR: Mainframe-Application for Direct Deposit-Knowledge Based Application-Primary Selections-Touchtone English 2,626 18.20%
25. ASR: Mainframe-Application for Direct Deposit-Password Authentication-Primary Selections-Speech English 36,709 18.80%
26. ASR: Mainframe-Direct Deposit-Password Authentication-Primary Selections-Speech English 646 18.89%
27. ASR: Mainframe-Direct Deposit-Primary Selection English 62 19.35%
28. ASR: Forms-S-Med Application-Primary Selections-Speech English 5 20.00%
29. ASR: Pamphlets Survivors-Primary Selection Spanish 5 20.00%
30. ASR: Mainframe-MED CARD-Voice Capture Application-Primary Selections-Touchtone English 3,714 20.11%
31. ASR: Mainframe-Change of Address-Subsequent Selections-Speech English 49 20.41%
32. ASR: Mainframe-Application for Direct Deposit-Knowledge Based Authentication Subsequent Selections-Speech English 2,618 21.43%
33. ASR: Mainframe-Speech Application for Replacement Medicare Card-Subsequent Selections-Speech English 9,564 23.13%
34. ASR: Mainframe-Change of Address-Subsequent Selections-Speech English 50 24.00%
35. ASR: Mainframe-Application for Replacement Medicare Card-Primary Selections-Touchtone English 71,704 24.67%
36. ASR: Mainframe-Pamphlets -SS Women-Primary Selections-Touchtone Spanish 4 25.00%
37. ASR: Mainframe-Application for Benefits Verification-Primary Selections-Touchtone English 59,083 25.57%
38. ASR: Mainframe-Application for Benefit Verification-Proof of Income-Subsequent Selections-Touchtone English 5,515 26.94%
39. Password Services-Title 16: Payment-Speech English 505 29.31%
40. Password Services-Password Registration: Payment-Touchtone English 6,255 29.59%
41. Password Services-Get Password-Touchtone English 17,790 29.95%
42. ASR: Mainframe-Application for Benefits Verification-Subsequent Selection-Speech English 2,488 30.51%
43. ASR: Mainframe-Speech Application for Replacement Medicare Card-Subsequent Selections-Speech English 53,466 32.19%
44. Password Services-Password Registration-Speech English 7,844 33.08%
45. ASR: Forms M Voice Capture Application Request for a Social Security Card Form SS5-Primary Selection-Touchtone English 6 33.33%
46. Password Services-Get A Password Request Code-Speech English 5,845 33.98%
47. ASR: Mainframe-Application for Benefits Verification-Subsequent Selection-Speech English 33,764 35.50%
48. ASR: Database-Field Office Location-Primary Selection-Touchtone Spanish 36,585 35.54%
49. Password Services-Password Use-Touchtone English 12,786 35.88%
50. Voice Portal: Speech Menu: Application for Social Security Card-Form SS5 English 193,707 37.52%
51. ASR: Forms-S-Med1020-Application for a Social Security Statement- Primary Selections-Touchtone English 3,397 38.83%
52. ASR: Forms M Voice Capture Application Request for a Social Security Statement-Form 7004-Primary Selections-Touchtone Spanish 50 40.00%
53. ASR: Forms m-Request for Social Security Card-Form SS5-Primary Selection-Touchtone English 481 42.20%
54. Voice Portal: Main Menu: Request for a Social Security Card-Form SS5 English 119,681 42.29%
55. Voice Portal: Main Menu: Request for a Social Security Statement-Form 7004 English 36,317 42.67%
56. ASR: Pamphlets-Work Affects-Primary Selection-Touchtone Spanish 7 42.86.%
57. ASR: Forms-S Speech Application for Help with Medicare Prescription Drug Plan Costs-Form 1020 -Primary Selection-Touchtone English 33,763 43.69%
58. ASR: Forms-S-Speech Application for Help with Medicare Prescription Drug Plan Costs-Form 1020-Subsequent Selection-Speech English 20,477 43.75%
59. Voice Portal: Main Menu: Pamphlet English 3,387 44.52%
60. ASR: Database-Field Office Locator-Primary Selection-Primary Selection-Touchtone English 3,515 44.86%
61. Voice Portal: Speech Menu: Pamphlet English 2,820 45.43%
62. ASR: Forms-M-Application for M-PEB-SS5-Touchtone Spanish 602 46.68%
63. Password Services-Title 16: Payment-Touchtone English 1,028 47.67%
64. ASR: Forms-M Application for Replacement Card-Primary Selections-Touchtone Spanish 11,263 48.98%
65. ASR: Forms-M PEB 7004 Request for Replacement Card-Primary Selections-Touchtone English 2 50%

Appendix D
Automated Services with Completion Rates of 50 Percent or Less: March 2007
The table below contains data on the volume of transactions for the Social Security Administration's (SSA) 800-number's automated services, which had completion rates of 50 percent or less in March 2007. The automated services are maintained in one of three different management information databases: Automated Services Report (ASR), Voice Portal Report, or Password Services Report. The Automated Service field in the table lists the database that maintains the service, as well as the name of the transaction (for example, "Password Services-Password Change" or "Voice Portal-Application for Social Security Card"). The field also indicates whether the item was the first automated transaction or a subsequent automated transaction selected by a caller. Lastly, it indicates the technological platform of the item, that is, whether the caller selected to use the Touchtone or Speak Freely Menus. The transactions selected refer to the number of times the automated service was selected in the month and the completion rate refers to the percentage rate of transactions that were completed.

Automated Service Language of Caller Transactions
Selected Completion Rate
1. Password Services: Password Change-Touchtone Spanish 91 0.00%
2. Password Services: Title 16 Information Menu: Payment-Touchtone Spanish 2 0.00%
3. Voice Portal: Main Menu-Application for Social Security Card Form SS5- English 19 0.00%
4. Voice Portal: Main Menu-Pamphlet English 5 0.00%
5. Password Services: Get A Password Request Code-Touchtone Spanish 1,247 0.08%
6. Password Services: Password Registration-Touchtone Spanish 323 0.31%
7. Password Services: Password Registration-Speech English 12,467 1.46%
8. Password Services: Get A Password Request Code-Speech English 12,495 1.85%
9. Password Services: Password Use-Touchtone Spanish 2,587 2.16%
10. Password Services: Password Block Access-Speech English 329 6.69%
11. Password Services: Password Change-Speech English 188 6.91%
12. Password Services: Password Block Access-Touchtone Spanish 147 7.48%
13. Voice Portal: Main Menu-Form-7004 English 7 14.29%
14. ASR: Mainframe-Change of Address-Knowledge-Based Authentication-Primary Selections-Speech English 71,420 17.75%
15. ASR: Mainframe-Application Change of Address Knowledge Based Authentication-Subsequent Selections-Speech English 7,842 19.13%
16. ASR: Forms-M Request for a Social Security Statement PEB 7004-Subsequent Selections-Touchtone Spanish 9 22.22%
17. ASR: Mainframe-Application for Change of Address/Telephone Number-Password Authentication-Primary Selections-Speech English 827 23.82%
18. ASR: Mainframe-Application for Replacement Medicare Card-Subsequent Selections-Speech English 18,253 25.37%
19. ASR: Mainframe-Application for Direct Deposit-Knowledge Based Authentication-Primary Selections-Speech English 59,042 28.86%
20. ASR: Mainframe-Application for Replacement Medicare Card-Primary Selections-Speech English 76,549 29.79%
21. ASR: Mainframe-Application for Direct Deposit-Knowledge Based Authentication-Subsequent Selections-Speech English 8,325 30.53%
22. Password Services: Password Use-Speech English 5,063 30.89%
23. ASR: Mainframe-Change of Address Password Authentication-Subsequent Selections-Speech English 90 31.11%
24. ASR: Database-Field Office Locator-Primary Selections-Touchtone Spanish 34,216 34.95%
25. ASR: Forms-M PEB-SS5 Application for Social Security Card-Subsequent Selections-Touchtone Spanish 84 36.90%
26. ASR: Mainframe-Benefit Verification-Primary Selections-Speech English 63,675 37.15%
27. ASR: Mainframe-Benefit Verification-Subsequent Selections-Speech English 8,875 38.46%
28. ASR: Pamphlets-Survivors-Primary Selections-Touchtone Spanish 5 40.00%
29. Voice Portal: Speech Menu-Application for Social Security Card Form SS5 English 363,297 40.14%
30. ASR: Database-Field Office Locator-Subsequent Selections-Touchtone Spanish 3,940 43.38%
31. ASR: Forms-M - Peb-SS5 Application for Social Security Card-Touchtone Spanish 694 43.80%
32. ASR: Forms-S-Application for Help with Medicare Prescription Drug Plan Costs-MED 1020-Subsequent Selections-Speech English 22,088 44.39%
33. ASR: Forms-M Request for a Social Security Statement-Peb7004-Primary Selections-Touchtone Spanish 54 44.44%
34. ASR: Mainframe-Speech Application for Direct Deposit-Password Authentication-Primary Selections-Speech English 973 45.73%
35. ASR: Mainframe-Speech Application for Direct Deposit-Password Authentication-Subsequent Selections-Speech English 146 46.58%
36. Forms-M Voice Capture Application for Replacement Medicare Card-Primary Selections-Touchtone Spanish 10,517 48.07%
37. ASR: Pamphlets Voice Capture Application-SS Women-Primary Selections-Touchtone Spanish 10 50.00%

Appendix E
Analysis of the 20 Most Frequently Used Automated Services: March 2006

The table below contains data on the volume of transactions for the Social Security Administration's (SSA) 800-number's automated services which were the 20 most frequently used in March 2006. The automated services are maintained in one of three different management information databases: Automated Services Report (ASR), Voice Portal Report, or Password Services Report. The Automated Service field in the table lists the database that maintains the service, as well as the name of the transaction (for example, "ASR-Field Office Locator" or "Voice Portal-Form SS5"). The field also indicates whether the item was the first automated transaction or a subsequent automated transaction selected by a caller. Lastly, it indicates the technological platform of the item, that is, whether the caller selected to use the Touchtone or Speak Freely Menus. The transactions selected refer to the number of times the automated service was selected in the month and the completion rate refers to the percentage rate of transactions that were completed.

Some automated service transactions were pre-programmed to be "0 percent complete" or "never complete." These transactions were intended to capture data only on 800-number callers that either were not qualified or did not have the information needed to use the selected services, so the transactions were always defined as "never complete." Also, some automated service transactions were pre-programmed to be "100 percent complete" or "always complete." These transactions provided informational messages to the caller. The transactions were "always complete" even if the caller did not listen to the entire message. We did not include these transactions in our analysis of the 20 most frequently accessed services.

Automated Service Language of Caller Transactions
Selected Completion Rate

1. ASR: Field Office Locator - Primary Selections-Touchtone English 259,501 61.57%
2. Voice Portal Report: Speech Menu: Form SS5 English 193,707 37.52%
3. Voice Portal Report: Main Menu: Form SS5 English 119,681 42.29%
4. ASR: Database-Field Office Locator-Primary Selections-Speech English 93,965 74.82%
5. ASR: Application for Replacement Medicare Card-Primary Selections-Touchtone English 71,704 24.67%
6. ASR: Speech Application for Change of Address/Telephone Number-Knowledge Based Authentication-Primary Selections-Speech English 63,110 10.24%
7. ASR: Voice Capture Application for Benefit Verification -Primary Selections-Touchtone English 59,083 25.57%
8. ASR: Speech Application for Replacement Medicare Card-Primary Selections-Speech English 53,466 32.19%
9. ASR: Application for Change of Address/Telephone Number-Knowledge Based Authentication-Primary Selections-Touchtone English 41,901 9.49%
10. ASR: Mainframe-DD-KB-Primary Selection-Speech English 36,709 18.79%
11. ASR: Field Office Locator-Database Spanish 36,585 35.54%
12. Voice Portal-Form 7004-Speech English 36,317 42.67%
13. ASR: Ben Ver-Mainframe-Speech English 33,764 35.50%
14. ASR: Form S Med 1020- Primary Selection -Touchtone English 33,763 43.69%
15. ASR: Mainframe-DD-KB-Primary Selection-Touchtone English 31,491 16.16%
16. ASR: Forms-S MED 1020-Subsequent Selection-Speech English 20,477 43.75%
17. ASR: Forms-M SSS-Primary Selection Spanish 18,070 52.10%
18. PSR: Password Services: Get PRC English 17,790 29.95%
19. ASR: Database-Field Office Locator English 16,966 68.89%
20. Password Services: PW Use English 12,786 35.88%

Appendix F
Analysis of the 20 Most Frequently Used Automated Services: March 2007

The table below contains data on the volume of transactions for the Social Security Administration's (SSA) 800-number's automated services which were the 20 most frequently used in March 2007. The automated services are maintained in one of three different management information databases: Automated Services Report (ASR), Voice Portal Report, or Password Services Report. The Automated Service field in the table lists the database that maintains the item, as well as the name of the transaction (for example, "ASR-Medicare-Drug Program" or "Voice Portal-Form SS5"). The field also indicates whether the item was the first automated transaction or a subsequent automated transaction selected by a caller. Lastly, it indicates the technological platform of the item, that is, whether the caller selected to use the Touchtone or Speak Freely Menus. The transactions selected refer to the number of times the automated service was selected in the month and the completion rate refers to the percentage rate of transactions that were completed.

Some automated service transactions were pre-programmed to be "0 percent complete" or "never complete." These transactions were intended to capture data only on 800-number callers that either were not qualified or did not have the information needed to use the selected services, so the transactions were always defined as "never complete." Also, some automated service transactions were pre-programmed to be "100 percent complete" or "always complete." These transactions provided informational messages to the caller. The transactions were "always complete" even if the caller did not listen to the entire message. We did not include these transactions in our analysis of the 20 most frequently accessed services.

Automated Service Language of Caller Transactions
Selected Completion Rate
1. Voice Portal: Speech Menu-Form SS5 English 363,297 40.14%
2. ASR: Field Office Locator English 204,060 69.90%
3. ASR: Voice Capture Application for Replacement Medicare Card English 76,549 29.79%
4. ASR: Application for Change of Address/Telephone Number-Knowledge-Based Authentication) English 71,420 17.75%
5. ASR: Mainframe-Benefits Verification-Primary Selections-Speech English 63,675 37.15%
6. ASR: Application for Direct Deposit-Knowledge-Based Authentication-Primary Selections-Speech English 59,042 28.86%
7. ASR: Field Office Locator Database-Subsequent Selections-Speech English 40,632 67.75%
8. ASR: Field Office Locator Database-Primary Selections-Touchtone Spanish 34,216 34.95%
9. ASR: Form S MED 1020 - Subsequent Selection-Speech English 22,088 44.39%
10. ASR: Mainframe-MED Card-Subsequent Selection-Speech English 18,253 25.37%
11. ASR: Form SS5-Primary Selection-Touchtone Spanish 17,953 52.04%
12. Password Services: Obtain PRC-Speech English 12,495 1.85%
13. Password Services: Password Reg-Speech English 12,467 1.46%
14. Voice Portal: 7004 Menu-Speech English 12,336 64.07%
15. ASR: Form M-Bene-Primary Selection-Touchtone Spanish 11,421 56.14%
16. ASR: Form M-MED Card-Primary Selection-Touchtone -Touchtone Spanish 10,517 48.07%
17. ASR: Mainframe BEN VER- Subsequent Selection-Speech English 8,875 38.46%
18. ASR: Mainframe Direct Deposit-Subsequent Selection-Speech English 8,325 30.53%
19. ASR: Mainframe-Change of Address-Knowledge-Based Authentication English 7,842 19.13%
20. ASR: Forms-M-Pebes-Primary Selection Spanish 6,381 56.28%

Appendix G
Cost Data for 800-Number Network Transaction Charges
The table below details the fees Verizon charges SSA for calls connected to the 800-number network. The fees consist of flat rate fees, as well as a charge for each minute the call is connected. The Agency is also charged a flat rate fee for each automated system transaction selected by callers. The fees for automated transactions are charged even if callers hang up before completing their business.

National 800-Number Network Basic Charges Cost Unit
Usage Fee $0.0160 Per minute
Database Access/Menu Routing $0.0300 Per call
Speak Freely Main Menu $0.0856 Per call
Transaction Type Cost Unit
Change of Address/Telephone Number(Knowledge Based and Password Authentication) $0.1284 Per completed call in automation
Direct Deposit $0.1284 Per completed call in automation
Benefit Verification $0.2900 Per completed call in automation
Medicare Options
Request for Replacement Medicare Card (Modified Knowledge Based Authentication) $0.2900 Per completed call in automation
Request for SSA-1020 (with message) $0.2900 Per completed call in automation
Message: Medicare Prescription Drug Program Not Charged
Social Security Card Application
(Form SS-5) Spanish Manual Transcription $0.2900 Per completed call in automation
Field Office Locator Database Not Charged
Password Services
Password Services Account Status (Check your benefits) Not Charged
Social Security Statement Options
Form SSA-7004: To request Social Security Statement $0.2900 Per completed call in automation
Form SSA-7004: To request SS Statement with Different Estimates of Future Earnings $0.2900 Per completed call in automation
Form SS-5: To correct Name or Date of Birth $0.2900 Per completed call in automation
Spanish Manual Transcription $0.4483 Per completed call in automation
Message: Why Social Security Statement Was Sent Not Charged
Message: Address on Social Security Statement Not Charged
Transaction Type Cost Unit
Message: Earnings This Year/Last Year on Social Security Statement Not Charged
Pamphlet Requests
Pamphlet: Understanding the Benefits $0.2900 Per completed call in automation
Pamphlet: Retirement Benefits $0.2900 Per completed call in automation
Pamphlet: Disability Benefits $0.2900 Per completed call in automation
Pamphlet: Survivors Benefits $0.2900 Per completed call in automation
Pamphlet: How Work Affects Your Benefits $0.2900 Per completed call in automation
Pamphlet: Benefits for Children with Disabilities $0.2900 Per completed call in automation
Pamphlet: What Every Woman Should Know $0.2900 Per completed call in automation
Spanish Manual Transcription $0.4483 Per call completed in transcription
Informational Messages
Message: Payment Delivery Dates Not Charged
Message: Direct Deposit Not Charged
Message: Best Times to Call 800-Number Not Charged
Message: SSA Internet Address and Services Not Charged
Message: Supplemental Security Income Message Not Charged
Message: SSA-1099 Benefit Statement Not Charged
Message: Cost-of-Living Adjustment Not Charged

Appendix H
Agency Comments

MEMORANDUM

Date: October 4, 2007 Refer
Refer To: S1J-3

To: Patrick P. O'Carroll, Jr.
Inspector General

From: Larry W. Dye

Subject: Office of the Inspector General (OIG) Draft Report, "The Effectiveness of the Social Security Administration's 800-Number Automation" (A-02-07-17049)--INFORMATION

We appreciate OIG's efforts in conducting this review. Our comments on the recommendations are attached.

Please let me know if we can be of further assistance. Staff inquiries may be directed to Ms. Candace Skurnik, Director, Audit Management and Liaison Staff, at (410) 965-4636.

Attachment

COMMENTS ON THE OFFICE OF THE INSPECTOR GENERAL'S (OIG) DRAFT REPORT, "THE EFFECTIVENESS OF THE SOCIAL SECURITY ADMINISTRATION'S 800-NUMBER AUTOMATION SERVICE" (A-02-07-17049)

Thank you for the opportunity to review and provide comments on this draft report. The objective of the report was to review the effectiveness of our 800-number automation service. However, the report only attempts to evaluate the effectiveness of automated services based on management information about completion and abandonment. We believe this type of evaluation is of limited use because it does not include a caller perspective. In addition, it seems to be based on two presumptions we do not believe are supportable, in light of various surveys we have conducted with 800-number callers; i.e., that 1) every automated service accessed represents the real reason for the call; and 2) inadequacy of the service is measured by a caller not completing their business through the automated service. For example, our 800-Number Speech Recognition Survey, released in December 2006, included about 400 callers who, according to the management information, had used the automated service for change of address or direct deposit. However, when asked why they called the 800-number that day, about 40 percent said the main reason for their call was something else. The survey was conducted based on contacts in March 2006, when half the callers had received the touchtone and half the speech recognition menu. Regardless of the menu system encountered, the proportion of "inadvertent" selections was the same: Callers using the touchtone system selected automated services they did not want; callers using the speech recognition system expressed what they were calling about in a way that directed them to an automated service that did not meet their needs.

We note that OIG recognizes the need for a caller perspective, but explains that a survey was not pursued because of the inability to target a sample of automated service users with unsuccessful outcomes. However, relevant information about callers who have accessed automated services, both successfully and unsuccessfully, can be obtained from more generalized samples of 800-number callers, as we have done in our annual 800-Number Caller Surveys and in several special study surveys. The Fiscal Year 2006 800-Number Caller Survey addressed the reasons callers who had used an automated service were unable to complete all of their business in the automated services. The majority of the responses reflected the following caller choice or circumstances: 23 percent - wanted the reassurance of talking to an agent; 22 percent - the automated service did not cover the caller's specific situation; 14 percent - had other business that could not be handled through an automated service; and 6 percent - the caller did not have the necessary information available. Another 22 percent of responses indicated the system advised the caller of the need to transfer to an agent, which could be a reflection of our screening criteria working as intended. Only 8 percent said it was just too hard to use the automated service.

Our comments on the draft recommendations are as follows.

Recommendation 1

Have its Office of General Counsel determine whether SSA's current 800-number service related contract(s) allow for the provision of management information that identifies 800-number callers with unsuccessful outcomes in the automated system at no additional cost to the Agency.

Comment

We agree. We will evaluate the current contracts to determine if we can obtain management information that identifies 800-number callers with unsuccessful outcomes in the automated system at no additional cost.

Recommendation 2

Ensure future contracts routinely provide management information needed to identify 800-number callers with unsuccessful outcomes in the automated system at no additional cost to the Agency.

Comment

We agree. We will ensure that future contracts include the requirement that a contractor provide such management information (this requirement would be included as part of the statement of work). However, it should be noted that this type of information may come at a cost. We believe even if the collection and reporting of management information for the 800-number environment is not a lined charge item in the contract, the cost of that service will be rolled into some greater part of the total price of the contract. If so, we must weigh the incurrence of such a cost with other competing Agency priorities.

Recommendation 3

Determine, once the necessary call data are obtained, why any of its automated options have low completion rates and take necessary steps to improve the service provided.

Comment

We agree. Once the data is made available, we will utilize the information as a tool to identify problems/weaknesses in our automated service applications. We will also work to improve the service and caller experience of our 800-number callers.

Recommendation 4

Develop quantitative performance measures and goals to measure the effectiveness of the automated service system.

Comment

We agree. We plan to develop outcome-oriented performance measures with targets that promote continuous improvement in order to assess the ongoing effectiveness of the 800-number automated service system. We expect to start the development of the performance measures and goals in Fiscal Year 2008.

Appendix I
OIG Contacts and Staff Acknowledgments
OIG Contacts
Tim Nee, Director, 212-264-5295
Victoria Abril, Audit Manager, 212-264-0504
Acknowledgments
In addition to those named above:
Christine Hauss, Senior Program Analyst, 212-264-5826

For additional copies of this report, please visit our web site at www.socialsecurity.gov/oig or contact the Office of the Inspector General's Public Affairs Specialist at (410) 965-3218. Refer to Common Identification Number A-02-07-17049.

Overview of the Office of the Inspector General
The Office of the Inspector General (OIG) is comprised of our Office of Investigations (OI), Office of Audit (OA), Office of the Chief Counsel to the Inspector General (OCCIG), and Office of Resource Management (ORM). To ensure compliance with policies and procedures, internal controls, and professional standards, we also have a comprehensive Professional Responsibility and Quality Assurance program.

Office of Audit
OA conducts and/or supervises financial and performance audits of the Social Security Administration's (SSA) programs and operations and makes recommendations to ensure program objectives are achieved effectively and efficiently. Financial audits assess whether SSA's financial statements fairly present SSA's financial position, results of operations, and cash flow. Performance audits review the economy, efficiency, and effectiveness of SSA's programs and operations. OA also conducts short-term management and program evaluations and projects on issues of concern to SSA, Congress, and the general public.

Office of Investigations
OI conducts and coordinates investigative activity related to fraud, waste, abuse, and mismanagement in SSA programs and operations. This includes wrongdoing by applicants, beneficiaries, contractors, third parties, or SSA employees performing their official duties. This office serves as OIG liaison to the Department of Justice on all matters relating to the investigations of SSA programs and personnel. OI also conducts joint investigations with other Federal, State, and local law enforcement agencies.

Office of the Chief Counsel to the Inspector General
OCCIG provides independent legal advice and counsel to the IG on various matters, including statutes, regulations, legislation, and policy directives. OCCIG also advises the IG on investigative procedures and techniques, as well as on legal implications and conclusions to be drawn from audit and investigative material. Finally, OCCIG administers the Civil Monetary Penalty program.

Office of Resource Management
ORM supports OIG by providing information resource management and systems security. ORM also coordinates OIG's budget, procurement, telecommunications, facilities, and human resources. In addition, ORM is the focal point for OIG's strategic planning function and the development and implementation of performance measures required by the Government Performance and Results Act of 1993.