MEMORANDUM

Date: April 30, 2003

To: The Commissioner

From: Inspector General

Subject: Performance Indicator Audit: Enumeration (A-02-02-11088)

We contracted with PricewaterhouseCoopers (PwC) to evaluate the data used to measure 18 of the Social Security Administration's Fiscal Year 2002 performance indicators established to comply with the Government Performance and Results Act. The attached final report presents the results of one of the performance indicators PwC reviewed. The objective of this audit was to assess the reliability of the data used to measure the percent of original and replacement Social Security number cards issued within 5 days of receiving all necessary documentation.

Please comment within 60 days from the date of this memorandum on corrective action taken or planned on each recommendation. If you wish to discuss the final report, please call me or have your staff contact Steven L. Schaeffer, Assistant Inspector General for Audit, at (410) 965-9700.

James G. Huse, Jr.

OFFICE OF
THE INSPECTOR GENERAL

SOCIAL SECURITY ADMINISTRATION

Performance Indicator Audit:
Enumeration

April 2003

A-02-02-11088

AUDIT REPORT

Mission

We improve SSA programs and operations and protect them against fraud, waste, and abuse by conducting independent and objective audits, evaluations, and investigations. We provide timely, useful, and reliable information and advice to Administration officials, the Congress, and the public.

Authority

The Inspector General Act created independent audit and investigative units, called the Office of Inspector General (OIG). The mission of the OIG, as spelled out in the Act, is to:
Conduct and supervise independent and objective audits and investigations relating to agency programs and operations.
Promote economy, effectiveness, and efficiency within the agency.
Prevent and detect fraud, waste, and abuse in agency programs and operations.
Review and make recommendations regarding existing and proposed legislation and regulations relating to agency programs and operations.
Keep the agency head and the Congress fully and currently informed of problems in agency programs and operations.

To ensure objectivity, the IG Act empowers the IG with:
Independence to determine what reviews to perform.
Access to all information necessary for the reviews.
Authority to publish findings and recommendations based on the reviews.

Vision

By conducting independent and objective audits, investigations, and evaluations, we are agents of positive change striving for continuous improvement in the Social Security Administration's programs, operations, and management and in our own office.

MEMORANDUM

To: Office of the Inspector General

From: PricewaterhouseCoopers LLP

Date: April 16, 2003

Subject: Performance Indicator Audit: Enumeration (A-02-02-11088)

The Government Performance and Results Act (GPRA) of 1993 requires the Social Security Administration (SSA) to develop performance indicators that assess the relevant service levels and outcomes of each program activity set forth in its budget. GPRA also calls for a description of the means employed to verify and validate the measured values used to report on program performance. The objective of this audit was to assess the reliability of the following Fiscal Year (FY) 2002 GPRA performance indicator:

1.1.1 Performance Indicator FY 2002 Goal

Percent of original and replacement Social Security number (SSN) cards issued within 5 days of receiving all necessary documentation. 97 percent

See Appendix A for a description of the audit scope and methodology.

BACKGROUND

SSA offers retirement and long-term disability programs to the general public. Old-Age, Survivors and Disability Insurance (OASDI) is authorized under title II of the Social Security Act (Act) and includes Retirement and Survivors Insurance and Disability Insurance. Through the OASDI program, eligible workers, and sometimes their families, receive monthly benefits if they retire at an appropriate age or are found to have a disability that either prevents them from engaging in substantial gainful activity for at least 12 continuous months or can be expected to result in death. The Supplemental Security Income program is authorized under title XVI of the Act and provides monthly payments to aged, blind and/or disabled individuals based on financial need and medical requirements.

The SSN is a nine-digit number assigned to almost everyone in the United States. It uniquely identifies individuals applying for or receiving benefits and is critical in implementing SSA's programs and operations.

SSA assigns SSNs to individuals through the enumeration process. SSA uses the Modernized Enumeration System (MES) to track and assign SSNs. Individuals apply for SSNs by filing SS-5 forms at their local field office or by postal mail. Parents of newborns can obtain SSNs for their children through the enumeration-at-birth process. SSA has a goal of issuing new and replacement SSN cards within 5 working days of receiving a completed SS-5 form and its associated documentation.

This report reviews the performance indicator that measures the "Percent of original and replacement SSN cards issued within 5 days of receiving all necessary documentation." The performance indicator is determined by calculating the difference between the date when all documentation was received and either the date when the SSN was assigned or when the request for a replacement SSN card was processed. SSA defines the date when all documentation was received as the date the SSA employee signs and dates the certification block on the SS-5 form. The SSA employee's signature indicates that the SS-5 is complete and all documentation and evidence have been received and reviewed. The certification block date is entered into the MES system when the SS-5 is key-entered and processed. A batch computer process assigns the SSN and records the date the SSN was assigned or when the request for a replacement SSN card was processed. SSA then prints and mails new and replacement SSN cards via the United States Postal Service.

To determine the annual percentage of SSNs issued within 5 days of receiving all necessary documentation, SSA divides the number of records where the date difference between SSN issuance and receipt of documentation was 5 days or less by the total number of records where all necessary documentation was received for a month and then averages the monthly percentage over the entire year.

RESULTS OF REVIEW

We evaluated the processes, controls, and accuracy of the underlying data that support this performance indicator. Our internal controls test, described below, indicated that the data was not reliable. We found that SSA has inadequate management controls for data entry of the certification date in the MES system. We also found that the performance indicator was calculated incorrectly and had inadequate documentation.

PERFORMANCE INDICATOR DATA WAS NOT RELIABLE

During our review of the performance indicator data, we tested the data quality for a random sample of 105 records from the MES data history file for June 2002. We compared the SS-5 certification block date from the paper form to the corresponding date in the MES system to determine whether they matched. We found that 17 of the 105 records sampled had an inaccurate certification date in the MES system. Based on this test, we estimate the overall error rate for the certification date in the MES system at 16.2 percent (+/- 7.0 percent at a 95-percent confidence interval). (Additional information on our controls test can be found in Appendix D.) Given this high error rate, we found the underlying performance indicator data was not reliable.

To determine the impact of these errors on the performance indicator result, we recalculated the value of the performance indicator for our sample of June 2002 applications based on the MES system date and the paper SS-5 form. Of the 17 errors where the dates did not match, 7 changed the value of the performance indicator by either increasing the processing time from 5 days or less to more than 5 days or by decreasing the processing time from more than 5 days to 5 days or less. Based on the certification date in the MES system, 98.1 percent of the records were processed within 5 days. Based on the paper SS-5 form, 95.2 percent of the records were processed within 5 days. The difference between these numbers was not statistically significant, as there were errors that increased the processing time and errors that decreased the processing time that cancelled each other out. However, we cannot be confident that SSA does not have a systemic error in the population data that will impact the value of the FY 2002 performance indicator.

We also identified a data integrity issue with the MES history data file. We found 132 of 1,134,522 records with a negative 1-day processing time from the June 2002 MES history data file. This issue did not have a material impact on the overall performance indicator result due to the small number of records, but indicates a weakness in the MES application controls. We found that SSA received these records in Germany or on the other side of the International Date Line in the Philippines, Mariana Islands, or Guam. Records key entered early in the morning in Germany could be enumerated late at night on the previous day in the United States leading to the negative 1-day processing time. Similarly, records received on the other side of the International Date Line are entered on 1 day and enumerated the previous day in the United States.

Management Controls Were Inadequate

We tested the internal controls for the data entry of the certification date as described above. We found that 17 of the 105 records tested had an incorrect certification date in the MES system. This error rate exceeds the allowable rate of 2 errors for a sample of 105 records from the General Accounting Office's Financial Audit Manual. We concluded that the management controls for data entry of the certification date were not operating effectively.

Performance Indicator Results Were Calculated Incorrectly

SSA calculates the value of the performance indicator each month. At the end of the FY, SSA averages the 12 monthly processing rates to calculate the result for the entire FY. This method of calculation assumes that SSA processes the same number of SSNs every month. In fact, a review of SSA workload shows that enumeration requests varied in FY 2002 from 848,204 in December 2001 to 1,505,109 in August 2002. A more accurate calculation would divide the annual count of records processed within 5 days by the total number of requests for the year.

We calculated the performance indicator value using SSA's current method and the more accurate calculation suggested above for October 2001 through August 2002. We found a .03 percent difference between the two calculations. While the difference is relatively small, it could impact the final result for the performance indicator due to rounding.

Performance Indicator Documentation Was Inadequate

We found that SSA did not maintain current and adequate documentation for the calculation of this performance indicator. The following items were missing:

Description of the enumeration process and how data is extracted from that process to report the performance indicator.
Description of data extracts from the MES, where they are stored, and how they are used to report the performance indicator.
Documentation regarding the daily, weekly, and monthly batch processes used to report and calculate the performance indicator results.
Data definitions of each field and data file used in the process.
Systems documentation on any system edits performed on the data including: purges, deletions, and data updates.

Office of Management and Budget Circular Number A-123, Management Accountability and Control, section II, page 6, states, in part, that, "The documentation for transactions, management controls, and other significant events must be clear and readily available for examination."

1.1.1.1.1.1.1.1.1 OTHER MATTERS

Since the start of our audit, SSA made several changes to the enumeration process. In July 2002, the Agency started verifying birth certificates for children born in the United States 1 year or older for new SSN cards. This can require purchasing a copy of the birth certificate from the State, which can take several weeks. SSA also started verifying all non-citizen Immigration and Naturalization Service (INS) documents with the INS for new and replacement SSN cards in September 2002. Depending on availability of INS information, this can take several weeks, or more than a month. Since SSA does not certify that all documentation is received until after these verifications are complete, these changes should not impact their ability to process new and replacement SSN card requests within 5 days. However, any delays performing these verifications will not be captured and reported as part of this performance measure. We believe SSA should internally monitor the processing time for the verifications to ensure they are reasonable.

1.1.1.2

1.1.1.3 CONCLUSIONS AND RECOMMENDATIONS

We found that the data SSA used to calculate the performance indicator was unreliable. We found that the performance indicator had inadequate management controls, was calculated incorrectly, and had insufficient documentation. Our recommendations are as follows:

1. Improve management controls

We recommend that SSA improve management controls for data entry of the certification date to ensure that the correct date is entered into the MES system. For example, SSA could conduct random audits to compare the paper SS-5 form with the MES system data to ensure the date is entered correctly.

2. Ensure processing times are accurate

We recommend that SSA include additional date edits so that processing times are accurately calculated.

3. Calculate performance indicator using exact annual counts

We recommend that SSA calculate this performance indicator by dividing the number of requests processed within 5 days by the total number of requests for the FY.

4. Document the performance indicator calculations

We recommend that SSA document its methodology to calculate the performance indicator results.

1.1.1.4 AGENCY COMMENTS

SSA agreed with Recommendation 1. SSA disagreed with Recommendation 2 since it did not impact the overall performance indicator result and there are limited Agency resources for system changes. SSA did not object to Recommendation 3; however, they stated that with limited Agency resources for system changes, they would not pursue the recommendation since the difference in the performance indicator calculation was .03 percent. With respect to Recommendation 4, SSA stated that the Agency no longer externally reports this performance indicator.

1.1.1.5 PWC RESPONSE

We understand that Agency resources are limited for system changes to address Recommendations 2 and 3. We therefore recommend that SSA address records with negative processing times by adding date edits (Recommendation 2) within its list of planned system changes and when other MES changes are planned. In the meantime, SSA could address the negative processing times at the backend of the process when calculating the final performance indicator result. We also recommend that SSA make software changes to use exact counts (Recommendation 3) when changes to the MES are scheduled. Since the raw numbers for exact counts are available, SSA could verify the performance indicator results manually in the interim. If SSA plans to continue to report the performance indicator results internally, we recommend it document the calculation based on good management practices (Recommendation 4).

1.1.1.5.1 Appendices

APPENDIX A - Scope and Methodology
APPENDIX B - Acronyms
APPENDIX C - Flowcharts and Descriptions
APPENDIX D - Internal Controls Test
APPENDIX E - Agency Comments

Appendix A
Scope and Methodology

We conducted this audit to examine the performance related to timeliness in issuing original and replacement Social Security number cards. We evaluated the underlying data and calculations for this performance indicator and reviewed its' controls and documentation. Our audit was performed from May 1, 2002 through December 18, 2002 as follows:
Reviewed Office of the Inspector General (OIG) reports related to the enumeration process including:
Social Security Administration (SSA) OIG, Social Security Number Integrity: An Important Link in Homeland Security (A-08-02-22077), May 2002.
SSA OIG, Effectiveness of Internal Controls in the Modernized Enumeration System (A-08-97-41003), September 2000.
SSA OIG, Performance Measure Review: Reliability of the Data Used to Measure Social Security Number Request Processing (A-02-99-01009), March 2000.
SSA OIG, Performance Measure Audit: Timely Issuance of Social Security Number Cards (A-02-97-93003), April 1998.
Obtained an extract of 3 months of enumeration data (June through August 2002) from the Modernized Enumeration System (MES);
Obtained a random sample of 105 SS-5 forms and compared data from the forms to data in the MES system to ensure data integrity and test data entry internal controls; and
Tested the data extract for various data quality controls.
In conducting this audit, we also:
Reviewed the SSA's Performance and Accountability Report for Fiscal Year (FY) 2001, SSA's Annual Performance Plan for FY 2001, and SSA's Revised Performance Plan for FY 2002 to determine related milestones, definition, and data sources for each performance indicator;
Reviewed Office of Management and Budget Circular No. A-11 Part 6: Preparation and Submission of Strategic Plans, Annual Performance Plans, and Annual Program Performance Reports (June 2002) to ensure that the performance indicator is appropriate; and
Interviewed the individuals working with the performance indicator to gain an understanding of the data, the process to calculate the data, and any related reporting processes.
Our audit was limited to testing at SSA's Headquarters in Woodlawn, Maryland and SS 5 testing. The procedures we performed were in accordance with the American Institute of Certified Public Accountants' Statement on Standards for Consulting Services and the General Accounting Office's Government Auditing Standards ("Yellow Book") for performance audits.

Appendix B
Acronyms

FO Field Office
FY Fiscal Year
GAO General Accounting Office
GPRA Government Performance and Results Act
INS Immigration and Naturalization Service
MES Modernized Enumeration System
MI Management Information
OIG Office of the Inspector General
OASDI Old-Age, Survivors and Disability Insurance
SSA Social Security Administration
SSN Social Security Number

Appendix C
Flowcharts and Descriptions

Enumeration Process:

Enumeration Process:

The enumeration process begins when applicants with all required documentation request new or replacement Social Security number (SSN) cards.
If the applicant is interviewed at the field office (FO), the FO employee enters the Form SS-5 into the Modernized Enumeration System (MES) through an "Interview" on-line mode. If the applicant mails the completed Form SS-5 and documentation to the FO, the FO employee enters the Form SS-5 into MES through the batch mode.
MES processes the application.
The application is compared to the NUMIDENT and the ALPHIDENT to verify the application is valid.
If the enumeration request is invalid, the file is assigned an "Investigate" status and returned for further research.
If the enumeration request is valid, the file is assigned a completed status.
If the request is for an original SSN card, an SSN is assigned.
The SSN cards are printed and sent to the mailroom.
A third-party vendor picks up the SSN cards and delivers them to the post office.
The management information (MI) database is updated daily.
Monthly reports are produced from the MI database.

Appendix D
Internal Controls Test
We used the General Accounting Office's (GAO) Financial Audit Manual to determine our audit sampling criteria for internal controls related to entry of the certification date in the Modernized Enumeration System (MES). We received 3 months of MES data for June through August 2002 from the Social Security Administration (SSA). To test the internal controls for this process, we randomly selected 105 records from the MES data history file for June 2002 records.
We obtained copies of the paper SS-5 forms and compared the certification date on the paper SS-5 form to the corresponding date in the MES system. We also calculated the processing time based on the certification date on the paper SS-5 form and the date in the MES system. Our first test determined whether the date on the form matched the date in the system. The second test assessed the impact on the performance indicator. For example, if SSA's processing time was 6 days based on the paper SS-5 form and 1 day based on the MES system, we counted this as an error according to both tests. First, the paper SS-5 form and MES date did not match. Second, the error changed the performance indicator result.
Our results were as follows:
88 records matched (SS-5 certification block date matched MES system date);
17 records did not match (SS-5 certification block date did not match the MES system date);
7 of the 17 records that did not match also changed the performance indicator result by either:
Decreasing the processing time from more than 5 days to 5 days or less (2 records); or
Increasing the processing time from 5 days or less to more than 5 days (5 records)
10 of the 17 records that did not match did not change the performance indicator results
103 records were processed within 5 days, based on the paper SS-5 certification date, versus 100 records, based on the MES system date.
We conclude that the MES system certification date is not reliable, as the number of errors (17 out of 105) exceeds GAO's guidelines in the Financial Audit Manual (2 errors out of 105 records tested).
Given the above results and conclusions of the internal controls test, we performed additional calculations with the sample data. Our findings from this internal controls test are as follows:
We estimate the overall error rate for the MES data history file for the certification date is 16.2 percent (+/- 7.0 percent at a 95-percent confidence interval).
We calculated the standard deviation to determine whether the difference between the performance indicator results for the sample was significant. Based on the MES system certification date, 98.1 percent of the requests were processed within 5 days in June 2002. Based on the paper SS-5 form certification date, 95.2 percent of the requests were processed within 5 days in June 2002. We found that the difference between the performance indicator results (2.9 percent) was less than the test value (5.09 percent) at a 95-percent confidence interval.
The first result quantifies the magnitude of the error rate for the certification date in the MES system. As a result, we conclude that the performance indicator data is not reliable. The second result shows that the errors in the MES system certification date did not result in a statistically significant change to the performance indicator value for the data sampled. Therefore, we did not find a systemic bias in any one direction for the performance indicator. However, until the management controls are improved for data entry of the certification date, we cannot be confident that these errors will not impact the performance indicator value.
Appendix E

Agency Comments

MEMORANDUM

Date: April 3, 2003

To: James G. Huse, Jr.
Inspector General

From: Larry W. Dye
Chief of Staff

Subject: Office of the Inspector General (OIG) Draft Report, "Performance Indicator Audit: Enumeration" (A-02-02-11088)-INFORMATION

We appreciate OIG's efforts in conducting this review. Our comments on the draft report content and recommendations are attached.

Staff questions may be referred to Laura Bell on extension 52636.

SSA Response

COMMENTS ON THE OFFICE OF THE INSPECTOR GENERAL (OIG) DRAFT REPORT "PERFORMANCE INDICATOR AUDIT: ENUMERATION"
(AUDIT NO. A-02-02-11088)

Thank you for the opportunity to review and comment on the draft report contents and recommendations. Our responses to the specific recommendations are provided below.

Recommendation 1

The Social Security Administration (SSA) should improve management controls.

SSA Response

We agree. Currently our field offices conduct random reviews of SS-5s as part of their weekly enumeration sample reviews. We will issue a specific request for field offices to review the certification date block (DCL) for the SS-5 and the Modernized Enumeration System (MES) "Date all evidence received" (DER) field in their weekly enumeration sample reviews. We will also remind field offices to ensure the date in the SS-5 certification date block and the DER field on the MES system reflect the date all required evidence is submitted. We plan to issue the specific request and reminder notices to the field the first week of April 2003.

Recommendation 2

SSA should ensure processing times are accurate.

SSA Comment

We disagree. The recommendation does not provide any specific information about what type of edits are being proposed, and the report refers to a problem with a very small number of foreign cases that caused a one-day negative processing time due to time differences around the world. In addition, according to the report, the very small number of cases did not impact the overall performance. Therefore, with limited Agency resources available for systems changes, we do not plan to pursue a systems project to address a limited problem that had no impact on overall performance.

Recommendation 3

SSA should calculate the performance indicator using exact annual counts.

SSA Comment

We have no objection to using exact annual counts; however, with limited Agency resources available for systems changes, we do not plan to pursue a systems project to revise the performance calculation since, according to the report, the difference in calculation was only .03 percent.

Recommendation 4

SSA should document the performance indicator calculations.

SSA Response

The Agency no longer externally reports on this performance measure. If we report on the performance measure in the future, we would consider documenting how we calculate performance results.

Overview of the Office of the Inspector General

Office of Audit
The Office of Audit (OA) conducts comprehensive financial and performance audits of the Social Security Administration's (SSA) programs and makes recommendations to ensure that program objectives are achieved effectively and efficiently. Financial audits, required by the Chief Financial Officers' Act of 1990, assess whether SSA's financial statements fairly present the Agency's financial position, results of operations and cash flow. Performance audits review the economy, efficiency and effectiveness of SSA's programs. OA also conducts short-term management and program evaluations focused on issues of concern to SSA, Congress and the general public. Evaluations often focus on identifying and recommending ways to prevent and minimize program fraud and inefficiency, rather than detecting problems after they occur.

Office of Executive Operations
The Office of Executive Operations (OEO) supports the Office of the Inspector General (OIG) by providing information resource management; systems security; and the coordination of budget, procurement, telecommunications, facilities and equipment, and human resources. In addition, this office is the focal point for the OIG's strategic planning function and the development and implementation of performance measures required by the Government Performance and Results Act. OEO is also responsible for performing internal reviews to ensure that OIG offices nationwide hold themselves to the same rigorous standards that we expect from SSA, as well as conducting investigations of OIG employees, when necessary. Finally, OEO administers OIG's public affairs, media, and interagency activities, coordinates responses to Congressional requests for information, and also communicates OIG's planned and current activities and their results to the Commissioner and Congress.

Office of Investigations
The Office of Investigations (OI) conducts and coordinates investigative activity related to fraud, waste, abuse, and mismanagement of SSA programs and operations. This includes wrongdoing by applicants, beneficiaries, contractors, physicians, interpreters, representative payees, third parties, and by SSA employees in the performance of their duties. OI also conducts joint investigations with other Federal, State, and local law enforcement agencies.

Counsel to the Inspector General
The Counsel to the Inspector General provides legal advice and counsel to the Inspector General on various matters, including: 1) statutes, regulations, legislation, and policy directives governing the administration of SSA's programs; 2) investigative procedures and techniques; and 3) legal implications and conclusions to be drawn from audit and investigative material produced by the OIG. The Counsel's office also administers the civil monetary penalty program.