A-02-05-15119 - Alternate Format

OFFICE OF
THE INSPECTOR GENERAL

SOCIAL SECURITY ADMINISTRATION

PERFORMANCE INDICATOR AUDIT:
JOB ENRICHMENT OPPORTUNITIES

August 2005

A-02-05-15119

AUDIT REPORT

Mission

We improve SSA programs and operations and protect them against fraud, waste, and abuse by conducting independent and objective audits, evaluations, and investigations. We provide timely, useful, and reliable information and advice to Administration officials, the Congress, and the public.

Authority

The Inspector General Act created independent audit and investigative units, called the Office of Inspector General (OIG). The mission of the OIG, as spelled out in the Act, is to:

Conduct and supervise independent and objective audits and investigations relating to agency programs and operations.
Promote economy, effectiveness, and efficiency within the agency.
Prevent and detect fraud, waste, and abuse in agency programs and operations.
Review and make recommendations regarding existing and proposed legislation and regulations relating to agency programs and operations.
Keep the agency head and the Congress fully and currently informed of problems in agency programs and operations.

To ensure objectivity, the IG Act empowers the IG with:

Independence to determine what reviews to perform.
Access to all information necessary for the reviews.
Authority to publish findings and recommendations based on the reviews.

Vision

By conducting independent and objective audits, investigations, and evaluations, we are agents of positive change striving for continuous improvement in the Social Security Administration's programs, operations, and management and in our own office.

MEMORANDUM
Date: August 12, 2005

To: The Commissioner

From: Inspector General

Subject: Performance Indicator Audit: Job Enrichment Opportunities (A-02-05-15119)

We contracted with PricewaterhouseCoopers, LLP (PwC) to evaluate 16 of the Social Security Administration's performance indicators established to comply with the Government Performance and Results Act. Attached is the final report presenting the results of one of the performance indicators PwC reviewed. For the performance indicators included in this audit, PwC's objectives were to:

Assess the effectiveness of internal controls and test critical controls over the data generation, calculation, and reporting processes for the specific performance indicator.

Assess the overall reliability of the performance indicator's computer processed data. Data are reliable when they are complete, accurate, consistent and are not subject to inappropriate alteration.

Test the accuracy of results presented and disclosed in the Fiscal Year 2004 Performance and Accountability Report.

Assess if the performance indicator provides a meaningful measurement of the program it measures and the achievement of its stated objective.

This report contains the results of the audit for the following indicator:

Number of Job Enrichment Opportunities (includes headquarters, component and regional development programs).

Please provide within 60 days a corrective action plan that addresses each recommendation. If you wish to discuss the final report, please call me or have your staff contact Steven L. Schaeffer, Assistant Inspector General for Audit, at
(410) 965-9700.

Patrick P. O'Carroll, Jr.

MEMORANDUM

Date: July 28, 2005

To: Inspector General

From: PricewaterhouseCoopers, LLP

Subject: Performance Indicator Audit: Job Enrichment Opportunities (A-02-05-15119)

OBJECTIVE

The Government Performance and Results Act (GPRA) of 1993 requires the Social Security Administration (SSA) to develop performance indicators that assess the relevant service levels and outcomes of each program activity. GPRA also calls for a description of the means employed to verify and validate the measured values used to report on program performance.

Our audit was conducted in accordance with generally accepted government auditing standards for performance audits. For the performance indicator included in this audit, our objectives were to:

1. Assess the effectiveness of internal controls and test critical controls over the data generation, calculation, and reporting processes for the specific performance indicator.

2. Assess the overall reliability of the performance indicator's computer processed data. Data are reliable when they are complete, accurate, consistent and are not subject to inappropriate alteration.

3. Test the accuracy of results presented and disclosed in the Fiscal Year (FY) 2004 Performance and Accountability Report (PAR).

4. Assess if the performance indicator provides a meaningful measurement of the program it measures and the achievement of its stated objective.

BACKGROUND

We audited the following performance indicator as stated in the SSA FY 2004 PAR:

Performance Indicator
FY 2004 Goal
FY 2004 Reported Results

Number of Job
Enrichment Opportunities
(includes headquarters, component and regional development programs.)
3% of workforce
8.2% of workforce

SSA offers a variety of training and development opportunities to its employees. These opportunities include traditional classroom training on multiple subject areas, e-learning programs that allow employees to participate in self-paced training, and national career development and management training programs where employees gain experience and become prepared to reach higher levels of leadership.

This performance indicator is linked to the SSA's strategic objective to "Recruit, develop and retain a high-performing workforce," which is linked to the SSA strategic goal "To strategically manage and align Staff to support SSA's mission." This strategic goal is linked to one of the five governmentwide goals on the President's Management Agenda, "Strategic Management of Human Capital," which addresses all Government agencies' ability to sustain a learning environment that drives continuous improvement in performance. To assess its progress in meeting this objective, the Office of Human Resources (OHR) established a performance indicator to measure the percentage of the Agency workforce participating in job enrichment opportunities.

Job enrichment opportunities were initially defined as "formal national development programs, component level programs, or regional level development programs of 1-2 weeks in duration." In February 2004, the indicator was redefined to include any job enrichment opportunity of a 5-day duration or more. These opportunities consist of any training or development courses completed by the employee including formal details (i.e. short-term assignments), internal/external training courses, completion of individual development plans, temporary promotions, and any other opportunities that offer employee development experience and address potential skill gaps.

Regional Training Officers (RTO) and managers in the Deputy Commissioners' Offices capture and report employee participation from their offices to OHR on a monthly basis. OHR adds the employee participation cumulatively on a monthly basis throughout the FY. The total number of employees that participate in enrichment opportunities is then divided by the total workforce. For purposes of reporting FY 2004 performance, the Agency defined its workforce to include 64,800 people employed in permanent or temporary full-time and part-time positions. (For additional detail on the calculation of this indicator, refer to the flowchart in Appendix C.)

RESULTS OF REVIEW

We found SSA did not have adequate internal controls over the processes used to generate, calculate, and report the performance data used in the performance indicator, "Number of Job Enrichment Opportunities." As a result, we were unable to examine support for the reported indicator data or conclude on the accuracy of the calculation. Additionally, there were a number of inaccuracies in the disclosure and presentation of the indicator's results. Finally, the performance indicator simply measured an output, the number of enrichment opportunities, and did not provide a meaningful assessment of how the number of job enrichment opportunities contributed to the Agency achieving its strategic objective of developing a high-performance workforce.

Internal Controls

Although SSA had documented polices and procedures related to this performance indicator, we found that SSA had not documented and clearly communicated the definition of "job enrichment opportunities" to the RTOs and Deputy Commissioners' Offices' training managers. SSA applied a broad definition that included employee development programs and enrichment opportunities (e.g. formal details, internal/external training courses, Individual Development Plans, training cadres, temporary promotions and any other opportunities of at least 5-days duration that addressed skill gaps through employee development). As a result, the data was not reliable because the individuals responsible for reporting the data were unclear about the reporting criteria and inconsistently interpreted the data definition. For example, RTOs and Deputy Commissioners' Offices' training managers reported participation in job enrichment opportunities that were not at least 5 days in duration or included training programs that should not have been included.

SSA did not maintain an adequate audit trail for reported indicator data; lacked controls to periodically monitor the data accumulation process; and did not perform adequate management reviews to ensure the accuracy of the reported results. OHR relied on summary information supplied by the RTOs and Deputy Commissioners' Offices' training managers; OHR did not receive supporting documentation for the summary information. The cumulative number of employees that participated in job enrichment opportunities was reported monthly to OHR; however, the results were inconsistently reported, and actual employee names were not reported or maintained. As a result, the same employees may have been included in the numerator multiple times.

In addition, we found an immaterial discrepancy in the numerator between the source document used to calculate and report the results (5,317 employees) and the actual results (5,325 employees) reported. Furthermore, the basis for SSA's estimated total workforce of 64,800, used in the denominator, was not documented and clearly supported. We requested from SSA staff documentation that defined the steps taken and sources used to calculate the 64,800 workforce figure, but we were not provided with documentation that specifically showed how SSA concluded that its workforce was 64,800 at the time it calculated this performance indicator. In an effort to test the accuracy of the 64,800 figure, we calculated our own estimate of the total workforce. Based on PwC's review of SSA published records (at 10/1/03, 1/29/04, and 10/1/04), we estimated an average workforce of 64,505 employees.

Accuracy of PAR Presentation and Disclosure

As a result of the lack of a clear audit trail, as described above, we were unable to examine support for the reported indicator data or conclude that the performance indicator was calculated in accordance with the indicator definition and formula.

We found the Agency should not have compared FY 2004 results with FY 2003 results. The indicator was not consistently measured in FY 2004 since the Agency changed the definition of "job enrichment opportunity" in the middle of the FY. Prior to the change, job enrichment opportunity was defined to include formal national development programs or experiences (formal career development training of 1 to 2 weeks), a component program, or a regional-level program. In February 2004, SSA changed the definition to include any job enrichment opportunity of 5-day duration or more. The Agency did not restate past performance on the same basis as the previous year's definition resulting in a distorted annual performance trend.

The title of the performance indicator, "Number of Job Enrichment Opportunities," was inconsistent in the FY 2004 PAR as the Agency was collecting the actual number of job enrichment opportunities that occurred, but reporting the percentage of the workforce that participated in these opportunities. Additionally, the description of the indicator in the PAR is misleading. It states: "In FY 2004, 5,325 employees representing 8.2 percent of the workforce participated in a developmental or job enrichment experience." The indicator is based on the number of job enrichment opportunities, rather than the number of employees, and the calculation allows for employees to be counted multiple times if they attended more than one enrichment opportunity. For example, if one out of five employees took advantage of five separate job enrichment opportunities, five job enrichment opportunities occurred, however only 20 percent of the workforce was counted in the result of this indicator. Thus, fewer than 5,325 unique individuals participated in job enrichment opportunities, and the disclosure is better stated as "…in FY 2004, employees participated in 5,325 job enrichment opportunities."

Lastly, the presentation of the indicator performance graph in the PAR was inaccurate. The prior year result was 4.2 percent, but it was displayed in the graph as 3 percent.

Performance Indicator Meaningfulness

This performance indicator does not provide the reader with a strong correlation in either the effectiveness or efficiency in achieving the Agency's goal of developing a high-performing workforce. This indicator is intended to be a results-oriented output measure. It infers that by providing more job enrichment opportunities (outputs) that the Agency will favorably progress in realizing its human capital goals. However, the indicator broadly defines job enrichment opportunities without distinguishing the quality of the opportunities or the results obtained by offering them.

A more meaningful, outcome-oriented indicator could be achieved by:

Tracking employees that undertake job enrichment opportunities and their improvement in job performance, additional responsibilities assumed or promotions obtained. A results-oriented linkage can provide a stronger basis to support future funding of additional enrichment opportunities.

Conducting anonymous employee surveys that identify employee satisfaction with existing job enrichment opportunities within 2 to 4 months of participating in the opportunity. The survey results could be used to assess what the employees liked about the opportunity as well as identify areas for improvement. Survey results could initially be tracked internally, then as results improve, included as a reportable performance indicator.

Interviewing the managers of the employees who participated in the job enrichment opportunities to learn how these experiences have benefited the employees.

Interviewing high performing individuals that have accepted positions outside of the SSA to determine if they were given the opportunity to participate in job enrichment opportunities.

Comparing the employees who participate in the job enrichment opportunities with their years of service with SSA to determine if job enrichment opportunities assist in employee retention.

Additionally, the information disclosed in the PAR, could be enhanced by:

Disclosing the Agency's efforts and resources used to encourage employees to participate in job enrichment opportunities.

Describing the Agency's plans to improve the quality and quantity of development programs and training.

Disclosing the basis for defining the workforce used in the indicator's calculation.

Disclosing the participation in mandatory training programs versus discretionary training offered to employees.

CONCLUSION AND RECOMMENDATIONS

SSA should have adequate controls over the generation, calculation, and reporting of performance data to ensure a reliable measurement of performance. Equally important is the development of a performance indicator that meaningfully measures the outcome related to the job enrichment opportunities that SSA provides to its employees. We understand that SSA did not include this performance indicator in its FY 2006 Annual Performance Plan and FY 2005 Revised Annual Performance Plan. If SSA should decide to measure the benefits it gains from offering job development opportunities to its staff in the future, it should develop a more meaningful indicator with better controls over the process used to collect data and calculate the indicator. Accordingly, we recommend:

1. The performance indicator provide a more direct linkage to the Agency's strategic goals and objectives. The indicator should clearly support the Agency's strategic goal to "Recruit, develop and retain a high-performing workforce." Additionally, SSA should enhance the supporting information disclosed to describe the performance results.

2. The revised measure have effective controls implemented including:

A clearly documented audit trail that provides the information required to enable a review of the underlying data, in accordance with policies and procedures.

Written policies and procedures that describe the process for accumulating performance data and reporting results that provide relevant, logical definitions that can be consistently applied in determining the types of opportunities to be included and data to be collected.

Interim reviews of reported information to make adjustments, reinforce policies and procedures, clarify definitions, and more effectively manage resources.

Management reviews of the underlying data and calculation that support this indicator, and the accuracy of the presentation and disclosure in the PAR.

AGENCY COMMENTS

The Agency agreed with our two recommendations. In general comments, SSA stated the report mischaracterized the documentation related to the performance indicator's definition. While the report concluded that the documentation did not clearly communicate the definition of the performance indicator, SSA suggested that a more accurate statement would be that the documentation was not uniformly interpreted by the staff responsible for measuring job enrichment opportunities. See Appendix D for the full text of the Agency Comments.

PWC RESPONSE

We appreciate SSA comments and its agreement with our recommendations. In regards to its comment on the definition documentation, we believe that it is management's responsibility to clearly define the methods to be used to measure program performance. We do not believe that the documentation provided to us by the Agency met the applicable standards. Specifically, OMB A-123, Management's Responsibility for Internal Control, requires: "...well-defined documentation processes that contain an audit trail, verifiable results, and specify document retention periods so that someone not connected with the procedures can understand the assessment process."

Appendices
APPENDIX A - Acronyms
APPENDIX B - Scope and Methodology
APPENDIX C - Process Flowcharts
APPENDIX D - Agency Comments

Appendix A
Acronyms
DC Deputy Commissioner
DSA Data Source Analyst
DSC Data Source Contact
FY Fiscal Year
GAO Government Accountability Office
GPRA Government Performance and Results Act
OHR Office of Human Resources
OMB Office of Management and Budget
OSM Office of Strategic Management
PAR Performance and Accountability Report
Pub. L. No. Public Law Number
RTO Regional Training Officer
SSA Social Security Administration
U.S.C. United States Code

Appendix B
Scope and Methodology
We updated our understanding of the Social Security Administration's (SSA) Government Performance and Results Act (GPRA) processes. This was completed through research and inquiry of SSA management. We also requested SSA to provide various documents regarding the specific programs being measured as well as the specific measurement used to assess the effectiveness and efficiency of the related program.

Through inquiry, observation, and other substantive testing, including testing of source documentation, we performed the following:

Reviewed prior SSA, Government Accountability Office, Office of the Inspector General and other reports related to SSA's GPRA performance and related information systems.
Met with the appropriate SSA personnel to confirm our understanding of the performance indicator.
Flowcharted the process. (See Appendix C).
Tested key controls related to manual or basic computerized processes (e.g., spreadsheets, databases, etc.).
Conducted and evaluated tests of the automated and manual controls within and surrounding each of the critical applications to determine whether the tested controls were adequate to provide and maintain reliable data to be used when measuring the specific indicator.
Identified attributes, rules, and assumptions for each defined data element or source document.
Recalculated the metric or algorithm of key performance indicators to ensure mathematical accuracy.
For those indicators with results that SSA determined using computerized data, we assessed the completeness and accuracy of that data to determine the data's reliability as it pertains to the objectives of the audit.

As part of this audit, we documented our understanding, as conveyed to us by Agency personnel, of the alignment of the Agency's mission, goals, objectives, processes, and related performance indicators. We analyzed how these processes interacted with related processes within SSA and the existing measurement systems. Our understanding of the Agency's mission, goals, objectives, and processes were used to determine if the performance indicators appear to be valid and appropriate given our understanding of SSA's mission, goals, objectives and processes.

We followed all performance audit standards in accordance with generally accepted government auditing standards.

Appendix C
Flowchart of Number of Job Enrichment Opportunities - 2004 Process Flowchart

Office of Human Resources (OHR) Data Source Analyst transmits electronic mail request to Regional Training Officers (RTO) and Deputy Commissioners (DC) level training contacts for job enrichment participation updates.
The RTOs and DC level training contacts send their updates to the OHR Data Source Analyst.
OHR Data Source Analyst compiles information received from RTOs and DC level training contacts into two reports:
a. Employee Development Programs
b. Enrichment Opportunities
OHR Data Source Analyst transmits the two reports to the Data Source Contact.
Data Source Contact reconciles the two reports with the previous month's reports. If reported data is inconsistent with prior months reported data, Data Source Contact consults with Data Source Analyst to reconcile discrepancies.
Data Source Contact calculates fiscal year to date percentage by dividing the number of participates by the defined workforce of 64,800 people.
By the 8th of each month, the Data Source Contact submits the indicator performance to OSM and relevant OHR staff.
Flowchart of Number of Job Enrichment Opportunities - 2004 Process Flowchart Narrative

Appendix D
Agency Comments

SOCIAL SECURITY

MEMORANDUM

Date: July 21, 2005

To: Patrick P. O'Carroll, Jr.
Inspector General

From: Larry W. Dye
Chief of Staff

Subject: Office of the Inspector General (OIG) Draft Report, "Performance Indicator Audit: Job Enrichment Opportunities" (A-02-05-15119) -- INFORMATION

We appreciate OIG's efforts in conducting this review. Our comments on the draft report content and recommendations are attached.

Let me know if we can be of further assistance. Staff inquiries may be directed to Candace Skurnik, Director, Audit Management and Liaison Staff on extension 54636.

SSA Response

COMMENTS ON THE OFFICE OF THE INSPECTOR GENERAL (OIG) DRAFT REPORT "PERFORMANCE INDICATOR AUDIT: JOB ENRICHMENT OPPORTUNITIES" (A-02-05-15119)

Thank you for the opportunity to review and comment on the draft report. We agree with the findings pertaining to the accuracy of the information presented in the Performance and Accountability Report and the performance indicator meaningfulness. The Agency has been proactive in establishing performance indicators that are more meaningful, which is why this particular indicator was dropped as an Agency performance measure for fiscal year (FY) 2005 and replaced with a new measure. As the report acknowledges, this performance indicator is not in the Agency's FY 2005/2006 Annual Performance Plan and there are no plans to include this indicator in future performance plans.

However, we do not agree with the audit's characterization of the "internal controls" that were in place for this indicator. Although the first sentence acknowledges that the Agency had documented policies and procedures, it also states that we had not "documented" and "clearly communicated" the definition of the indicator. We believe this is not an accurate statement and implies that documentation did not exist. Documentation of the indicator's definition can be found in the data source sheets and in the February 2004 email communication that was provided to the Regional Training Offices (RTOs) and the Deputy Commissioners' Offices' training managers.

Consequently, we recommend that the first sentence be changed to read: "Although SSA had documented policies and procedures related to this performance indicator, we found that the definition of "job enrichment opportunities" was not uniformly interpreted by the RTOs and Deputy Commissioners' offices' training managers."

Our specific responses to the report's recommendations are provided below.

Recommendation 1

The performance indicator should provide a more direct linkage to the Agency's strategic goals and objectives. The indicator should clearly support the Agency's strategic goal to "Recruit, develop and retain a high-performing workforce." Additionally, SSA should enhance the supporting information disclosed to describe the performance results.

Response

We agree. If the Agency does decide to revisit measuring/tracking the benefits of employee training and development experiences as Agency strategic objectives, we will implement the recommendations and corrective actions described in the audit report.

Recommendation 2

The revised measure should have effective controls implemented including: a clearly documented audit trail that provides the information required to enable a review of the underlying data, in accordance with policies and procedures; written policies and procedures that describe the process for accumulating performance data and reporting results that provide relevant, logical definitions that can be consistently applied in determining the types of opportunities to be included and data to be collected; interim reviews of reported information to make adjustments, reinforce policies and procedures, clarify definitions, and more effectively manage resources; management reviews of the underlying data and calculation that support this indicator, and the accuracy of the presentation and disclosure in the PAR.

Response

Overview of the Office of the Inspector General

The Office of the Inspector General (OIG) is comprised of our Office of Investigations (OI), Office of Audit (OA), Office of the Chief Counsel to the Inspector General (OCCIG), and Office of Executive Operations (OEO). To ensure compliance with policies and procedures, internal controls, and professional standards, we also have a comprehensive Professional Responsibility and Quality Assurance program.

Office of Audit

OA conducts and/or supervises financial and performance audits of the Social Security Administration's (SSA) programs and operations and makes recommendations to ensure program objectives are achieved effectively and efficiently. Financial audits assess whether SSA's financial statements fairly present SSA's financial position, results of operations, and cash flow. Performance audits review the economy, efficiency, and effectiveness of SSA's programs and operations. OA also conducts short-term management and program evaluations and projects on issues of concern to SSA, Congress, and the general public.

Office of Investigations

OI conducts and coordinates investigative activity related to fraud, waste, abuse, and mismanagement in SSA programs and operations. This includes wrongdoing by applicants, beneficiaries, contractors, third parties, or SSA employees performing their official duties. This office serves as OIG liaison to the Department of Justice on all matters relating to the investigations of SSA programs and personnel. OI also conducts joint investigations with other Federal, State, and local law enforcement agencies.

Office of the Chief Counsel to the Inspector General

OCCIG provides independent legal advice and counsel to the IG on various matters, including statutes, regulations, legislation, and policy directives. OCCIG also advises the IG on investigative procedures and techniques, as well as on legal implications and conclusions to be drawn from audit and investigative material. Finally, OCCIG administers the Civil Monetary Penalty program.

Office of Executive Operations

OEO supports OIG by providing information resource management and systems security. OEO also coordinates OIG's budget, procurement, telecommunications, facilities, and human resources. In addition, OEO is the focal point for OIG's strategic planning function and the development and implementation of performance measures required by the Government Performance and Results Act of 1993.