Office
of the Inspector General
Jo Anne B.
Barnhart
Commissioner
Inspector
General
Performance
Measure Review: Reliability of the Data Used to Measure Anti-Fraud Performance
(A-02-01-11013)
We recently
completed our first 3-year cycle of reviewing the Social Security Administration's
performance measures. In conducting this work, we used the services of an outside
contractor, PricewaterhouseCoopers, LLP, (PwC) to assist us in our efforts.
For this
report, we used PwC to conduct the review of four of the Agency’s performance
indicators related to anti-fraud activities. The objective of the review was
to assess the reliability of the data used to measure the Agency’s anti-fraud
efforts. These anti-fraud indicators measure the results of the work conducted
by the Office of Investigations within the OIG. To ensure an independent review
of these indicators, we relied on our contractor to assess the reliability of
the data behind these measures.
The attached
final report presents the results of the contractor’s work and its recommendations
for improvement. We are generally in agreement with all of the contractor’s
recommendations focusing on OIG operations. My office will track the individual
recommendations and we will periodically provide our status in implementing
these recommendations. We plan to work closely with the Agency on those recommendations
where coordination would be appropriate. If you wish to discuss the final report,
please call me or have your staff contact Steven L. Schaeffer, Assistant Inspector
General for Audit, at (410) 965-9700.
James
G. Huse, Jr.
OFFICE
OF
THE
INSPECTOR GENERAL
SOCIAL
SECURITY ADMINISTRATION
PERFORMANCE
MEASURE REVIEW:
RELIABILITY
OF THE DATA USED TO
MEASURE
ANTI-FRAUD PERFORMANCE
January
2002
A-02-01-11013
EVALUATION
REPORT
Evaluation of Selected Performance Measures of the Social Security Administration:
Reliability
of the Data Used to Measure
Anti-Fraud
Performance
Office
of the Inspector General
Social Security Administration
INTRODUCTION
This report
is one of five separate stand-alone reports, corresponding to the following
Social Security Administration (SSA) process and performance measures (PM):
Number
of investigations conducted (i.e., closed) (PM #5)
Fiscal
Year (FY) 2000 Goal: 7,600
Old-Age,
Survivors and Disability Insurance (OASDI) dollar amounts reported from investigative
activities (PM #6)
FY 2000
Goal: $40 million
Supplemental
Security Income (SSI) dollar amounts reported from investigative activities
(PM #7)
FY 2000
Goal: $80 million
Number
of criminal convictions (PM # 8)
FY 2000
Goal: 1,800
This report
reflects our understanding and evaluation of the process related to PMs #5 through
#8. To achieve its strategic goal "To make SSA program management the best-in-business,
with zero tolerance for fraud and abuse," SSA has developed several strategic
objectives. One of these objectives is "To aggressively deter, identify,
and resolve fraud." SSA’s FY 2001 Annual Performance Plan (APP) contains
four performance indicators developed to meet this objective as follows:
Number
of investigations conducted (i.e., closed) - Prior to FY 2000, this goal
was based on cases opened. The FY 2000 goal was revised upward from 7,200
to 7,600 investigations conducted (i.e., closed) in SSA’s FY 2000 Revised
Final Performance Plan; this reflects the benefits of increased resources
the Office of the Inspector General (OIG) devoted to investigative activities.
OASDI
dollar amounts reported from investigative activities - The FY 2000 goal
was revised upward from $9 million to $40 million in SSA’s FY 2000 Revised
Final Performance Plan; this reflects an anticipated return on investment
from investigative activities as described above.
SSI
dollar amounts reported from investigative activities - The FY 2000 goal
was revised upward from $55 million to $80 million in SSA’s FY 2000 Revised
Final Performance Plan to reflect an anticipated return on investment from
investigative activities as described above.
Number
of criminal convictions - The FY 2000 goal was 1,800 convictions.
We performed
our testing from September 21, 2000 through February 15, 2001. Our engagement
was limited to testing at SSA’s headquarters in Woodlawn, Maryland and the OIG
office in Philadelphia, Pennsylvania. The procedures that we performed were
in accordance with the American Institute of Certified Public Accountants’ Statement
on Standards for Consulting Services, and are consistent withappropriate
standards for performance audit engagements in Government Auditing Standards
(Yellow Book, 1994 version). However, we were not engaged to and did not conduct
an audit, the objective of which would be the expression of an opinion on the
reliability or accuracy of the reported results of the performance measures
evaluated. Accordingly, we do not express such an opinion.
BACKGROUND
SSA has
been engaged in an aggressive program to deter, detect, investigate and prosecute
fraud. To carry out this effort, SSA and the OIG have cooperated in developing
a comprehensive anti-fraud plan. The SSA OIG, Office of Investigations (OI)
is responsible for investigating allegations of fraud, waste and abuse. The
four indicators evaluated focus on the OIG’s output efforts to achieve improvements
in deterring, identifying and resolving fraud.
The first
performance measure, "Number of investigations conducted," represents
the number of investigations "conducted" by OI resulting from allegations
that have sufficient information or potential risk to warrant further review
or action by a criminal investigator. Allegations may be received from Congress,
SSA employees, or from the public by mail, e-mail, fax, Internet, or telephone
(i.e., 800 Hotline). Investigations are counted as "conducted" when
all OI actions have been completed or the investigator has determined that further
action is not warranted due to lack of investigative leads. The objective is
to raise the number of investigations conducted (i.e., closed), which relates
to the strategic goal to aggressively deter, identify and resolve fraud. This
performance measure is presented as a workload count, and includes every closed
case, including cases for which it was determined that no fraud was involved.
The second
and third performance measures, "OASDI dollar amounts reported from investigate
activities" and "SSI dollar amounts reported from investigative activities,"
represent amounts reported by the OI from fines/penalties, assessments, savings,
recoveries and restitution/judgments related to investigative activities. The
components of investigative activities, or "monetary achievement"
are defined as follows:
Fines/penalties
are court-ordered, and include any special assessment fees, imposed upon conviction
in a criminal case or judgment in a civil case, which require a specified
sum of money be paid to the court.
Program
savings is a calculation of the avoidance of actual dollar loss by actions
that result in the termination of improper payments of program funds, which
only can relate to SSA cases.
Restitution
can only be recorded in a criminal case, in which the court-ordered repayment
that resulted from pretrial diversions and convictions. The amount of restitution
may be categorized as SSA or non-SSA program amounts. In addition, the amount
of the restitution claimed by OI should match the amount of restitution documented
on the Judgment and Commitment Order (J&C) or Pretrial Diversion agreement.
A judgment
is a judicially ordered payment resulting from a civil action, either through
Department of Justice civil proceedings or the Civil Monetary Penalty Program,
which can be categorized as SSA or non-SSA program related. The distinguishing
characteristic of a judgment is its nexus to a civil action, as opposed to
a criminal restitution.
A scheduled
recovery is a total sum of non-court ordered repayment of funds to which an
individual was not entitled, or the total of funds to be returned because
the individual was not entitled, which can be categorized as SSA or non-SSA
program related. In those SSA program cases in which the court does not order
restitution because of a defendant’s inability to pay, OI will provide a copy
of the J&C to SSA, accompanied by a memorandum requesting SSA to recover
the program losses. SSA’s initiation of withholding future payments will constitute
a scheduled recovery.
The objective
is to report the dollar values of the OI efforts/activities to identify and
resolve fraud, which is directly related to the strategic goal "To aggressively
deter, identify and resolve fraud."
The last
performance measure, "Number of Criminal Convictions," represents
the number of criminal convictions as a result of OI activities. This performance
measure is presented as a workload count of all cases concluding in a criminal
conviction. In addition to purposes served by formally charging a person with
the commission of a crime, the criminal prosecution process has an impact, which
may deter others from committing violations, and therefore is directly related
to the strategic goal "To aggressively deter, identify and resolve fraud."
Allegation
and Case Investigative System (ACIS)
Information
concerning potential wrongdoing involving SSA programs, employees, or operations
which are received by an SSA OIG component are accounted for in ACIS. The system
encompasses the initial receipt of an allegation, all steps taken throughout
the investigative process, and the final outcome of all investigations. The
system resides in a database (ADABAS) application, at the Center for Information
Technology (CIT) of the National Institutes of Health (NIH).
The ACIS
process begins with the receipt of an allegation. The Allegation Management
Division (AMD) staff records these allegations in the allegation management
module of ACIS. They are then electronically forwarded to either an OI field
division (FD), a SSA field component, another agency, or are closed. Allegations
may be received from Congress or from the public via mail, e-mail, fax, Internet,
or telephone (i.e., 800 Hotline). FDs may also receive allegations directly,
and are also required to enter the allegation information into ACIS. If an investigation
is warranted, all information and supporting documentation will be forwarded
to the appropriate individuals within the FD.
When an
investigation is opened, ACIS generates an OI-1, ACIS Case Opening Report. To
complete and document an investigation, the Special Agent (SA) is required to
complete a series of forms. Once the case is taken to court, if the court orders
a restitution or judgment, the agent completes the OI-68, Report of Court Ordered
Restitution/Judgment. The OI-67, Monetary Achievement Worksheet, may also accompany
an OI-68. The OI-67 is the form used to document any civil monies recovered
during the investigation, as well as to assist in calculating the monetary achievement
documented on the OI-68. Once all aspects of the investigation are completed
the SA completes an OI-9, ACIS Criminal and Administrative Disposition Form.
This form includes the judicial and criminal disposition data, as well as criminal
and administrative monetary achievements, and is used by the administrative
personnel, SAs, Assistant Special Agents-in-Charge (ASAC) or Special Agents-in-Charge
(SAC) to input the monetary achievements into ACIS.
The OI-68
is mailed to OI Manpower and Administration Division (MAD) in Woodlawn, Maryland.
It is then reviewed against the data, which is entered into ACIS for accuracy
and distributed to SSA’s Debt Management Section for notification of the court-ordered
restitution. In addition, the OI-68 and the payments are sent to the Mid-Atlantic
Program Service Center (MATPSC) to be processed. The MATPSC cross-references
the checks to the OI-68 and posts them to the correct Social Security numbers
(SSN).
RESULTS
OF EVALUATION
During the
period of September 21, 2000 to February 15, 2001, we evaluated the current
processes, systems and controls, which support the FY 2000 SSA performance measurement
process. In addition, we determined the accuracy of the underlying performance
measure data. Our evaluation of the information provided by SSA management as
of February 15, 2001, allowed us to determine that the reported FY 2000 results
of the four performance measures tested (as itemized below) were reasonably
stated based on the methodology used by SSA.
Performance
Measure
Reported
Result
Number
of investigations conducted (i.e., closed)
8,051
OASDI
dollar amounts reported from investigative activities
$46
million
SSI
dollar amounts reported from investigative activities
$128
million
Number
of criminal convictions
2,604
However,
we noted certain deficiencies in SSA’s methodology used to analyze the performance
measures and certain limitations with ACIS. Although we do not consider any
of the deficiencies noted during our evaluation to be significant, consideration
should be given to the following recommendations, as we believe there are opportunities
for improvement, thereby increasing the value of the performance measures as
management tools.
ACIS
has some data integrity deficiencies.
The savings
calculation is not adequately supported.
SSA OIG
lacks sufficient documentation for several key control areas related to ACIS.
These areas are change control policies and procedures, user access recertification
procedures and systems development documentation.
SSA OIG
lacks appropriate supervisory review in the change control process.
OASDI
and SSI dollar amounts from investigations may be over or understated.
These items
were noted as a result of our testing the underlying performance measure data,
as well as the Electronic Data Processing (EDP) and manual controls of the systems
generating the performance measure data, and are discussed in detail below.
1. ACIS
has some data integrity deficiencies.
To ensure
that the amounts reported in the SSA Government Performance and Results Act
of 1993 (GPRA) section of the Performance and Accountability Report were reasonably
accurate and reliable, we obtained a random sample of 45 OASDI, 45 SSI and 20
criminal conviction cases from the ACIS system and requested supporting documentation
for each of the cases. In addition, we judgmentally selected 10 cases from the
closed-case cabinet at the Philadelphia sub-office, traced them to ACIS and
ensured the cases were properly documented.
The results
of our evaluation of the cases are as follows:
Overstatement
of OASDI Monies from Investigative Activities
We noted
that 5 of the 45 OASDI cases evaluated had double-counting errors totaling $235,911
for OASDI amounts reported from investigative activities. This resulted from
dollar amounts in both the scheduled recovery and restitution fields in ACIS.
Per the Special Agent Handbook (SAH), these two fields are mutually exclusive,
and therefore a case should not have the same dollars entered in both fields.
These two fields are then combined with savings, judgments, settlements, fines,
assessments, and penalties to arrive at the OASDI monies from investigative
activities.
Policy
Errors
In our evaluation
of the 120 ACIS cases selected, we found 2 cases in which the procedures used
to calculate savings were not consistent with those procedures outlined in the
SAH. The first was a fugitive felon case, for which the amount of savings was
calculated over 3 years. Per the SAH, Chapter 3, Section 3-75-C8 "Program
savings for fugitive felon cases will be calculated on a 2-year projection."
In the second case OI found an individual embezzling benefit checks from an
entitled beneficiary. A savings amount was calculated for this case, even though
in a similar case, savings were not calculated.
Data
Anomalies
Our evaluation
of the cases also noted that 36 out of the 120 ACIS cases selected contained
data anomalies. The data anomalies represent instances where ACIS data did not
match the data reported in the supporting documentation (OI-68, OI-9, OI-4,
or OI-1). For example, in several cases the fraud loss entered into ACIS was
higher than the fraud loss reported in the supporting documentation. In some
cases, the recovery or assessment included in the supporting documentation was
not reported in ACIS. These errors could have been caused by either 1) a data
entry error made by administrative personnel, SACs, ASACs or SAs, or 2) by documentation
errors within the case file, made by SAs.
OI has in
place a supervisory review of case files at the field division level. However,
the supervisory review, Form OI-20, includes a review of the status of the case,
not a review of the forms within the case file to ensure that the forms are
completed appropriately. The SAH, Chapter 3, Section 3-60 B, states the following:
"While the review of documents and evidence in the case file is important,
the case review process should not be viewed as merely a review of documents.
Rather it is an evaluation of the investigative progress and potential of the
SA’s cases." In addition, the SAH does not include procedures to ensure
that the data entered in ACIS by the OI field divisions is correct.
Although
the OI MAD has a Quality Review process, this process is limited to a comparison
of Form OI-68 "Report of Court-Ordered Restitution/Judgment" to the
restitution report produced at MAD, and the restitution amounts reported in
ACIS. In addition, since not all cases require an OI-68, not all cases are reviewed.
The lack of supervisory review of the forms and limited review by OI Headquarters
has led to errors going undetected, thus diminishing the accuracy of the number
reported by OI as part of the GPRA section of SSA’s Performance and Accountability
Report.
ACIS is
an antiquated system that lacks sufficient capabilities to continue to meet
OIG’s needs. The OIG stated that the system has been "frozen" and
that it is in the process of evaluating commercial off the shelf software packages
as an alternative system. These data integrity opportunities for improvement
should be considered when selecting and implementing the new system.
Savings
calculation for disability cases is not adequately supported.
A program saving is an integral part of the dollars reported from the investigative
activities calculation. For disability cases, the cooperative disability investigation
(CDI) teams claim a set amount of savings of $66,500 per case, for applicants
who are denied eligibility due to the findings of the investigation. SSA arrived
at this figure in 1997 by applying the amount of an average lifetime benefit
figure to an average Disability Determination Services claim that is denied.
This amount has not been adjusted since 1997 to reflect the yearly increase
of SSI benefit payments. OI, in conjunction with SSA’s Office of the Actuary,
is currently developing a method to calculate the savings figure for each case
based on a claimant’s age, gender, life expectancy and amount of benefit received.
The new method will be applied to all CDI cases beginning in FY 2002.
SSA OIG
lacks sufficient documentation for some control areas related to ACIS.
There is a lack of documentation for some ACIS processes in SSA OIG operations,
as follows:
ACIS
Change Control: Process in which any modification requests to ACIS, and
subsequent modifications of ACIS are formally requested, documented, tested,
and implemented.
ACIS
User Access Recertification: Review of users’ access rights to ACIS, to
ensure that their access matches their job responsibilities.
ACIS
System Design and Development: System design and development documentation
includes addressing the entire life cycle of a system, (i.e., initiation,
definition, system design, programming and training, evaluation and acceptance,
and installation and operations). As such it would include, for example, the
functional requirements, data requirements, system/sub-system requirements,
and both user and technical manuals, etc.
Without
formally documented processes, management cannot be assured that SSA OIG personnel
understand all of the requirements for successful change control, user access
recertifications, and systems design and development processes.
When processes
are not documented, such as formal processes for change control, user access
recertification, and ACIS system design and development, there is no accountability
that the procedures were followed. When there is no accountability for the procedures
associated with these processes, it becomes difficult to resolve any problems
or issues. This lack of assurance may negatively impact SSA OI operations.
SSA OIG
lacks appropriate supervisory review in the change control process.
Both of the OIG personnel that are responsible for programming ACIS changes
have the ability to promote program changes to production. This increases the
risk that inappropriate or incorrect changes could be placed into production,
thereby compromising the functionality of ACIS.
Office of
Management and Budget (OMB) Circular A-130, Appendix III, Security of Federal
Automated Information Resources, states that agencies are required to establish
controls to assure adequate security for all information processed, transmitted,
or stored in Federal automated information systems. This appendix stresses the
importance of management controls affecting individual users of information
technology. The appendix states: "Technical and operational controls support
management controls. To be effective, all must interrelate."
In addition,
Federal Information Processing Standards (FIPS) Publication 73 entitled Guidelines
for Security of Computer Applications, Section 6.3.4 highlights the increased
risk associated with programmers having access to program code once an application
is operational. This increases the possibility for unauthorized changes to existing
code that could benefit the programmer without being detected.
OASDI
and SSI dollar amounts from investigations may be over or understated
The program
category field within ACIS is used to identify which program (OASDI or SSI)
the fraud was committed against. The possibility exists that an investigation
may involve fraud against both OASDI and SSI concurrently. While agents have
the ability to enter in more than one program category per case into ACIS, the
associated Monetary Statistics report only pulls the information from the first
program category in each case. Therefore, in the case of concurrent fraud, agents
are instructed to enter the dollar amounts into the program category where the
greater amount of fraud occurred. This results in an overstatement of dollar
amounts in the program category that the case was entered into, and an understatement
of dollar amounts in the other program category.
CONCLUSIONS
AND RECOMMENDATIONS
Throughout
our evaluation of the four performance measures, we noted the strong commitment
of SSA's OIG staff to correctly implement GPRA. Our evaluation found that the
FY 2000 results of the four performance measures tested were reasonably stated.
However, our evaluation noted that: 1) ACIS has some data integrity deficiencies;
2) the savings calculation is not adequately supported; 3) SSA OIG lacks sufficient
documentation for several key ACIS control areas; 4) SSA OIG lacks appropriate
supervisory review in the change control process; and 5) OASDI and SSI dollar
amounts from investigations may be over or understated. We recommend that the
OIG take the following corrective actions:
ACIS has
a number of data integrity deficiencies.
To ensure
that the OI offices across the country are following the procedures outlined
in the SAH, that data entry and documentation errors do not go undetected, and
to correct internal control issues found during our case testing we recommend
that OI:
Expand
its supervisory review of the cases, by including a comprehensive review of
the contents of all forms included in the case, to ensure the accuracy and
appropriateness of the information.
Perform
a FD-level review of the information entered into ACIS either by the SAC,
in cases where an SA or administrative staff has input the information into
ACIS, or by a supervisor in cases were the SAC has input the information into
ACIS.
Expand
the Quality Assurance review performed at MAD to include all cases. The Quality
Assurance process should include a review of the OI-9 "ACIS Criminal
and Administrative Disposition Form" for all cases and the supporting
documentation. The review should be documented to enable OI to identify systematic
or widespread problems throughout the FDs before they cause substantial errors
in the information reported
These
procedures will prevent erroneous data being entered into ACIS in the future,
and therefore prevent inaccurate GPRA reporting.
The savings
calculation is not adequately supported.
We recommend
that OI continue its work with SSA and the Office of the Actuary to develop
and update the savings calculation for CDI cases.
SSA OIG
lacks sufficient documentation for several key ACIS control areas.
To ensure
that the SSA OIG has proper controls in place for change control, user access
recertification, and system design and development, we recommend that the
Office of Executive Operations (OEO):
Formalize
and then document the change control process. This includes the creation of
a standardized change control form, incorporating a tracking number, the reason
for the request, testing, sign-offs and promotion of the program into production.
Create
and document the user access recertification process. This should be an annual
process, which will ensure all users have access commensurate with their position
through verification by user management.
Document
all system design and development information. This documentation should provide
both the systems management and OEO with the rationale for the design, as
well as its functionality and data structure. Documentation is especially
useful during the implementation of new systems.
SSA OIG
lacks of appropriate supervisory review in the change control process.
SSA OIG
should ensure proper authorization exists prior to ACIS program changes being
promoted to production.
In addition,
the SSA OIG should use a computer operator, or other non-programmer for the
actual movement of programs into production.
OASDI
and SSI dollar amounts from investigations may be over or understated.
To report
accurate amounts for the OASDI and SSI dollars associated with investigative
activities, we recommend that the OIG reconsider the use of the second program
category and the design of the Monetary Statistics report. Agents should be
instructed to break the dollar amounts down by program category, and record
multiple entries for the same subject. This should not affect the number of
investigations or cases and should produce more accurate OASDI and SSI dollar
amounts.
APPROPRIATENESS
OF THE PERFORMANCE MEASURES
As part
of this engagement, we evaluated the appropriateness of each of the performance
measures with respect to GPRA compliance and SSA’s APP. We determined whether
the specific indicators and goals corresponded to the strategic goals identified
in SSA’s APP, determined whether each of these indicators accurately measure
performance, and determinedtheir compliance with GPRA requirements.
The relationships
between PMs #5 through 8 and the applicable SSA Strategic Goal is depicted
in the following figure:
The SSA
mission is supported by five strategic goals, including Goal 3, "To make
SSA program management the best-in-business, with zero tolerance for fraud
and abuse." Goal 3, in turn, is supported by several strategic objectives,
including the relevant objective "to aggressively deter, identify, and
resolve fraud." Performance Measures #5 through #8 address the OIG’s
OI work related to SSA programmatic fraud. Assuming that the metrics are reliable,
the diagram indicates that PMs #5 - #8 logically align with SSA’s strategic
planning process.
Based
on the taxonomy of performance measures included in Appendix F, PMs #5 through
#8 are measures of accomplishment because they report on a result achieved
with SSA resources. They are further categorized as output measures because
they indicate the accomplishments or results that occur because of the SSA
services provided. As shown in Appendix F, output measures include the number
of investigations conducted.
Within
the framework of GPRA, Performance Measures #5 through #8 fall within the
intent of an output measure because they provide, "…a description of
the level of activity or effort that will be produced." Just as with
counts of workload, dollar amounts of workload are also a "level of activity
required" or "workload" measurement. In addition, one output
can be the workload driver for another output (e.g., the number of investigations
conducted is a factor that drives the number of criminal convictions). Therefore,
all four measures are considered output measures. The intent of these four
performance measurements is to address the "zero tolerance for fraud"
strategic goal. QAThey can all be useful to management and external stakeholders,
as encouraged by OMB Circular A-11, which provides guidance on the creation
of an agency’s performance measures. However, if corrective actions are not
implemented to correct the issues identified as part of this report, potential
errors in the data may in the future produce unreliable results.
OTHER
MATTERS
As part
of this evaluation, we identified several points that the OIG should consider
when designing the new AMS, as well as other less significant matters that
are peripheral to this engagement. The points are discussed below.
There
is a need for Service Level Agreements.
In the
current organizational structure of the SSA OIG, the personnel responsible
for supporting ACIS are part of the OIG’s OEO. The personnel who use ACIS
for their day to day job functions reside in the OI. During the course of
our evaluation, we discovered that there was not any formal agreement between
the OEO and the OI, which details the expectations and responsibilities of
both offices with respect to ACIS and its support.
Control
Objectives for Information and related Technology (COBIT) developed as generally
applicable and accepted standard for good Information Technology (IT) security
and control practices, states the following:
"Users
and the IT function should have a written agreement which describes the service
level in qualitative and quantitative terms. The agreement defines the responsibilities
of both parties. The IT function must offer the agreed quality and quantity
of service and the users must constrain the demands they place upon the service
within the agreed limits."
The lack
of such an agreement can lead to miscommunication and unfulfilled expectations,
both of which could hamper the ability of affected OIG staff to perform their
job functions effectively and efficiently. An agreement between both sides,
such as a Service Level Agreement (SLA), could detail each office’s expectations
and associated job duties, and provide accountability for their performance.
Recommendation
To ensure
continued effective communication between OEO and OI, we recommend SSA OIG
draft a SLA between these offices, detailing the responsibilities of both
offices, as well as each office’s expectations. The cognizant personnel in
both offices should sign this agreement.
There
is no correlation between dollars reported from investigations and dollars
collected by SSA.
Both OASDI
and SSI dollars reported from investigative activities are recorded into ACIS.
These figures are collected by case number, and the SSN upon which the fraud
was committed. However, it does not reference the SSN of the individual who
committed the fraud. ACIS does not provide for accounting of the dollars actually
collected and the tracking ceases with the closing of a case.
The accounting
for the dollars, which are actually collected by SSA, is performed by SSA’s
Debt Management System (DMS). The DMS contains information on the SSN to which
the payment is posted. This may not always be the SSN that the fraud was committed
against, but may be the SSN of the individual making the payment. Because
the SSNs cannot always be cross referenced, neither the OIG nor DMS can categorize
dollars collected as dollars associated with OIG cases.
Recommendation
We recommend
SSA’s DMS include an OIG case number with its SSN information, if applicable,
or that ACIS include the SSN of the individual committing the fraud. This
will then give both groups the ability to cross reference these payments,
and accumulate not only the dollars reported from investigative activities,
but also the dollars that were collected.
The title
"Criminal Convictions" for PM #8 is misleading.
The category
"Criminal Convictions" contains several types of case resolutions.
In addition to criminal convictions, the category also includes work with
immigration and deportation cases, the satisfaction of a fugitive warrant,
and civil judgments. While there is a footnote explaining that the category
contains more than criminal convictions, it does not specifically state what
is included in the total.
Based on our recommendation, OI has changed the name of this performance measure
in the draft SSA’s FY 2003 APP to "Number of Judicial Actions."
In addition, SSA’s FY 2003 APP includes the following definition for Judicial
Actions: A judicial action is any event during the criminal justice process
that causes an individual suspected of committing a crime to be arrested for
the crime, or to appear before a judge to enter a plea of guilty, or to face
trial before a judge or jury.
There
is a lack of disaster recovery documentation.
ACIS,
used by the SSA OIG to maintain its investigation information, resides at
the NIH CIT. As such, the SSA OIG is considered to be a user bureau by NIH,
and is occasionally asked by NIH to participate in its annual disaster recovery
tests.
The ACIS
systems administrator also performs the duties of a security coordinator at
NIH. As such, NIH notifies him when a disaster occurs and is at that time
given his instructions. However, a formal copy of the NIH disaster recovery
plan is not kept by the SSA OIG.
Recommendation
To ensure
that the SSA OIG has an understanding of the NIH disaster recovery plan and
its associated responsibilities, we recommend that the SSA OIG request a copy
of the NIH disaster recovery plan, and that the plan be reviewed and updated
for specific OIG-related matters on an annual basis.
APPENDICES
APPENDIX
A – Scope and Methodology
APPENDIX
B – Acronyms
APPENDIX
C – Performance Measure Summary Sheets
APPENDIX
D – Performance Measure Process Maps
APPENDIX
E – Performance Measure Taxonomy
Scope
and Methodology
The Social
Security Administration’s (SSA) Office of the Inspector General (OIG) contracted
PricewaterhouseCoopers, LLP (PwC) to evaluate 11 SSA performance indicators
identified in its Fiscal Year (FY) 2001 Annual Performance Plan (APP). This
report reflects our understanding and evaluation of the process related to PMs
#5 through #8. We performed our testing from September 21, 2000 through February
15, 2001. Since FY 2001 performance results were not yet available as of the
date of our evaluation, we performed tests of the performance data and related
internal controls surrounding the maintenance and reporting of the results for
FY 2000. Specifically, we performed the following:
Obtained
an understanding and reviewed the current Allegation and Case Investigative
System (ACIS) data flows and processes;
Identified
and tested critical controls (both electronic data processing (EDP) and manual)
of ACIS;
Tested
the accuracy of the underlying FY 2000 data for each of the specified performance
measures;
Recalculated
each specific FY 2000 measure to ascertain its mathematical accuracy;
Determined
whether performance measures were meaningful and in compliance with the Government
Performance and Results Act of 1993 (GPRA);
Evaluated
the impact of any relevant findings from prior and current audits with respect
to SSA's ability to meet performance measure objectives; and
Identified
findings relative to the above procedures and provided recommendations for
improvement.
Our engagement
was limited to testing at SSA’s headquarters in Woodlawn, Maryland and the OIG
office in Philadelphia, Pennsylvania. The procedures that we performed were
in accordance with the American Institute of Certified Public Accountants’ Statement
on Standards for Consulting Services, and are consistent withappropriate
standards for performance audit engagements in Government Auditing Standards
(Yellow Book, 1994 version). However, we were not engaged to and did not conduct
an audit, the objective of which would be the expression of an opinion on the
reliability or accuracy of the reported results of the performance measures
evaluated. Accordingly, we do not express such an opinion.
Obtained
an understanding and reviewed the current ACIS data flows and processes.
We obtained
an understanding of the underlying processes and operating procedures surrounding
ACIS and the generation of performance measures through interviews and meetings
with the appropriate SSA OIG personnel and by reviewing the following documentation:
Policies
and procedures manual for procedures surrounding the processing, accumulating,
and reporting of the data for the four performance measures;
PwC system
walk-through descriptions;
PwC system
flowcharts;
SSA OIG-provided
system descriptions, including the ACIS User Manual and the Special Agent
Handbook;
FY 2000
ACIS data and corresponding data definitions;
The National
Institutes of Health Statement on Auditing Standards (SAS) 70 Reports for
FY 1999 and FY 2000, describing the general controls surrounding the ACIS
application;
Internal
report on Investigation Related Recoveries; and
Internal
and external reports on the four performance measures (including OIG, General
Accounting Office, etc.)
Identified
and tested critical controls (both EDP and manual) of ACIS.
Based on
the understanding we obtained during the planning part of this engagement, a
review of related prior-year audit work at SSA, and our understanding of the
Federal Information Systems Controls Audit Manual (FISCAM) methodology, we developed
and performed tests of internal controls (both general and application) related
to ACIS, for the following areas:
Access
Control (including Separation of Duties);
Data
Input;
Data
Rejection;
Data
Processing (including backup and recovery); and
Data
Output.
Tested
the accuracy of the underlying FY 2000 data for each of the specified performance
measures.
To verify,
validate, and test the accuracy of the FY 2000 data we performed the following:
Selected
a random sample of 45 OASDI, 45 SSI, and 20 criminal conviction cases from
ACIS by performing the following:
Obtained
a copy of the ACIS data as of September 30, 2000 from the OIG Office of Executive
Operations;
Created
and executed audit control language (ACL) programs to extract random samples
for Old-Age, Survivors and Disability Insurance, Supplemental Security Income
and Conviction Cases; and
Created
an ACL program to display the monetary figures for the cases selected.
We requested
copies of the OI-1 "Case Opening Report," OI-9 "ACIS Criminal
& Administrative Disposition Form" and the OI-68 "Court-Ordered
Restitution/Judgment Form" and performed the following:
Verified
that a completed OI-1 was properly signed and completed;
Verified
that the OI-68 was completed and was supported by documentation in the file;
Verified
that a completed OI-9 was properly signed by the Field Division SAC;
Traced
the monetary achievement documentation on the OI-9 and OI-68 to the amount
of monetary achievement in ACIS to ensure the monetary amount reported in
ACIS was in fact correct; and
Ensured
that the OI-4 agreed with information on OI-9, OI-68, and ACIS. OI-4 obtained
from sub-offices that choose to include them in the documentation sent.
Judgmentally
selected a sample of 10 case files from the closed cases at the Philadelphia
field division (FD), and with the assistance of FD personnel we traced and
agreed the OI-9 to the information in ACIS. In addition, we ensured that the
information included on the OI-9 was valid by performing the following:
Ensured
the following documents were included in the file:
OI-1
Case Opening Report, properly signed and completed by the FD Special Agent-in-Charge
(SAC).
Overpayment
recipient Numident, and Master Beneficiary Record (MBR)/Supplemental Security
Record (SSR).
OI-4
Report of Investigation, reviewed and signed by the FD Assistant Special Agent-in-Charge
(ASAC).
Supervisory
File review sheet.
OI-9
ACIS input form, properly signed and completed by the FD SAC. Ensured that
the OI-9 agreed with the information on the OI-4.
OI-31
Case Closing Checklist reviewed and signed by the FD ASAC.
For cases
involving restitution we ensured that in addition to the above documents the
case file included the following:
OI-16A
Statement of Overpayment Recipient signed by the subject and the SAC.
Copy
of check provided by the subject.
Check
receipt from SSA personnel.
Recalculated
each specific FY 2000 measure to ascertain its mathematical accuracy
Based on
our understanding of SSA and Performance Measures (PM) #5 through #8, we obtained
the FY 2000 ACIS data and performed Computer Assisted Audit Techniques (CAATs)
using ACL to accumulate counts for four measures, and then compared those results
to the figures reported for GPRA. We then reconciled any differences through
data analysis and subsequent discussions with SSA OIG personnel.
Determined
whether performance measures were meaningful and in compliance with GPRA
As part
of this engagement, we evaluated the appropriateness of each of the performance
measures with respect to GPRA compliance and SSA’s APP. We determined whether
the specific indicators and goals corresponded to the strategic goals identified
in SSA’s APP, determined whether each of these indicators accurately measure
performance, and determinedtheir compliance with GPRA requirements.
ACRONYMS
ACIS Allegation
and Case Investigative System
ADABAS A
Database
AMD Allegation
Management Division
AMS Allegation
Management System
APP Annual
Performance Plan
ASAC Assistant
Special Agent-in-Charge
CAATs Computer
Assisted Audit Techniques
CIT Center
for Information Technology
DI Disability
Insurance
DMS Debt
Management System
EDP Electronic
Data Processing
FD Field
Division
FIPS Federal
Information Processing Standards
FISCAM Federal
Information System Controls Audit Manual
FY Fiscal
Year
GPRA Government
Performance and Results Act
IT Information
Technology
J&C Judgment
and Commitment Order
MAD Manpower
and Administration Division
MATPSC Mid-Atlantic
Program Service Center
MBR Master
Beneficiary Record
NIH National
Institutes of Health
OASDI Old-Age
and Survivors and Disability Insurance
OEO Office
of Executive Operations
OI Office
of Investigations
OIG Office
of the Inspector General
OMB Office
of Management and Budget
PM Performance
Measure
PwC PricewaterhouseCoopers
LLP
SA Special
Agent
SAC Special
Agent-in-Charge
SAH Special
Agent Handbook
SAS Statements
on Auditing Standards
SLA Service
Level Agreement
SSA Social
Security Administration
SSI Supplemental
Security Income
SSN Social
Security number
SSR Supplemental
Security Record
Performance
Measure Summary Sheets
Name
of Measure
Measure
Type
Strategic
Goal/Objective
5)
Number of investigations conducted (i.e., closed)
Workload
Goal:
To make SSA program management the best-in-business, with zero tolerance
for fraud and abuse.
Objective:
To aggressively deter, identify and resolve fraud.
Purpose
Report
Frequency
To
increase the number of investigations conducted resulting from allegations
that have sufficient information or potential risk to warrant further
review or action by a criminal investigator.
Semiannual
How
Computed
Data
Source
Data
Availability
Data
Quality
Investigations
are counted as "conducted" when all OIG actions have been completed,
i.e., the investigator has presented the facts of the case to a prosecutor
or has determined that further action is not warranted due to lack of
investigative leads.
ACIS
Adequate
Adequate
Target
Goal
Division
Designated
Staff Members
7,600
OIG
Mike
Arbuco, Amy Shemenski, Dennis Fabel, Dawn Zgorski, Oliver Webb
EDP
Controls Testing and Results
We
performed tests of internal controls (both general and application) related
to ACIS, for the following areas:
Access
Control (including Separation of Duties);
Data
Input;
Data
Rejection;
Data
Processing (including backup and recovery); and
Data
Output.
Refer
to "Results of Evaluation" for a description of the findings.
CAATs
Testing and Results
We
obtained the FY 2000 ACIS data and performed CAATs using ACL to accumulate
counts for this measure, and then compared those results to those reported
for GPRA. We then reconciled any differences through data analysis and
subsequent discussions with SSA OIG personnel.
Refer
to "Results of Evaluation" for a description of the findings.
Data
and Manual Controls Testing and Results
We
verified the completeness and accuracy of the ACIS data by judgmentally
selecting a sample of 10 case files from the closed cases cabinet in the
Philadelphia OI office and performed the following:
Ensured
the following documents were included in the file:
Allegation
report (i.e., SSA-8551, Fraud Referral Form, or other type of allegation
report)
OI-1
Case opening report, properly signed and completed by the field division
SAC
Overpayment
recipient Numident, and MBR/SSR
OI-4
Report of Investigation, reviewed and signed by the field division ASAC
Supervisory
File review sheet
OI-9
ACIS input form, properly signed and completed by the Field Division
SAC. Ensured that the OI-9 agreed with the information on the OI-4.
OI-31
Case closing Checklist, reviewed and signed by the field division ASAC
For
cases involving restitution, ensured that in addition to the above documents
the case file included the following:
OI-16A
statement of overpayment recipient signed by the claimant and the SAC
Copy
of check provided by the claimant
Check
receipt from SSA personnel
Traced
and agreed the information in the OI-9 to the information in ACIS.
Refer
to "Results of Evaluation" for a description of the findings.
Name
of Measure
Measure
Type
Strategic
Goal/Objective
6)
OASDI dollar amounts reported from investigative activities
Workload
Goal:
To make SSA program management the best-in-business, with zero tolerance
for fraud and abuse.
Objective:
To aggressively deter, identify and resolve fraud.
Purpose
Report
Frequency
To
report OASDI dollars from penalties, assessments, savings, recoveries
and restitutions related to investigative activities.
Semi-Annual
How
Computed
Data
Source
Data
Availability
Data
Quality
OASDI
dollars from penalties, assessments, savings recoveries, and restitutions
related to investigative activities that are reported by OIG filed divisions
and included in OIG semi-annual reports.
ACIS
Adequate
Adequate,
except for the data integrity anomalies found as part of our evaluation.
Refer to "Results of Evaluation" for a description of the findings.
Target
Goal
Division
Designated
Staff Members
$
40 Million
OIG
Mike
Arbuco, Amy Shemenski, Dennis Fabel, Dawn Zgorski, Oliver Webb
EDP
Controls Testing and Results
We
performed tests of internal controls (both general and application) related
to ACIS, for the following areas:
Access
Control (including Separation of Duties);
Data
Input;
Data
Rejection;
Data
Processing (including backup and recovery); and
Data
Output.
Refer
to "Results of Evaluation" for a description of the findings.
CAATs
Testing and Results
We
obtained the FY 2000 ACIS data and performed CAATs using ACL to accumulate
counts for this measure, and then compared those results to those reported
for GPRA. We then reconciled any differences through data analysis and
subsequent discussions with SSA OIG personnel.
Refer
to "Results of Evaluation" for a description of the findings.
Data
and Manual Controls Testing and Results
To
ensure that accuracy of the reported information we selected a judgmental
sample of 45 OASDI cases from ACIS and performed the following:
Verified
that a completed OI-9 "ACIS input form" was properly signed
by the Field Division SAC
Verified
that a completed OI-1 form was properly signed and completed
Verified
that the OI-68 has been completed and was supported by documentation
in the file
Trace
monetary achievement documentation in case file to amount of monetary
achievement reported in ACIS
Refer
to "Results of Evaluation" for a description of the findings.
Name
of Measure
Measure
Type
Strategic
Goal/Objective
7)
SSI dollar amounts reported from investigative activities
Workload
Goal:
To make SSA program management the best-in-business, with zero tolerance
for fraud and abuse.
Objective:
To aggressively deter, identify and resolve fraud.
Purpose
Report
Frequency
To
report SSI dollars from penalties, assessments, savings, recoveries and
restitutions related to investigative activities.
Semi-annual
How
Computed
Data
Source
Data
Availability
Data
Quality
SSI
dollars from penalties, assessments, savings recoveries, and restitutions
related to investigative activities that are reported by OIG filed divisions
and included in OIG semi-annual reports.
ACIS
Adequate
Adequate,
except for the data integrity anomalies found as part of our evaluation.
Refer to "Results of Evaluation" for a description of the findings.
Target
Goal
Division
Designated
Staff Members
$
80 Million
OIG
Mike
Arbuco, Amy Shemenski, Dennis Fabel, Dawn Zgorski, Oliver Webb
EDP
Controls Testing and Results
We
performed tests of internal controls (both general and application) related
to ACIS, for the following areas:
Access
Control (including Separation of Duties);
Data
Input;
Data
Rejection;
Data
Processing (including backup and recovery); and
Data
Output.
Refer
to "Results of Evaluation" for a description of the findings.
CAATs
Testing and Results
We
obtained the FY 2000 ACIS data and performed CAATs using ACL to accumulate
counts for this measure, and then compared those results to those reported
for GPRA. We then reconciled any differences through data analysis and
subsequent discussions with SSA OIG personnel.
Refer
to "Results of Evaluation" for a description of the findings.
Data
and Manual Controls Testing and Results
To
ensure that accuracy of the reported information we selected a judgmental
sample of 45 SSI cases from ACIS and performed the following:
Verified
that a completed OI-9 "ACIS input form" was properly signed
by the Field Division Special Agent In Charge (SAC)
Verified
that a completed OI-1 form was properly signed and completed
Verified
that the OI-68 has been completed and was supported by documentation
in the file
Traced
monetary achievement documentation in case file to amount of monetary
achievement reported in ACIS
Refer
to "Results of Evaluation" for a description of the findings.
Name
of Measure
Measure
Type
Strategic
Goal/Objective
8) Number of Criminal Convictions
Workload
Goal:
To make SSA program management the best-in-business, with zero tolerance
for fraud and abuse.
Objective:
To aggressively deter, identify and resolve fraud.
Purpose
Report
Frequency
To
report the number of criminal convictions as related to SSA/OIG investigative
activities.
Semi-Annual
How
Computed
Data
Source
Data
Availability
Data
Quality
Number
of criminal convictions as related to SSA/OIG investigative activities.
ACIS
Adequate
Adequate
Target
Goal
Division
Designated
Staff Members
1,800
OIG
Mike
Arbuco, Amy Shemenski, Dennis Fabel, Dawn Zgorski, Oliver Webb
EDP
Controls Testing and Results
We
performed tests of internal controls (both general and application) related
to ACIS, for the following areas:
Access
Control (including Separation of Duties);
Data
Input;
Data
Rejection;
Data
Processing (including backup and recovery); and
Data
Output.
Refer
to "Results of Evaluation" for a description of the findings
CAATs
Testing and Results
We
obtained the FY 2000 ACIS data and performed CAATs using ACL to accumulate
counts for this measure, and then compared those results to those reported
for GPRA. We then reconciled any differences through data analysis and
subsequent discussions with SSA OIG personnel.
Refer
to "Results of Evaluation" for a description of the findings
Data
and Manual Controls Testing and Results
To
ensure that accuracy of the reported information we selected a judgmental
sample of 20 criminal convictions from ACIS and performed the following:
Verified
that a completed OI-9 "ACIS input form" was properly signed
by the Field Division Special Agent In Charge (SAC)
Verified
that a completed OI-1 form was properly signed and completed
Verified
that the OI-68 has been completed and was supported by documentation
in the file
Traced
monetary achievement documentation in case file to amount of monetary
achievement reported in ACIS
Refer
to "Results of Evaluation" for a description of the findings
