Significant Progress Has Been Made in Consolidating Mainframe Computer Operations, But Risks Remain
June 2000
Reference Number: 2000-20-085
This report has cleared the Treasury Inspector General for Tax Administration disclosure review process and information determined to be restricted from public release has been redacted from this document.
June 14, 2000
MEMORANDUM FOR COMMISSIONER ROSSOTTI
FROM: Pamela J. Gardiner /s/ Pamela J. Gardiner
Deputy Inspector General for Audit
SUBJECT: Final Audit Report - Significant Progress Has Been Made in Consolidating Mainframe Computer Operations, But Risks Remain
This report presents the results of our review of the Internal Revenue Service’s (IRS) Service Center Mainframe Consolidation (SCMC) project. We conducted the review to determine whether SCMC project management controls are adequate to ensure that continuing service center consolidations are successful. We also evaluated some of the corrective actions taken as a result of our prior audit of SCMC.
In summary, we found the IRS has made significant progress in ensuring the computing and service centers’ readiness for consolidation and in minimizing any effect on operations and taxpayers; however, additional performance testing and processing controls are needed.
We issued a draft of this report to IRS management on April 7, 2000, with a May 8, 2000, response period. However, management’s response was not available as of the date this report was released.
Copies of this report are also being sent to the IRS managers who are affected by the report recommendations. Please contact me at (202) 622-6510 if you have questions, or your staff may call Scott E. Wilson, Associate Inspector General for Audit (Information Systems Programs), at (202) 622-8510.
Appendix I – Detailed Objective, Scope, and Methodology
Appendix II – Major Contributors to This Report
Appendix III – Report Distribution List
The Internal Revenue Service (IRS) is in the process of consolidating the mainframe computer operations used to process tax data. The consolidation involves moving mainframe processing from the IRS’ 10 service centers into new mainframe computers located in 2 computing centers—the Tennessee Computing Center (TCC) and the Martinsburg Computing Center (MCC). A successful mainframe consolidation effort is critical since the IRS must rely on these systems each year to process tax returns and to assist taxpayers by providing answers to questions about their tax accounts.
This is our third audit of the Service Center Mainframe Consolidation (SCMC) Project. The second audit, The Service Center Mainframe Consolidation Project Has Made Significant Progress, But Project Execution and Administration Risks Remain (Reference Number 199920068, dated September 1999) reported that: (1) critical risks related to SCMC remain unresolved, (2) controls over technical aspects of contract administration remain inadequate, and (3) staffing costs of consolidation were not accurately budgeted, captured, and reported.
The overall objective of our current review was to determine whether SCMC project management controls are adequate to ensure that continuing service center consolidations are successful. In addition, we evaluated some of the corrective actions taken as a result of the prior audit.
Results
The IRS has made significant progress in ensuring the computing and service centers’ readiness for consolidation and in minimizing any effect on operations and taxpayers. The SCMC Project Office has initiated process improvements identified during the initial service center consolidations and established risk management activities. In addition, the computing centers have conducted extensive recruiting efforts to mitigate staffing shortages. However, unresolved risks may impact successful consolidation of mainframe computer operations for the remaining five service centers.
A Segment of the Consolidated Mainframe Computer Environment May Not Support the Additional Workload of the Remaining Service Centers
Since the initial migrations of service center processing to the computing centers, there have been multiple instances when some components of the consolidated mainframe computer environment have experienced capacity related problems and could not process the workload effectively. Senior IRS management has been tracking the capacity issue since December 1998 and recently recommended the creation of a separate committee to identify and recommend solutions to outstanding issues.
During June 13-17, 1999, the Capacity Management Branch of the Systems Support Division (SSD) performed a capacity analysis of a segment of the consolidated computer environment. The SSD report included several recommendations to improve system performance and stated that additional performance testing is required to ensure that sufficient capacity is provided to ensure that each computing center can handle a five service center configuration. Although the SCMC Project Office, in conjunction with other IRS organizations and contractors, has closely analyzed problems and has taken measures to address concerns with system performance, additional testing has not been performed to establish that computing centers can support the workload of five service centers.
The capacity issue could also affect the capability of the IRS to implement planned disaster recovery activities, as each computing center was supposed to be able to process 70 percent of the workload from the other computing center in case of a disaster.
Without conducting additional performance testing, including disaster recovery capability, management cannot be assured that the consolidated mainframe environment has sufficient computer capacity for timely tax return processing.
Continued Information System Control Weaknesses Could Result in Incomplete or Inaccurate Data Processing
The General Accounting Office’s Standards for Internal Control in the Federal Government identifies two types of information system controls. Application controls, one of the control types, help ensure completeness, accuracy, authorization, and validity of all transactions during application processing. Application controls are also installed where data is transferred from one system to another system to ensure that all inputs are received and are valid, and outputs are correct and properly distributed. In our September 1999 audit, we reported application control weaknesses.
To address the control deficiencies, IRS management requested computer programming changes to automate the run-to-run balancing process and began developing the Output Tracking System (OTS) to track files transferred between the service centers and computing centers. However, with 5 of the 10 service centers consolidated, these control initiatives are not complete and may not be completed before the remaining consolidations are implemented. Additional service center consolidations increase the risk of incomplete or inaccurate tax return processing that could result in untimely or erroneous notices to taxpayers.
Output Tracking System Development Efforts Did Not Always Adhere to Prescribed Policies and Procedures
The OTS development efforts indicate that the SCMC Project Office did not always adhere to IRS standards. For example, OTS test results were not forwarded to the Office of Security, Evaluation, and Oversight for an independent review and approval before pilot implementation. In addition, the SCMC Project Office did not obtain proper approval to operate the OTS pilot in the production environment.
System development efforts that do not follow prescribed policies and procedures could result in new systems negatively affecting system processing or potentially creating security weaknesses in the production environment.
Summary of Recommendations
The Chief Information Officer (CIO) should ensure that additional performance testing establishes that the consolidated mainframe environment can support the workload of five service centers, including disaster recovery processing requirements. The CIO should also ensure that adequate processing controls are in place for automated balancing and file tracking processes prior to SCMC Project completion and that the OTS development efforts follow established IRS system development procedures.
Management’s Response: We issued a draft of this report to IRS management on April 7, 2000, with a May 8, 2000, response period. However, management’s response was not available as of the date this report was released.
This review was initiated as a continuation of a prior Service Center Mainframe Consolidation (SCMC) review (The Service Center Mainframe Consolidation Project Has Made Significant Progress, But Project Execution and Administration Risks Remain, Reference Number 199920068, dated September 1999). The overall objective of the current review was to determine whether SCMC project management controls were adequate to ensure that continuing service center consolidations are successful. In addition, we evaluated some of the corrective actions taken as a result of the prior audit.
We conducted the review between September 1999 and January 2000 in accordance with Government Auditing Standards. We interviewed key personnel and reviewed documentation at the following sites.
Details of our audit objective, scope, and methodology are presented in Appendix I. Major contributors to this report are listed in Appendix II.
In 1995, the Office of Management and Budget issued Bulletin Number 96-02 providing a requirement for consolidation of data centers with large mainframe operations. In 1997, the Internal Revenue Service (IRS) formed the SCMC Project Office with an objective to consolidate mainframe computers from 10 service centers into 2 computing centers. In addition, over 17,000 desktop computer terminals were to be replaced in conjunction with the consolidation effort. A successful mainframe consolidation effort is critical since the IRS must rely on these systems each year to process over 200 million tax returns and to assist over 130 million taxpayers by providing answers to questions about their tax accounts.
The following systems residing on four mainframe computers in each of the service centers are being consolidated as part of the SCMC Project:
The replacement of CRS with the Security and Communications System (SACS) was completed in February 1999. In addition, the SCMC Project Office reported that all terminals had been replaced as of September 30, 1998.
The IRS has experienced delays in its consolidation of the remaining mainframe computers and an increase in the estimated SCMC Project cost. The IRS had planned to consolidate all 10 service centers by December 1998; however, only 3 service centers (Brookhaven, Kansas City and Memphis) were actually consolidated by this date. Two additional service centers (Andover and Cincinnati) were consolidated in 1999. The IRS delayed the consolidations due to the identification of additional service center prerequisites for consolidation, limited Information System resources because of computer programming changes necessary for the Year 2000, expanded business requirements, and concerns about the ambitious schedule. The remaining five service centers are scheduled for consolidation between June and December 2000, which coincides with several upgrades to mainframe processors, operating systems, and storage devices.
In addition, the SCMC Project Office reported in March 2000 that the five-year cost estimate for the SCMC Project increased by 11 percent to nearly $480 million. The SCMC Project Office attributed the increase to new or expanded business requirements for disaster recovery and customer service.
The IRS has made significant progress in ensuring the computing and service centers’ readiness for consolidation and in minimizing any effect on operations and taxpayers. The consolidation of service center mainframe computer operations is obviously a very complex task, which has required extensive coordination and effort by several contractors and IRS functions. The SCMC Project Office has initiated process improvements identified during the initial service center consolidations and established risk management activities, such as the Risk Oversight Committee and the SCMC Executive Steering Committee (ESC). In addition, the computing centers have conducted extensive recruiting efforts to mitigate staffing shortages. However, the following unresolved risks may impact successful consolidation of mainframe computer operations for the remaining five service centers:
The Chief Information Officer (CIO) should ensure that additional performance testing establishes that the consolidated mainframe environment can support the workload of five service centers, including disaster recovery processing requirements. The CIO should also ensure that adequate processing controls are in place for automated balancing and file tracking processes prior to SCMC Project completion and that the OTS development efforts follow established IRS system development procedures.
Process Improvements and Recruiting Efforts Have Enhanced Consolidation Readiness and Minimized the Effect on Operations and Taxpayers
In response to the problems encountered during the initial consolidations, the SCMC Project Office began implementing process improvements. The computing centers also conducted extensive recruiting efforts to address staffing shortages. These actions have significantly improved the readiness of the service centers and computing centers for consolidation and minimized any effect on operations and taxpayers.
Process Improvements
As a result of implementing process improvements, the consolidation of mainframe computer operations from the Andover and Cincinnati Service Centers to MCC and TCC, respectively, went smoothly and had little or no effect on service center operations and taxpayers. The amount of cooperation and communication between the centers was a significant contributing factor to the successful consolidation. Some of the process improvements implemented for the 1999 consolidation of computer mainframe operations included:
Other process improvements that enhanced the service centers’ consolidation readiness included improvements in programming changes to the Executive Control Language, building of the service center processing schedule, developing plans for the initial transfer of data from the service centers to the computing centers, and preparing the Service Level Agreement between the centers.
Recruiting Efforts
The IRS and the National Treasury Employees Union (NTEU) signed a memorandum of understanding on February 9, 1998, providing placement opportunities for those service center employees whose positions were affected by the consolidation effort.
In our September 1999 audit, we reported that the TCC and MCC continued to report staffing shortages of approximately 25 percent with few options remaining to correct the problem. If not corrected, the computing centers might not be able to accommodate increasing workloads resulting in processing delays, which could adversely affect a significant number of taxpayers. In response, the IRS implemented additional measures.
Although the measures taken by the IRS have improved the staffing situation, the computing centers continue to experience staffing shortages.
The MCC is authorized to have 177 employees for service center mainframe processing when all 5 service centers are consolidated. As of January 20, 2000, 21 of the 177 (12 percent) MCC positions were vacant.
The TCC is authorized to have 172 employees for service center mainframe processing when all 5 service centers are consolidated. Currently, 46 of the 172 (27 percent) TCC positions are vacant.
MCC and TCC management are continuing recruiting efforts to fill the additional positions and are confident that they can effectively manage the consolidated workloads with the current staffing levels.
A Segment of the Consolidated Mainframe Computer Environment May Not Support the Additional Workload of the Remaining Service Centers
There have been multiple instances when the Virtual Machine (VM) Server-Client Server Component experienced capacity related problems and could not process the Automated Tape Library (ATL) workload effectively. In addition, the TCC has expressed a concern with the number of tape slots within the ATL needed to support a five service center workload. The SCMC ESC has been tracking the VM Server/ATL complex as a capacity issue since December 1998 and recently recommended the creation of an Engineering Review Board (ERB). On November 3, 1999, an ERB working group, entitled the Computing Center Tape Processing Environment, was formed to identify and recommend solutions to outstanding issues.
The IRS Systems Support Division (SSD), Capacity Management Branch personnel perform periodic reviews of computer systems to assure optimal performance of the computer equipment and software. During June 13-17, 1999, SSD personnel performed an analysis of the VM Server. The report issued by the SSD included several recommendations. The first recommendation, which was required to be implemented before consolidating a third service center, focused on preventing a catastrophic failure. Additional recommendations included reducing the number of tape mounts per hour on the VM Server by shifting some of the workload to another server, copying disaster recovery tapes during off peak processing hours, and combining several databases onto one tape cartridge. The report stated that these recommendations should be implemented before consolidating the fourth and fifth service centers. The SSD report also said that additional performance testing of the VM Server/ATL complex is required after a tape mounting problem is corrected to ensure that sufficient capacity is provided to handle a five service center configuration.
The SCMC Project Office, in conjunction with other IRS organizations and contractors, have closely analyzed problems and have taken measures to address concerns with the performance of the VM Server. The SCMC Project Office reported that tests performed in September and October of 1999 established that the first recommendation to prevent catastrophic failure had been addressed. In addition, the ERB working group is currently overseeing the activities to reduce the number of tape mounts per hour on the VM Server.
However, past performance indicates that various VM Server/ATL problems occur as a computing center adds additional service centers to its workload. For example, even after implementing the first recommendation, the TCC still experienced a VM Server/ATL complex related problem when consolidating its third service center in November 1999. The problem, which resulted in tapes not being mounted in the ATL, occurred more than once and one incident resulted in a work stoppage that impacted the daily runs.
Without conducting additional performance testing of the VM Server/ATL complex to establish that it can support the workload of five service centers, management cannot ensure that the consolidated environment has sufficient computer capacity for timely tax return processing. The VM Server/ATL capacity issue could also affect the capability of the IRS to implement planned disaster recovery activities, as each computing center was supposed to be able to process 70 percent of the workload from the other computing center in case of a disaster. Therefore, the additional testing activities should not only establish whether the computing center processing environment can support the workload of 100 percent of the processing capacity for 5 service centers, but also whether it can support the additional workload of 70 percent of the processing capacity from the other computing center.
Recommendation
Management’s Response: Management’s response was not available at the time this report was released.
Continued Information System Control Weaknesses Could Result in Incomplete or Inaccurate Data Processing
The General Accounting Office’s Standards for Internal Control in the Federal Government identifies the two broad groupings of information system controls as general and application controls. General controls are the structure, policies, and procedures that apply to an entity’s overall computer operations. Application controls are designed to help ensure completeness, accuracy, authorization, and validity of all transactions during application processing.
Some application controls are installed where data is transferred from one system to another system to ensure that all inputs are received and are valid, and outputs are correct and properly distributed. The prior review of the SCMC Project reported application control weaknesses with run-to-run balancing and file tracking.
To address the control deficiencies, IRS management implemented two courses of action.
These actions should address the application control weaknesses reported. However, with five service centers consolidated, these control initiatives are not complete and may not be completed before the remaining consolidations are implemented. Additional service center consolidations increase the risk of incomplete or inaccurate tax return processing that could result in untimely or erroneous notices to taxpayers.
Run-to-Run Balancing
During the initial service center consolidations, run-to-run balancing controls were identified by IRS Information Systems (IS) management as a high priority risk to be reduced. Two requests for computer programming changes were submitted to automate the run-to-run balancing process. One request was prepared to automate the balancing of records between input and output files during internal system processing. This computer programming change was reported by IS as completed in March 2000.
The second request was submitted to automate the balancing of records for files exchanged between the computing centers and service centers. The proposed operational date was July 1, 1999. The request was recently closed with no action taken because of a delay in defining the requirements by the requesting IRS organization. Another request will be submitted once the requirements are clearly defined.
File Tracking
To ensure that all input files are received and output files are properly distributed, a control should be implemented to track files transferred between systems. The first off-site service center consolidation demonstrated that the IRS needed to be able to better track data and print files transferred between the service center and computing center. The Output Tracking System (OTS) is the system under development to meet this need. The OTS was originally scheduled for implementation by August 1999; however, the development effort has experienced several setbacks that were partially caused by technical design deficiencies. Currently, the OTS is scheduled for implementation by June 2000.
Incorporating automated run-to-run balancing and file tracking controls into a system helps to ensure completeness, accuracy, authorization, and validity of all transactions during processing. Controls also ensure that all inputs are received and are valid, and outputs are correct and properly distributed. Due to the delay in implementing OTS, a manual process was developed in the interim to balance and track exchanged files; however, it was not consistently implemented and needs to be automated prior to SCMC Project completion.
Recommendation
Output Tracking System Development Efforts Did Not Always Adhere to Prescribed Policies and Procedures
The major phases of the System Development Life Cycle (SDLC) are initiation, development, and operation. Several activities occur during the SDLC phases to ensure that security safeguards work effectively and the system meets applicable security policies and procedures. We determined that the SCMC Project Office did not always adhere to IRS standards during OTS development. For example:
System development efforts that do not follow prescribed policies and procedures could result in new systems negatively affecting system processing or potentially creating security weaknesses in the production environment.
Proof of Concept Test Results
As the principal advisor to senior management on the IRS’ Security Program, the SEO focuses on security issues in the SDLC process. The change management process developed by the SEO for mainframe consolidation requires that upon completion of testing, the test results must be forwarded to the SEO for review and approval before proceeding to the next phase of the SDLC. Although the SCMC Project Office had proceeded to move the OTS from the test phase to production in a pilot operation, the SEO was not provided the test results for review and approval, as required.
The OTS was accepted by the SCMC Project Office for pilot implementation in a production environment beginning on August 16, 1999, at one service center after completion of the POC test in July 1999. However, the Program Manager acknowledged some flaws in the POC test in that it did not completely simulate a production environment, which contributed to some of the problems experienced during the pilot and negatively impacted service center processing.
On September 2, 1999, the SEO Director issued a memorandum requesting the test report and raising concerns about the stability of the OTS and the potential risks to operations in implementing the system. As of December 7, 1999, the test report and other requested documents had not been provided to the SEO. The OTS has since been removed from the production environment because Year 2000 compliance testing had not been completed. The OTS pilot is scheduled to resume in early 2000. The SCMC Project Office explained that with the delay in the pilot, the decision was made to respond to the SEO’s request for information when the pilot was resumed.
Security Certification and Accreditation
In August 1999, the SCMC Project Office placed the OTS pilot in the production environment without completing a security certification and accreditation. Systems that process sensitive but unclassified information (i.e., tax return information) require a formal review (certification) and issuance of an official declaration (accreditation) to operate. Pending accreditation, an interim authority to operate (IAO) is permitted if certain conditions are satisfied. A typical situation where an IAO might be used is when a new system is in an advanced test stage and a risk analysis has concluded that no apparent security problems exist.
The SCMC Project Office obtained an IAO dated July 30, 1999, for the mainframe consolidation effort. The risk assessments and security evaluation report were based upon an evaluation of the consolidated processing environment that excluded OTS. The SCMC Project Office assumed that the current IAO would allow authorization for the OTS pilot. However, IRS guidelines require management to update the risk assessment and security evaluation report when there is a significant change to the current system environment. By not following established certification and accreditation procedures, new systems introduced into a production environment may negatively affect system processing or create security weaknesses.
Recommendation
The SCMC Project Office has made significant progress in ensuring the timely and accurate processing of tax data for the consolidated service centers. However, the unresolved risks discussed in this report may impact successful consolidation of mainframe computer operations for the remaining five service centers.
Appendix I
Detailed Objective, Scope, and MethodologyThe overall objective of this review was to determine whether the Internal Revenue Service’s (IRS) Service Center Mainframe Consolidation (SCMC) project management controls were adequate to ensure that the continuing service center consolidations are successful. We also evaluated corrective actions taken as a result of the prior review (The Service Center Mainframe Consolidation Project Has Made Significant Progress, But Project Execution and Administration Risks Remain, Reference Number 199920068, dated September 1999).
I. To evaluate management’s efforts to identify, control, and mitigate key risks present in the SCMC Project, we:
A. Attended project conference calls and reviewed project status reports and risk oversight committee meeting minutes to determine if high-risk issues are identified and controlled for resolution.
III. To conduct follow-up to recommendations/corrective actions from prior audits on the SCMC Project, we:
Appendix II
Major Contributors to This ReportScott E. Wilson, Associate Inspector General for Audit (Information Systems Programs)
Gary Hinkle, Director
Danny Verneuille, Audit Manager
Van Warmke, Senior Auditor
Olivia Jasper, Auditor
Kim McManis, Auditor
Barbara Sailhamer, Auditor
Linda Screws, Auditor
Tina Wong, Auditor
Appendix III
Report Distribution ListDeputy Commissioner Modernization C:DM
Deputy Commissioner Operations C:DO
Chief Operations Officer OP
Chief Information Officer IS
Deputy Chief Information Officer, Operations IS
Deputy Chief Information Officer, Systems IS
Executive Officer for Service Center Operations OP:SC
Director, Enterprise Operations IS:E
Director, Service Center Operations IS:SC
Director, Systems Development IS:SD
Director, Martinsburg Computing Center IS:E:MC
Director, Tennessee Computing Center IS:E:TC
Director, Mainframe Consolidation Project Office IS:E:CO
Director, Security Evaluation and Oversight IS:SPO:S
Director, Information Resources Management IS:O:IR
Director, Office of Program Evaluation and Risk Analysis M:O
National Director for Legislative Affairs CL:LA
Office of Management Controls M:CFO:A:M
Office of the National Taxpayer Advocate C:TA
Office of the Chief Counsel CC
Audit Liaison, Program Oversight & Management Control Office IS:IR:OM