Additional
Actions Are Needed to Establish and Maintain Controls Over Computer Hardware
and Software Changes
December 2003
Reference
Number: 2004-20-026
This report has cleared the Treasury
Inspector General for Tax Administration disclosure review process and
information determined to be restricted from public release has been redacted
from this document.
December
16, 2003
MEMORANDUM
FOR CHIEF INFORMATION OFFICER
FROM: Gordon C.
Milbourn III /s/ Gordon C. Milbourn III
Acting Deputy Inspector General for Audit
SUBJECT: Final Audit Report – Additional
Actions Are Needed to Establish and Maintain Controls Over Computer Hardware
and Software Changes (Audit # 200320015)
This
report presents the results of our review of the Internal Revenue Service’s
(IRS) configuration management (CM) process for computer hardware and software. The overall objective of this review was to determine whether the
IRS’ Modernization and Information Technology Services (MITS) organization
effectively implemented an enterprise-wide CM process.
The IRS is
dependent on a large collection of computer systems with complex
interdependencies among a network of mainframe computers, mid-range computers,
individual computers, several hundred vendor-supplied software products, and
millions of lines of computer code. The
MITS organization is currently modernizing, consolidating, and maintaining
these computer systems to support the mission of the IRS. Responsibility for managing these computer
systems is divided among the MITS organizations as follows:
·
The
Information Technology Services (ITS) organization develops, operates, and
maintains computer hardware and software that supports the production
environment.
·
The Business
Systems Modernization Office (BSMO) acquires and delivers new computer hardware
and software for the IRS’ modernized business processes.
Among the
disciplines needed to manage and coordinate these efforts is an integrated CM
process to ensure that the integrity and consistency of the IRS’ computer
systems are maintained throughout their life cycles. The CM process systematically identifies and baselines the items
that make up a system (identification), formally controls any modifications to
those items (control), reports on the status of the CM process (status
accounting), and ensures that baseline configurations are implemented (audit).
In summary, the MITS organization has made progress in defining
and establishing an enterprise-wide CM process through the issuance of a CM
Directive that describes the CM process to be used throughout the MITS
organization, and standard operating procedures (e.g., configuration control
boards, configuration items and baselines, and configuration control). The MITS organization has chartered a
Configuration Management Working Group to establish, maintain, and improve the
CM process.
However, the CM
functions (i.e., identification, control, status accounting, and audit) have
not been uniformly implemented within the MITS organization. An integrated, enterprise-wide
implementation of the CM process within the MITS organization is particularly
important for modernized systems that will migrate in stages or releases to
ensure computer system changes are properly managed throughout their life
cycles. In addition, this process
provides a means to document, communicate, and coordinate system development
and production CM baselines between the BSMO and the ITS organizations. For example, as a result of CM control weaknesses,
the Enterprise
Systems Management (ESM) project incurred additional contractor costs of
approximately $216,500 and a 4-month schedule delay to rollout ESM Release
2.1. Without an integrated and uniform
CM process, there is an increased potential that modernized and existing
systems will require extensive rework resulting in additional costs, schedule delays, and other risks
to the IRS’ computer operations (e.g., system outages and data
corruption).
The implementation deficiencies found in the MITS organization’s CM processes occurred
because the MITS CM
Directive and procedures did not establish executive level responsibility that
would ensure that:
·
The CM
processes were implemented throughout the ITS organization and coordinated with
the BSMO.
·
Deficiencies
identified in internal CM assessments were appropriately addressed.
·
The different CM
software used by the MITS
organization facilitated enterprise-wide CM.
·
Policies
were established defining authority levels and threshold criteria to approve
and control production changes in the ITS organization.
To promote the establishment of an
integrated MITS organization CM process, we recommended that the Chief Information Officer (CIO)
modify the MITS CM Directive and procedures to: 1) assign responsibility for ensuring that MITS CM processes are
implemented throughout the ITS organization and coordinated with the BSMO and
that CM deficiencies are appropriately addressed; and 2) establish governance
policies, similar to those used by the BSMO, for defining the authority levels
and threshold criteria to approve and control changes to the production
environment in the ITS organization. Additionally, we recommended that the CIO develop a transition
plan to implement standardized Enterprise Architecture compliant CM software to
be used throughout the MITS organization to facilitate CM on an enterprise-wide
level.
Management’s
Response: IRS management agreed with our
recommendations. The MITS organization
will revalidate its CM Directive to address organizational responsibility,
governance policy, and needed improvements in the Configuration Control Board
(CCB) structure. In addition,
management will address the establishment of governance policies and threshold
criteria to approve and control changes to the ITS production environment while
establishing plans to organize separate CCBs for the ITS and BSMO
organizations. Regarding CM software,
the MITS organization will identify acceptable CM software and publish
applicable guidance upon the completion of ongoing CM software assessments
within the ITS organization.
Management’s complete response to the draft report is included as
Appendix VI.
Copies
of this report are also being sent to the IRS managers who are affected by the
report recommendations. Please contact
me at (202) 622-6510 if you have questions or Margaret
E. Begg, Assistant Inspector General for Audit (Information Systems Programs),
at (202) 622-8510.
Progress Has Been Made in Implementing
Configuration Management Processes
Appendix I – Detailed Objective,
Scope, and Methodology
Appendix II – Major Contributors to
This Report
Appendix III – Report Distribution
List
Appendix IV – Outcome Measures
Appendix V – Overview of Configuration
Management Functions
Appendix VI – Management’s Response to
the Draft Report
The Internal Revenue Service
(IRS) is dependent on a large collection of computer systems with complex
interdependencies among a network of mainframe computers, mid-range computers,
individual computers, several hundred vendor-supplied software products, and
millions of lines of computer code. The
IRS’ Modernization and Information Technology Services (MITS) organization is
currently modernizing, consolidating, and maintaining these computer systems to
support the IRS’ mission. Responsibility
for managing these computer systems is divided among the MITS organizations as
follows:
·
The Information Technology
Services (ITS) organization develops, operates, and maintains computer hardware
and software that supports computer systems in production.
·
The Business Systems
Modernization Office (BSMO) acquires and delivers new computer hardware and
software for the IRS’ modernized business processes.
An integrated, enterprise-wide
configuration management (CM) process is essential for ensuring that the
integrity and consistency of the IRS’ computer systems are maintained
throughout their life cycles. The
purpose of CM is to systematically identify and baseline the items that make up
a system (identification), formally control any modifications to those items
(control), report on the status of the CM process (status accounting), and
ensure that baseline configurations are implemented (audit).
Both the Treasury Inspector
General for Tax Administration (TIGTA) and the General Accounting Office (GAO)
have issued reports on the IRS’ Business Systems Modernization efforts that
commented on the MITS organization’s CM process. Our report focuses on the implementation of the CM processes
throughout the MITS organization and on selected modernization projects that
had migrated from the BSMO to support and maintenance within the ITS
organization. Our audit work in the
Office of Security Services was limited due to the anticipated restructuring of
that office as part of the realignment of the IRS’ management structure. Personnel from the Office of Security
Services indicated that they plan to place security policy documentation under
CM control; therefore, no additional fieldwork was performed in that office.
Audit
work was conducted in the MITS organization at the IRS National
Headquarters in New Carrollton, Maryland, from May through September 2003. The audit was conducted in accordance
with Government Auditing Standards. Detailed
information on the audit objective, scope, and methodology is presented in
Appendix I. Major contributors to the
report are listed in Appendix II.
MITS organization management has
recognized the need to institutionalize an enterprise-wide CM process
throughout their organization and issued a CM directive in August 2002 to
support that need. Management has taken
several specific actions to implement this directive as well as an
enterprise-wide CM process.
Specifically, the MITS organization has taken the following
actions:
·
Chartered
a MITS Configuration
Management Working Group (CMWG) to establish,
maintain, and improve CM processes, procedures, and techniques to be used
throughout the MITS organization.
·
Issued various CM standard operating
procedures (e.g., configuration control boards [CCB], configuration items [CI]
and baselines, configuration control, and configuration management process
compliance assessments).
·
Conducted CM process
compliance assessments of eight BSMO projects to evaluate whether CM policy and
procedures were being followed.
·
Chartered a MITS CCB as the
authority for receiving, reviewing, and approving proposed system change
requests and changes to system baselines that have a cost impact that exceeds
the dollar threshold and authority levels established for lower level project
CCBs. The MITS organization CCB is also
intended to be the forum to resolve conflicts such as those resulting from
request impact analysis and authority issues that occur at or among the
subordinate CCBs for individual projects.
·
Chartered project level CCBs
within the BSMO, such as the Internet Refund/Fact of Filing (IRFOF) Project and
the Infrastructure Modernization Project.
·
Issued a directive that
defined the BSMO’s authority levels (e.g., for BSMO project level CCBs) and
threshold criteria for changing BSMO project baselines for schedule, cost, or
requirements. For example, the BSMO Infrastructure Modernization Project CCB has the
authority for approving proposed change requests that affect infrastructure
modernization projects with a cost impact threshold of less than $500,000, and
those above this level would be forwarded to a higher level CCB.
·
Chartered organizational
level CCBs within ITS (e.g., for the Detroit, Martinsburg, and Tennessee
Computing Centers).
·
Established the Office of
Configuration Management (OCM) within the BSMO, whose chief, as chair of the
MITS CMWG, is responsible for establishing, maintaining, and improving CM
processes and procedures throughout the MITS organization.
The ITS organization has an
effort underway to align the existing computing center CM processes with the
Triplex Strategy. Further, the BSMO has
developed a CM training plan, developed CM training courses, and held initial
CM classes for the MITS organization.
Our review identified that the
MITS organization has made progress in implementing a CM process; however, as
explained below, further actions are needed to establish and integrate uniform
CM implementation processes across the MITS organization.
Treasury Directive 84-01, Information
Systems Life Cycle Manual, dated March 2002, requires CM to be used
throughout every project’s life cycle.
It also defines the four CM functions of identification, control, audit,
and status accounting. The IRS has
incorporated these requirements into its systems life cycle methodologies
(Enterprise Life Cycle [ELC] and ELC-Lite), the Enterprise Architecture (EA),
and the MITS organization CM Directive and procedures. The MITS organization CM Directive also
cites the American National Standards Institute/Electronic Industries Alliance
Standard 649, National Consensus Standard for Configuration Management,
an industry CM best practice.
Additionally, the Office of Management and Budget Circular A-123, Management
Accountability and Control, dated July 1995, requires that
the appropriate authority, responsibility, and accountability are defined and
delegated to accomplish the implementation of the CM process and that an
appropriate organizational structure is established to effectively carry out
these CM responsibilities.
However, the CM functions outlined
in Treasury Directive 84-01 have not been uniformly implemented within the MITS
organization. Specifically, the
following areas could be improved for each of the four required CM
functions:
Identification: The IRS identified CIs for the current
production environment that affected modernization project releases in 2002. However, not all ITS divisions
have identified and baselined the CIs for their production systems. An OCM contractor was identifying the CIs;
however, the effort was not completed because funding for the contractors was
cut in February 2003. An effective CM process requires that CIs be
identified. These items must be
identified and controlled prior to establishing system baselines for production
systems that will be affected by BSMO projects.
Control: The BSMO has
chartered project level CCBs to control changes to the BSMO project baselines
and established threshold criteria for decision-making by the project and MITS
CCBs, as well as Executive Steering Committees. However, the ITS organization has not chartered lower level CCBs,
except for the Enterprise Operations Services’ (EOS) Computing Center CCBs,
which are change management rather than CM oriented. The EOS plans to establish a CM process as part of its Triplex
initiative. An effective CM process
requires a CCB to set and control baselines.
Status
Accounting and Audit: The BSMO is performing the CM status
accounting and audit functions. The OCM
performed CM compliance assessments of several BSMO projects and identified
problems with at least one of these functions in each assessment. Although the BSMO project managers have the
responsibility to correct project specific CM process issues, the OCM does not
have the authority to ensure that the issues are corrected. Also, the ITS organization is not performing
the status accounting and audit functions because it has not finished
establishing the identification and control processes.
Without an integrated,
enterprise-wide CM process, the IRS cannot adequately assure that changes to its computer
system configurations are properly managed throughout their life cycles. An
integrated, enterprise-wide implementation of the CM process within the MITS
organization is particularly important for modernized systems that will migrate
in stages or releases to ensure computer system changes are properly managed
throughout their life cycles. In
addition, this process provides a means to document, communicate, and
coordinate system development and production CM baselines between the BSMO and
the ITS organizations.
For
example, the Enterprise Systems Management (ESM) project experienced
schedule delays and incurred additional costs because it did not have an
integrated, enterprise-wide CM process.
For the ESM project, such a process is necessary since different
organizations are responsible for developing and deploying ESM releases. Specifically, the ESM team from the IRS’
PRIME Business Systems Modernization contractor (PRIME) is responsible for
development, and the ITS’ End-User Equipment and Services (EUES) organization
is responsible for deployment.
In
February 2003, during the ESM Release 2.1 deployment, the EUES organization,
with support from the PRIME, upgraded the ESM production environment. During this upgrade, changes were introduced to the production environment
without adequate testing or adherence to CM processes. As a result of these CM weaknesses, a
database server experienced serious, unexpected performance problems. Resolution of the performance problems delayed the implementation
of ESM Release 2.1 for 4 months and increased the PRIME contractor’s cost by
approximately $216,500 since this work fell outside the scope of the PRIME
contractor’s existing task orders.
The ESM
system is just one of several modernized systems that will be migrated in stages
over the next several years. Between
Fiscal Year (FY) 2003 and FY 2005, the IRS scheduled nine other modernized
systems to migrate to the production environment. These modernized systems include new tax administration and
financial management systems. These
modernized systems not only have interdependencies with each other, but also
with the existing IRS systems.
Consequently, delays in one project can cause delays in others. For example, the delays in implementing ESM
Release 2.1 delayed full management reporting functionality for the IRFOF
system.
Several
factors contributed to the implementation deficiencies found in the MITS organization. First, the MITS organization CM Directive
and procedures did not establish executive level responsibility that would
ensure that the MITS organization CM processes are implemented throughout the
ITS organization and coordinated with the BSMO, and that deficiencies
identified in internal CM assessments are appropriately addressed. Second, governance policy has not been
established for defining authority levels and threshold criteria to review,
approve, and control production changes in the ITS organization or for
elevating change requests from the ITS organization to a higher-level CCB or
committee for approval or coordination such as that found in the BSMO
organization.
In addition, some ITS
organizations use different
CM software that does not comply with the current EA and may not readily facilitate the coordination of system
baseline information on an enterprise-wide CM level. The current revision of the IRS’ EA, dated July 9, 2003, includes
an Enterprise Standards Profile that identifies commercial off-the-shelf
software products for use in various computer environments. However, prior EA versions and the MITS
organization CM policies and procedures did not identify approved CM software
for enterprise-wide use. As a result,
some ITS organizations use CM software that does not comply with the current
EA. The use of non-compliant software
to automate the CM process places the ITS organizations at risk of not being
able to effectively communicate and coordinate changes on the production
environment with affected organizations and projects, such as BSMO projects
that have production environment interdependencies. Since a variety of CM software is currently being used, a period
of transition will be needed to review and establish the CM software to be used
and integrated throughout the MITS organization.
For example, a contributing cause
for not adhering to CM processes for the ESM project was the use of different automated software to
manage changes for the project. The
BSMO and the PRIME use IBM’s Rationalâ software for configuration
management, and the EUES organization uses IRS developed software to control
change requests.
Without
an integrated and uniform CM process, there is an increased potential that
modernized and existing systems will require extensive rework resulting in
additional costs, schedule delays, and other
risks to the IRS’ computer operations (e.g., system outages and data
corruption).
To promote the establishment of an
integrated MITS organization CM process, we recommend that the Chief
Information Officer:
1.
Modify the MITS organization
CM Directive and procedures to: a)
assign organizational responsibility for ensuring that MITS organization CM
processes are implemented throughout the ITS organization and coordinated with
the BSMO and that CM deficiencies are appropriately addressed; and b) establish
governance policies, similar to those used by the BSMO, for defining the
authority levels and threshold criteria to approve and control changes to the
production environment in the ITS organization.
Management’s Response: The MITS organization will revalidate its CM Directive to address organizational responsibility, governance policy, and needed improvements in the CCB structure. In addition, management will address the establishment of governance policies and threshold criteria to approve and control changes to the ITS production environment while establishing plans to organize separate CCBs for the ITS and BSMO organizations.
2.
Develop a transition plan to
implement standardized EA-compliant CM software to be used throughout the MITS
organization to facilitate CM on an enterprise-wide level.
Management’s Response: The MITS organization will identify acceptable CM software and publish applicable guidance upon the completion of ongoing CM software assessments within the ITS organization.
Appendix I
Detailed Objective,
Scope, and Methodology
The overall
objective of this review was to determine whether the Modernization and
Information Technology Services (MITS) organization effectively implemented an enterprise-wide
configuration management (CM) process.
As part of
this review, we interviewed personnel and reviewed CM documentation throughout
the MITS organizations of Business Systems Modernization Office (BSMO),
Information Technology Services (ITS), Business Planning and Assurance, and
Security Services. Within the BSMO, we
interviewed personnel from the Office of Configuration Management and Systems
Engineering & Integration Division as well as the project teams for the
Enterprise Systems Management (ESM) and Internet Refund/Fact of Filing (IRFOF)
projects. Within the ITS organization,
we interviewed personnel from the Infrastructure Architecture and Engineering,
Business Systems Development, Enterprise Operations Services, End User
Equipment and Services, Enterprise Networks, and Web Services functions.
This
audit assessed the CM processes throughout the MITS organization and selected projects
that migrated from acquisition by the BSMO to support and maintenance by the
ITS organization. The ESM and IRFOF projects were judgmentally
selected from the population of BSMO projects based on one project having been
fully migrated and another being migrated in stages or releases to the
production environment. The IRFOF
Project was selected as a project that had migrated from the BSMO environment
to the production environment, which is operated and supported by the ITS
organization. The ESM project was
selected as a project that migrated in stages or releases since a release was
already being supported in the ITS’ production environment and future releases
were being developed by the IRS’ PRIME Business Systems Modernization
contractor that is overseen by the BSMO.
To accomplish
the overall objective for this audit, we:
|
I. |
Identified applicable Federal Government standards and
industry best practices that guide the CM process. This included Department of the Treasury directives, Office of
Management and Budget circulars, and information technology standards
organization documents. |
|
II. |
Evaluated the ITS Executive and the BSMO Office of
Configuration Management roles and responsibilities for administering the
enterprise-wide CM process. |
|
III. |
Evaluated the policies and procedures supporting the
enterprise-wide CM process. |
Appendix II
Major Contributors to This Report
Margaret E. Begg,
Assistant Inspector General for Audit (Information Systems Programs)
Gary V. Hinkle, Director
Theodore Grolimund,
Audit Manager
Kevin Burke, Senior
Auditor
Christopher Funke,
Senior Auditor
Frank Greene, Senior
Auditor
Michael Howard,
Senior Auditor
Tina Wong, Senior
Auditor
Olivia Jasper,
Auditor
Appendix III
Commissioner C
Office of the
Commissioner – Attn: Chief of
Staff C
Deputy Commissioner for Operations Support OS
Associate Commissioner for Modernization OS:CIO:B
Chief, Information Technology Services OS:CIO:I
Chief,
Mission Assurance OS:MA
Director,
Business Systems Development
OS:CIO:I:BSD
Acting Director, End User
Equipment and Services OS:CIO:I:EU
Director, Enterprise
Networks OS:CIO:I:EN
Director, Enterprise Operations OS:CIO:I:EO
Director, Infrastructure,
Architecture and Engineering
OS:CIO:I:IA
Director,
Portfolio Management OS:CIO:R:PM
Director, Web Services OS:CIO:I:W
Manager, Enterprise Systems
Management OS:CIO:I:EU:ESM
Manager, Office of Configuration Management OS:CIO:B:MP:CM
Chief Counsel CC
National Taxpayer Advocate TA
Director, Office of Legislative Affairs CL:LA
Director, Office of
Program Evaluation and Risk Analysis
RAS:O
Office of
Management Controls OS:CFO:AR:M
Audit Liaisons:
Associate Commissioner
for Modernization OS:CIO:B
Chief, Information
Technology Services OS:CIO:I
Chief, Mission
Assurance OS:MA
Director, Business
Systems Development OS:CIO:BSD
Director, End User
Equipment and Services OS:CIO:I:EU
Director, Enterprise
Networks OS:CIO:I:EN
Director, Enterprise
Operations Services OS:CIO:I:EOS
Director,
Infrastructure, Architecture and Engineering
OS:CIO:I:IA
Director, Web
Services OS:CIO:I:W
Program Manager, Program Oversight and Coordination OS:CIO:R:PM:PO
Appendix IV
This appendix presents detailed
information on the measurable impact that our recommended corrective actions
will have on tax administration. This
benefit will be incorporated into our Semiannual Report to the Congress.
Type and Value of Outcome
Measure:
·
Inefficient use of resources
– Actual; $216,500 (see page 4).
Methodology Used to Measure the
Reported Benefit:
Not
having an integrated, enterprise-wide Configuration Management (CM) process was
demonstrated by the schedule delays and increased costs of the Enterprise
Systems Management (ESM) project. The
ESM Release 2.1 was developed by the Internal Revenue Service’s (IRS) PRIME
Business Systems Modernization contractor (PRIME) and deployed by the
Information Technology Services’ End-User Equipment and Services (EUES)
organization. In February 2003, during
the ESM 2.1 deployment, the EUES organization, with support from the PRIME,
upgraded an ESM production environment.
During
this upgrade, changes were introduced to the
production environment without adequate testing or adherence to CM processes
and, as a result, a database server experienced serious, unexpected performance
problems. A contributing cause for not adhering to CM processes was
the use of different
automated software to manage changes for the ESM project. The Business Systems Modernization Office
and the PRIME use IBM’s Rationalâ software for configuration
management and the EUES organization uses IRS-developed software to control
change requests. Resolution of the
performance problems delayed the implementation of ESM Release 2.1 for 4 months
and increased the PRIME contractor’s cost by approximately $216,500 since this
work fell outside the scope of their existing task orders. This cost was documented in a notification
of change letter to the PRIME contract, sent to the IRS on April 11, 2003.
Appendix V
Overview of
Configuration Management Functions
The chart was removed due to its size. To see the chart, please go to the Adobe PDF
version of the report on the TIGTA Public Web Page.
Appendix VI
Management’s Response
to the Draft Report
The response was removed due to its
size. To see the response, please go to
the Adobe PDF version of the report on the TIGTA Public Web Page.