OIG/98A-07 - Review of NRC's Controls Over the PC Refresh Program
October 9, 1998
| MEMORANDUM TO: |
L. Joseph Callan Anthony J. Galante |
| FROM: | Thomas J. Barchi Assistant Inspector General for Audits |
| SUBJECT: | REVIEW OF NRC'S CONTROLS OVER THE PC REFRESH PROGRAM |
Attached is the Office of the Inspector General's audit report entitled "Review of NRC's Controls Over the PC Refresh Program."
On August 7, 1998, we provided a draft of this report to Anthony J. Galante, Chief Information Officer (CIO) and Edward L. Halman, Director, Office of Administration (ADM). On September 3 and 10, 1998, respectively, the CIO and the Director, ADM responded to our draft report.
The CIO questioned the basis for our conclusions and some of the report's factual content. However, he agreed that a Project Management Plan should provide the details needed to manage projects effectively. We believe that when implemented, such Project Management Plans will address the intent of our recommendations. We have responded to the CIO's concerns about the report's factual content and conclusions in the Agency Comments section of our report.
The Director, ADM, agreed with our recommendations concerning the Property Accounting System.
Please contact me on 415-5915 if we can assist you further in this matter.
Attachment: As stated
- Report Synopsis
- Introduction
- Background
- Findings
- NRC Continues to Lack Adequate Management Information for Overseeing IT Projects
- The NT Installation Plan Seems Inefficient and Costly
- Conclusion
- Recommendations
- Agency Comments
- Appendices
Report Synopsis
The Office of the Inspector General (OIG) reviewed the management controls over the U.S. Nuclear Regulatory Commission's (NRC) PC Refresh program. A Capital Planning and Investment Control document guides the program, which calls for replacing about 4,000 personal computers (PCs) with Pentium-class systems by the end of fiscal year 1999, at an approximate cost of $7.9 million.
While this report focuses on the Office of the Chief Information Officer's (OCIO) management of the PC Refresh program, OIG has long been concerned with the need for adequate management controls over information technology (IT) activities at NRC. OIG has previously reported on several issues related to the control over IT projects. The concerns that we found with the PC Refresh program are similar to management control shortcomings noted in several previous reports covering IT activities.
In addition, we reviewed OCIO's plan for upgrading the agency's operating system to Microsoft Windows NT (NT). OCIO advised that they will place Pentiums on desktops in headquarters and then upgrade the machines with NT later. However, OCIO's plan for regional and remote resident inspector sites is to install some Pentiums already upgraded with NT. We believe that this one-step approach will result in fewer disruptions to NRC staff and lower agency costs.
Our report makes three recommendations to address the issues identified.
Introduction
We reviewed the adequacy of management controls for the U.S. Nuclear Regulatory Commission's (NRC) PC Refresh program, which is managed by the Office of the Chief Information Officer (OCIO). Under this program, the OCIO acquires, configures, installs, and maintains new agency personal computers (PCs). Appendix I contains additional information on our objectives, scope, and methodology.
Background
While this report focuses on OCIO's management of the PC Refresh program, the Office of the Inspector General (OIG) has long been concerned with the need for adequate management controls over information technology (IT) activities at NRC. OIG has previously reported on shortcomings in the agency's controls over IT projects and contracts.
In a March 1993 report,(1) OIG found that the Office of Information Resources Management (IRM)(2) had not exercised adequate control over IT contracts and the activities of project officers. We reported that IRM had not established agency-wide policies and procedures needed to ensure that IT functions are performed consistently and in accordance with applicable regulations. As a result, IRM exceeded specific procurement limits and made unauthorized commitments. Later, in April 1996, we conducted a special evaluation(3) of a major agency IT contract and identified several lessons the agency needed to address to effectively develop systems and oversee IT contracts. We concluded that IRM's management control processes did not work as intended. In a September 1996 report,(4) OIG again reported that NRC lacked the controls to systematically provide NRC management with the information needed to assess the status of IRM projects. To its credit, NRC has since taken several positive steps, including the appointment of a Chief Information Officer (CIO) on February 2, 1997, and the creation of OCIO. In addition, to improve "front-end" IT planning, OCIO developed and implemented the Capital Planning and Investment Control (CPIC) process.
Currently, OCIO is in the midst of an agency-wide upgrade of its PCs. Called PC Refresh, this effort is projected to cost about $7.9 million and is guided by the CPIC process. The PC Refresh program calls for upgrading approximately 93 percent of NRC's PCs with Pentium-class systems as follows: replace 40 percent in fiscal year (FY) 1997, 26 percent in FY 1998, and 27 percent in FY 1999.
As OCIO continues to refresh agency desktop PCs with Pentiums, it is also beginning to upgrade the agency's existing operating system to Microsoft Windows NT (NT) using a two-step process. Its plan is to first place Pentiums on agency desktops and then upgrade the operating system with NT at a later date.(5)
Findings
Similar to prior OIG findings, our review of the PC Refresh program disclosed that NRC continues to lack adequate management information for effectively overseeing IT projects. In addition, the planned two-step approach for installing the NT operating system appears inefficient and may increase costs. These findings are discussed in the following sections.
NRC Continues to Lack Adequate Management Information for Overseeing IT Projects
NRC has taken a number of steps to strengthen oversight of information resources management projects. However, OIG found that the PC Refresh program suffers from many of the same deficiencies that plagued previous IT projects we reviewed. While PC Refresh relies on CPIC guidance to provide estimates and budget information as baseline data for IT project management, we found that fundamental day-to-day information normally associated with overseeing a project's costs and status was lacking.
As OIG previously reported,(6) IRM's (and now OCIO's) management of project development focuses on achieving technical performance goals. Although we agree that successfully achieving technical performance is important, so is the need to effectively manage project costs and schedules and re-validate planned requirements. This aspect of project management is especially important given the agency's increasingly scarce resources.
To assess PC Refresh requirements, milestones, and project status, we asked OCIO staff to provide information about the number of Pentiums needed, purchased, and installed. The OCIO staff could not readily provide this information. They told us that obtaining the number of Pentiums installed would be a labor intensive process. Furthermore, OCIO staff told us that they use the CPIC and a draft installation schedule to assess budget information, schedule installations, and assess progress. The CPIC calls for replacing about 4,000 PCs based on assumptions made in an April 1997 benefit-cost analysis. This estimate includes one PC for each NRC staff member, as well as PCs for non-staff requirements such as the Technical Training Center and special projects. In June 1998, OCIO staff told us that 3,230 PCs had been refreshed with Pentiums. However, the continued need to purchase 4,000 PCs has not been validated.
At the conclusion of our fieldwork, OCIO staff informed us that they would conduct another review in August 1998 to firm up the number of Pentiums actually needed. Without such a validation, the agency could potentially purchase more Pentiums than needed. For example, since the CPIC estimate was developed, reflecting a staff requirement for 3,300 Pentiums, agency staffing levels have declined. The full-time equivalent ceiling for FY 1998 is only 2,985 and is projected to drop in FY 1999. Therefore, if the CPIC includes more PCs than needed, OCIO now has the opportunity to potentially achieve significant savings by purchasing less PCs. At a cost of $1,258 each, every 100 computers purchased that are not needed cost the agency $125,800.
Prior to learning that OCIO staff planned to validate PC Refresh requirements, we asked for the rationale upon which they based the need for about 4,000 computers. OCIO staff provided us with a list showing a need for 812 computers beyond staff requirements. They told us that there are a variety of uses for additional PCs such as training, special task forces, and contractor needs. OCIO staff acknowledged that their list of 812 PCs was only an estimate because they were unable to access the Office of Administration's (ADM) Property Accounting System (PAS) to provide more accurate information. However, OCIO staff had previously advised us that PAS data was inaccurate and could not be relied upon. Our own independent tests confirmed that the PC data in PAS was not reliable (see Appendix III).
Also, OCIO uses information supplied by IT coordinators to schedule PC upgrades. IT coordinators serve as the liaison between office staff and OCIO to ensure that IT services requests are completed. Furthermore, IT coordinators are supposed to have full responsibility for overseeing IT equipment assigned to their office property accounts. However, we found that IT coordinators use various methods, including PAS, to track PC inventory for their offices. OCIO staff acknowledged that the information received from IT coordinators is not always accurate.
Nevertheless, OCIO uses information supplied by IT coordinators to schedule PC upgrades. As a result, Applied Management Systems, Inc. (AMS)(7) contractor technicians are not always able to perform scheduled upgrades. PC tag numbers at agency workstations do not always match the information listed on the technicians' work orders. Our tests of the accuracy of PC location data in PAS confirmed these problems and disclosed an error rate of almost 50 percent. A representative from AMS confirmed that scheduling a PC to be refreshed is a significant problem for the technicians.
Each time a refresh cannot be completed, AMS charges the agency for the site visit time. An OCIO staff member told us that the contractor technicians annotate their work orders each time they are unable to do the work. However, OCIO staff admitted that they have no reliable method for tracking each missed opportunity. Therefore, OCIO does not know how many times this has occurred or the cost to the agency when AMS technicians cannot perform scheduled refreshes.
During the course of our audit, OCIO issued a draft revision to the CPIC process requiring the preparation of a Project Management Plan as part of the CPIC process. This guidance underscores the importance of timely and accurate management information concerning the cost, schedule, and performance of projects. However, the guidance does not detail the project management data needed, such as, when it is needed or how it is to be used. We believe it should include this critical information. As the July 1998 OCIO presentation, IT for Managers and Supervisors, notes, a "successful" project is: on time, within budget, meets requirements, and is accepted by users.
The NT Installation Plan Seems Inefficient and Costly
OCIO was in the midst of refreshing agency PCs with Pentiums when they decided to use Microsoft Windows NT 4.0 as the agency's standard operating system. The PC Refresh CPIC implies that NRC would save labor costs by providing Pentiums with the NT operating system already installed. However, OCIO staff decided that they would continue to install Pentiums with the old operating system. OCIO's planned approach is to revisit each work station and install NT rather than use a one-step process that is potentially less costly and disruptive. We were told by OCIO staff that installing a new Pentium upgraded with NT would be technically complex.
A two-step process results in repeat visits to agency workstations by contractor technicians and ultimately the agency will incur additional costs. OCIO's installation contractor told us that the NRC could save about ½ to 1 hour per installation and an extra service call by doing the PC refresh and NT installation in one step. In addition, agency personnel will need to be at their workstations when contractor technicians come back to install NT. Multiple contractor visits to agency workstations increase disruptions to NRC staff.
At the conclusion of our fieldwork, OCIO staff advised us that they plan to use a one-step process for region and resident inspector sites by installing new Pentiums already upgraded with NT at these locations. The rationale behind this decision was that it would reduce NRC travel costs. OIG inquired about OCIO's rationale that installing a new machine upgraded with NT is too complex. OCIO staff responded that to overcome the technical complexity a higher skilled technician would perform such installations at remote sites. We believe this approach will be more cost effective by negating the need for multiple visits. We note that the contractor for the Office for Analysis and Evaluation of Operational Data is also using the one-step process in the headquarters Operations Center.
As is being done in the regions, resident inspector sites, and the headquarters Operations Center, we believe that it will be more cost effective and efficient for OCIO to install new Pentiums already upgraded with NT in headquarters offices. Further, it will lessen the disruptions to the NRC staff that rely on PCs to do their jobs.
Conclusion
Although PC Refresh may meet its implementation schedule, overall project management controls are weak. During our review, OCIO staff could not readily provide reliable and accurate project management information such as the number of PCs purchased, installations completed, and Pentiums still needed. By the conclusion of our review, OCIO staff advised that they had compiled some of this information. We believe that effective project management requires on-going, up-to-date management information and it should not be labor intensive to obtain. Given the government's goal of obtaining measurable performance results, it would seem that such management information would be an effective tool in assessing how well managers are ensuring that programs are efficient, economical, and effective.
Given the NRC's need to closely manage its increasingly scarce resources, accurate requirements and reliable cost information are becoming more critical. Also, because of the impact of information technology on the agency's operations, it is important that implementation projects such as PC Refresh and NT installations be efficiently managed to ensure minimal staff disruptions.
Because PC Refresh program shortcomings are similar to those previously reported regarding inadequate management controls over IT activities, we plan to schedule and do follow-up audit work on the agency's implementation of corrective actions identified in this and previous OIG audits.
Recommendations
To ensure that PC Refresh and future OCIO projects are efficiently and effectively managed, we recommend that the CIO:
-
Require that Project Management Plans specify the management information and reporting requirements needed by managers to oversee a project through its life cycle.
- Reconsider the planned two-step NT installation plan for the remainder of headquarters with a focus on minimizing cost and disruption.
If PAS is to be used for determining agency inventory, we recommend that the Director, ADM:
- Develop a quality assurance process to periodically test PAS information to improve its accuracy and reliability.
Agency Comments
On September 3 and 10, 1998 respectively, the CIO and the Director, ADM, responded to our draft report. We have included both responses in Appendix II of this report.
The CIO questioned the basis for our conclusions and some of the report's factual content. However, he agreed that a Project Management Plan should provide the details needed to manage projects effectively. We believe that when implemented, such Project Management Plans will address the intent of our recommendations.
The Director, ADM, agreed with our recommendations concerning the Property Accounting System.
As with all audits, OIG relied on the information provided by the OCIO staff and, where possible, OIG auditors attempted to validate the information. Throughout the course of this audit, basic information concerning the current status of cost, schedule, and performance of the PC Refresh Program and the installation of the NT software was not available and/or changed. OIG auditors found it difficult, at best, to obtain reliable data upon which to test the status of the effort. OCIO staff continually referred us to the PC Refresh CPIC document as the criteria for cost and schedule information.
Furthermore, when OCIO staff made information available, OIG found it was, at times, unreliable. For example, our concern about OCIO's reliance on PAS information to identify the location of computers prompted us to test PAS. As we reported, we found an error rate of 49.3 percent. We also sought to validate how PC Refresh information was developed through other sources that the CIO cited, such as IT coordinators. We found, however, the potential for a high degree of inconsistency. Most IT coordinators did not have a clear understanding of their respective roles or responsibilities, including their roles in reporting this information.
Our attempts to validate the strategy for the NT installations was confusing because OCIO staff continued to present different strategies. During the course of the audit, the Director, Information Technology Infrastructure Division, OCIO, led us to believe that a two-step installation would be followed except for the PCs at resident inspector sites. (A two-step installation entails first the installation of the hardware [Pentium desktop computers] and then, at a later date, the NT installation). He reconfirmed this strategy at our exit briefing of June 9, 1998. On June 10, 1998, the Director advised that the regions would be refreshed under a one-step process, simultaneously receiving Pentiums upgraded with Windows NT. He stated this decision was made during the OCIO budget preparation process in March or April. Subsequently, the CIO's response to our draft report provides yet another strategy.
As we all agree, the need for accurate and timely information is critical to manage any project, especially one that has the potential to disrupt NRC employees' productivity. OIG has expressed project management concerns in previous reports relating to IT projects, and the concern remains today. The CIO acknowledges this need and states, "Improvement in readily accessing this data is needed and will be addressed by OCIO." We agree with his observation that basic information concerning the process was not "effectively communicated" to OIG staff. He also agrees that OCIO staff should "improve its ability to quickly summarize data from its record and tracking systems, and we will do so." Our experience has shown that this ability is characteristic of effective project management.
In his concluding comments, the CIO projects the successful implementation of the project, within budgeted cost. Given the uncertainty of the data that OCIO staff provided, we cannot validate this projection. We look forward to the CIO's implementation of our recommendation that requires this information as an integral part of project management. As stated in the body of our report, we plan to schedule and do follow-up audit work on the agency's implementation of corrective actions identified in this and previous OIG audits.
Appendices
I. Objectives, Scope, and Methodology
The objectives of our audit were to (1) identify the management controls associated with the PC Refresh program and (2) determine if those controls are effective. The scope of our audit was limited to the Office of the Chief Information Officer's (OCIO) upgrading of agency information technology (IT) inventory in accordance with the PC Refresh program. The audit focused on OCIO's management controls over the program for refreshing about 4,000 agency desktop personal computers (PCs) with Pentium-class systems and upgrading the agency's operating system with Microsoft Windows NT 4.0.
We reviewed OCIO's management controls and the effectiveness of those controls related to the PC Refresh program. We interviewed personnel in OCIO, the Office of Administration (ADM), and the agency's IT coordinators. In addition, we reviewed the communication process between OCIO, ADM, and other agency offices to determine if the project is receiving appropriate support and involvement. We also conducted a statistical sample activity to determine if agency computers are properly accounted for in ADM's inventory control system, the Property Accounting System. Furthermore, we reviewed previous Office of the Inspector General reports related to the control over IT projects and contracts.
To evaluate if PC Refresh will meet the schedule to support new agency applications such as ADAMS, RPS, and STARFIRE, we reviewed NRC guidance and conducted interviews with personnel in OCIO, ADM, and the agency's IT coordinators. We also reviewed the PC Refresh program's approved plan (the Capital Planning and Investment Control document) to identify the project's milestones and budget. We did this to evaluate OCIO's internal control system over the PC Refresh program.
Our audit was conducted from March 1998 to June 1998 in accordance with generally accepted Government auditing standards.
II. Statistical Sample Review of the Office of Administration's Property Accounting System (PAS)
The Office of Administration's Property Accounting System (PAS) is the official tracking system for NRC's property inventory. Agency property custodians have had access to their property accounts through PAS since June 1997 and are responsible for updating the system to keep their accounts current.
To determine the accuracy of personal computers (PCs) listed in PAS, the Office of the Inspector General selected a statistical sample to determine if PAS correctly accounts for agency computers. Specifically, we selected a systematic random sample of 231 PCs from a universe size of 3,979 PCs. We used the following guidelines to set errors for this review: 1) NRC tag number and/or room number do not match the PAS report; 2) no PC at the location cited on the PAS report; or, 3) the location (room number) cited on the PAS report cannot be found.
Following is a compilation of the findings from our review of PCs at NRC's headquarters offices:
| Building | Wrong Tag # | No PC | No Location | Total Errors |
|---|---|---|---|---|
| OWFN | 11 | 3 | 8 | 22 |
| TWFN | 24 | 67 | 1 | 92 |
| TOTALS: | 35 | 70 | 9 | 114 |
Of the 231 PCs that we examined, PAS contained 114 or 49.3 percent errors. Therefore, with a 95 percent confidence rate, we can estimate, relative to a population of 3,979 PCs, that the locations of between 1,707 and 2,220 PCs are in error.
III. Abbreviations and Acronyms
| ADAMS | Agencywide Document Access and Management System |
| ADM | Office of Administration |
| AMS | Applied Management Systems, Inc. |
| CIO | Chief Information Officer |
| CPIC | Capital Planning and Investment Control |
| FY | fiscal year |
| IRM | Office of Information Resources Management |
| IT | information technology |
| NT | Microsoft Windows NT 4.0 |
| NRC | U.S. Nuclear Regulatory Commission |
| OCIO | Office of the Chief Information Officer |
| OIG | Office of the Inspector General |
| PAS | Property Accounting System |
| PC | personal computer |
| Pentium | Pentium-class system |
| RPS | Reactor Program System |
| STARFIRE | Agencywide Financial & Resource Management System |
IV. Major Contributors to this Report
Anthony C. Lipuma
Team Leader
Cheryl A. Miotla
Management Analyst
V. Glossary: Office of the Inspector General Products
Investigative
1. INVESTIGATIVE REPORT - WHITE COVER
An Investigative Report documents pertinent facts of a case and describes available evidence relevant to allegations against individuals, including aspects of an allegation not substantiated. Investigative reports do not recommend disciplinary action against individual employees. Investigative reports are sensitive documents and contain information subject to the Privacy Act restrictions. Reports are given to officials and managers who have a need to know in order to properly determine whether administrative action is warranted. The agency is expected to advise the OIG within 90 days of receiving the investigative report as to what disciplinary or other action has been taken in response to investigative report findings.
2. EVENT INQUIRY - GREEN COVER
The Event Inquiry is an investigative product that documents the examination of events or agency actions that do not focus specifically on individual misconduct. These reports identify institutional weaknesses that led to or allowed a problem to occur. The agency is requested to advise the OIG of managerial initiatives taken in response to issues identified in these reports but tracking its recommendations is not required.
3. MANAGEMENT IMPLICATIONS REPORT (MIR) - MEMORANDUM
MIRs provide a "ROOT CAUSE" analysis sufficient for managers to facilitate correction of problems and to avoid similar issues in the future. Agency tracking of recommendations is not required.
Audit
4. AUDIT REPORT - BLUE COVER
An Audit Report is the documentation of the review, recommendations, and findings resulting from an objective assessment of a program, function, or activity. Audits follow a defined procedure that allows for agency review and comment on draft audit reports. The audit results are also reported in the OIG's "Semiannual Report" to the Congress. Tracking of audit report recommendations and agency response is required.
5. SPECIAL EVALUATION REPORT - BURGUNDY COVER
A Special Evaluation Report documents the results of short-term, limited assessments. It provides an initial, quick response to a question or issue, and data to determine whether an in-depth independent audit should be planned. Agency tracking of recommendations is not required.
Regulatory
6. REGULATORY COMMENTARY - BROWN COVER
Regulatory Commentary is the review of existing and proposed legislation, regulations, and policies so as to assist the agency in preventing and detecting fraud, waste, and abuse in programs and operations. Commentaries cite the IG Act as authority for the review, state the specific law, regulation or policy examined, pertinent background information considered and identifies OIG concerns, observations, and objections. Significant observations regarding action or inaction by the agency are reported in the OIG Semiannual Report to Congress. Each report indicates whether a response is required.
1. Review of IRM's Management of Its Contracts, OIG/92A-10, March 8, 1993.
2. The Chief Information Officer restructured IRM and created OCIO in 1997.
3. Selecting, Managing, and Utilizing the M-Cubed Contract, OIG/96E-13, April 17, 1996.
4. Improvements Needed in Agency Oversight of Information Resources Management Activities, OIG/96A-11, September 24, 1996.
5. The information presented in this report reflects the methodology and project status presented to us during our audit field work. After the audit was concluded, OCIO presented different scenarios for refreshing PCs and installing Windows NT. These scenarios concern the methodology for using a one or two-step process for PC Refresh and NT. The effect of this changing methodology on our conclusions is addressed in the Agency Comments section of the report.
6. Improvements Needed in Agency Oversight of Information Resources Management Activities, OIG/96A-11, September 24, 1996, pp 3-4.
7. Applied Management Systems, Inc. is the OCIO contractor providing NRC with microcomputer support services which include upgrading agency PCs under the PC Refresh program.
