| |
Search Options | |||
| Index | Site Map | FAQ | Facility Info | Reading Rm | New | Help | Glossary | Contact Us | |
|||
June 29, 2000
The U.S. Nuclear Regulatory Commission (NRC) began planning for the Standard Financial and Integrated Resource Enterprise (STARFIRE) system in early 1997, with a vision of it serving as the single, authoritative source of financial and resource information for NRC. Three offices-the Office of the Chief Financial Officer, the Office of the Chief Information Officer, and the Office of Human Resources-are directly involved in this effort. According to STARFIRE planning documents, the system would consist of 10 modules, be operational by October 1999, and cost approximately $6.9 million to implement.
Work on the system's Core Accounting Module (Core), considered the system's "cornerstone," was terminated in July 1999, after the contractor missed several delivery dates. Pursuit of a Core has been delayed until Fiscal Year 2002 or later, and NRC is currently working on implementation of the Basic Human Resources, Payroll, Time and Labor, and Cost Accounting modules. By the end of March 2000, NRC had spent approximately $6.7 million on this project, awarded another $2 million to contractors for work on STARFIRE, and had expected to implement three of the modules for parallel testing.1 (During parallel testing, data is entered into both the new modules and the existing NRC payroll system to determine if both systems produce consistent results.) The Office of the Inspector General's review focused on whether NRC has a sound methodology in place for developing and implementing an integrated financial management system that meets expectations, at cost, and within expected time frames.
We found that (1) the implementation of STARFIRE, as envisioned, is uncertain, (2) STARFIRE managers have not identified the overall cost to NRC or updated key planning documents, and (3) the STARFIRE project bears similarities to previous agency information technology development efforts. Our report makes six recommendations to address the issues identified.
The Office of the Inspector General (OIG) initiated a review of the development and implementation of STARFIRE,2 the U.S. Nuclear Regulatory Commission's (NRC) planned integrated financial management system, following the agency's termination of a contract to implement the system's Core Accounting Module (Core). NRC postponed implementation of this Core, described in a key planning document as the "cornerstone of the STARFIRE system," until Fiscal Year (FY) 2002 or later and to proceed with implementation of other system modules. Given the size of the STARFIRE project, its characterization as a relatively high risk project, and NRC's past difficulties managing large-scale information technology (IT) projects, we were concerned about how the agency was planning, managing, and overseeing this project.
NRC has stated that STARFIRE will bring the agency into compliance with the Government-wide requirement that agencies employ managerial cost accounting as a fundamental part of their financial management systems. For both FY 1998 and FY 1999, OIG identified the absence of a managerial cost accounting process as a material weakness, relative to the Federal Managers' Financial Integrity Act. The fact that NRC plans to have the components of STARFIRE in place to address this weakness by the end of September 2000 gave us further impetus to conduct this review.
Our objective was to determine whether NRC has a sound methodology for implementing an integrated financial management system that meets expectations, at cost, and within expected time frames. Appendix I contains additional information on our objective, scope, and methodology.
STARFIRE is the name given to NRC's vision of a fully integrated, agency-wide financial and resource management system. According to NRC's STARFIRE website,3 the system is expected to comprise 10 modules (i.e., commercial off-the-shelf [COTS] software customized by NRC) plus an executive information system and data warehouse. The 10 modules anticipated are Core Accounting, Cost Accounting, Time and Labor, Payroll, Human Resources, Budget Formulation, Travel Management, Property, Procurement, and Performance Measurement. When completed, NRC expects the system to serve as the single, authoritative source of financial and resource information.
NRC initiated STARFIRE planning in early 1997. At that time, former NRC Chairman Shirley Jackson requested that NRC articulate a plan to develop an agency-wide financial management system that would be operational within the next 2 years. By August 1997, NRC had developed a document identifying functional requirements for the STARFIRE components. By December 1997, the agency had produced a Capital Planning and Investment Control (CPIC) document (i.e., benefit-cost analysis) on STARFIRE and a Project Action Plan describing the management and technical approach, schedule, organizational commitment, and risks associated with the project.
Together, these three documents composed the business case, or justification, for developing the STARFIRE project, characterized in the business case as having a relatively high likelihood that it could "fail" because of the "broad scope of the undertaking." According to the business case documents, STARFIRE (inclusive of 10 modules) would be totally operational by October 1999 at an estimated cost of $6.9 million.4 The documents also specified that in developing the project, NRC would, as required, follow an approach consistent with the agency's Systems Development Life-Cycle Management (SDLCM) methodology.
The STARFIRE Project Charter, dated December 1997, designated the agency's Chief Financial Officer (CFO) to serve as STARFIRE's Executive Sponsor. The Executive Sponsor is the chief decision-maker and person with overall responsibility for the project. The Charter also designated an Office of the Chief Financial Officer (OCFO) staff member to serve as the STARFIRE Project Manager, responsible for planning, controlling, and coordinating the project, as well as for project schedule, cost, and quality. The Project Manager also serves as the Business Team Project Manager. In this capacity, the Project Manager is responsible for reporting progress to the Executive Sponsor and recommending whether or not to proceed at critical milestones for the project. Finally, the Charter designated an Office of the Chief Information Officer (OCIO) staff member to serve as the Technical Team Project Manager, reporting to the STARFIRE Project Manager.
In February 1998, Chairman Jackson approved the Executive Director for Operations' request to proceed with the STARFIRE project and authorized the Contracting Officer to enter into a contractual agreement exceeding $3 million. In that same month, NRC awarded a purchase order5 to ICF Information Technology, Inc. (ICF), for the acquisition and installation of five of the STARFIRE modules. Plans were for ICF to implement the Core Accounting Module first. ICF's total estimated cost to NRC was $1.46 million for the base year (and $3.4 million for a full 5-year effort, including maintenance, training, and optional requirements). Initially, ICF had a February 1999 deadline for producing a fully functional Core Accounting Module. When the contractor missed this deadline, NRC gave two subsequent extensions, which were also missed. On July 23, 1999, after the contractor failed to deliver a fully functional Core by a June 1999 deadline (established in an amendment to the purchase order), NRC terminated the purchase order for default. A settlement was reached and, ultimately, NRC agreed to a termination for convenience and to pay ICF $450,000. As part of the settlement, NRC received the software for the Travel Management Module.
After the termination, NRC senior management decided to postpone further pursuit of a Core Accounting Module for 2 to 3 years because they felt that, at that future time, available core accounting software would be more mature. This differs from the business case documents, which describe an approach for implementing the Core first because all of the remaining modules must interface with this "cornerstone." However, OCIO and OCFO staff involved with the project told us they are confident that they can bring in the Core at a later date and that the necessary integration will be achievable.
In the meantime, NRC had moved ahead with other portions of STARFIRE. In September 1998, NRC accepted a proposal from Andersen Consulting LLP (Andersen) to implement three modules (Time and Labor, Payroll, and Human Resources6) which the agency had purchased from a different company for STARFIRE. The initial purchase order, for $2.14 million, called for Andersen to implement the three modules by September 30, 1999. There have been four amendments to this original agreement. Together, these amendments extended the implementation deadline for the three modules to January 31, 2000, and increased the original dollar amount by nearly 100 percent, raising the total to $4.02 million. A portion of the increase was related to Full HRIS and, therefore, not attributable to STARFIRE. By March 2000, the contractor had expended all funds, and OCFO requested that an additional $1.05 million be budgeted for the Andersen work. The agency's Division of Contracts and Property Management (DCPM) considered this increase "significant" enough to warrant the Chairman's notification and approval. Pending the Chairman's review, a purchase order for $150,000 was issued so Andersen could continue to work on STARFIRE. This purchase order also extended the implementation deadline to October 2000. Subsequently, the Chairman approved OCFO's request for $1.05 million for Andersen to continue work to implement the three STARFIRE modules. Ultimately, a purchase order was issued for $574,000 instead of the higher figure.7 Also, in March 2000, NRC awarded a contract to Price Waterhouse Coopers, LLP, in the amount of $1.3 million to implement the Cost Accounting Module.8 In examining the status of the full STARFIRE project, we found that NRC had, as of March 2000, spent approximately $6.7 million on the project and had expected to implement three of the modules for parallel testing purposes.9
Although NRC prepared the necessary planning documents to initiate the STARFIRE project, we found that NRC has not employed adequate oversight to the project. We found that (1) the implementation of STARFIRE as envisioned is uncertain, (2) STARFIRE managers have not determined the overall cost to NRC or updated key planning documents, and (3) the STARFIRE project bears similarities to previous agency IT development efforts.
While NRC envisions STARFIRE to become a fully integrated agency-wide financial management system, the effort to implement a Core Accounting Module was unsuccessful. Without the Core, the goal of a fully integrated system cannot be achieved. Although there may be various reasons for the inability to implement the Core - some within the agency's control and others outside - NRC has not made a focused, concerted effort to document what it learned from that experience to prevent a recurrence in the future. Furthermore, it has not documented its revised approach to implement the system. As a result, we believe, the implementation of a fully integrated financial management system, as envisioned for STARFIRE, is uncertain.
The STARFIRE planning documents describe the system as an automated and integrated approach to conduct agency-wide financial and resource functions that has as its "cornerstone" a Core Accounting Module to which all the other modules interface. According to NRC's functional requirements for STARFIRE, this Core will house the General Ledger and interface with all other modules to produce standard and required reports. The Core, identified as the centerpiece of the system, was to be operational in 1½ years.10 STARFIRE managers told us that their intent was to implement the Core first and then implement the other modules to integrate with the Core. Furthermore, they had expected that the vendor selected to implement the Core would also be able to implement a large number of the remaining modules.
Despite the intended approach for implementing STARFIRE, NRC's effort to implement the Core Accounting Module did not succeed and the agency decided to undertake a different approach to accomplish the same vision. We did not attempt to focus on the reasons underlying the unsuccessful attempt to implement the Core, although we received various suggestions as to why this may have occurred. Possible explanations from NRC staff included (1) immature technology in terms of in-house, client-server core accounting software for use by the Federal Government, (2) the contractor's overestimation of its abilities, (3) an inadequate financial software certification process, (4) a project schedule that lacked interim deliverables, (5) inadequate functional requirements developed by NRC, and (6) an unrealistic initial time frame for implementing the system.
After the unsuccessful Core Accounting Module development effort, NRC decided not to pursue a Core for several years. It seems crucial, therefore, that a written lessons learned analysis be prepared to document the reasons for the lack of success and what could be done differently to prevent the same problems from recurring. According to one STARFIRE manager, there were valuable lessons learned through the experience with the Core. One lesson was that a more intensive demonstration of the product being evaluated and considered for purchase should be completed. Nevertheless, we were told that this and other ideas had not been formally documented.
In a 1997 OIG Special Evaluation,11 we noted that NRC had a history of problems in developing and implementing management information systems. We also identified the need for the agency to learn from its own successes and failures, as well as those of others. In addition, agency guidance in Management Directive (MD) 2.212 specifies the importance of lessons learned documents. While MD 2.2 refers specifically to the preparation of lessons learned papers after an IT system becomes operational, we believe it is of equal importance to document lessons learned in a situation such as STARFIRE, where the Core never became operational.
While agency guidance concerning the development of IT systems describes specific measures for managing major projects, STARFIRE managers have not followed certain basic portions of this guidance. By failing to follow these fundamental project management measures - such as maintaining updated project schedules and management plans - the managers place a "high risk" project further at risk and contribute to STARFIRE's uncertain future.
NRC's CPIC process and SDLCM methodology outline the steps needed to keep IT projects within budget and on schedule. The CPIC guidance presented in MD 2.2, and the accompanying Handbook, are aimed at ensuring that NRC has a CPIC process that maximizes the value and assesses and manages the risk of IT investments. It is also intended to promote managers' accountability. Earlier CPIC guidance, effective prior to MD 2.2's approval in May 1999, also emphasized the importance of a project management plan, with milestones and deliverables, as a basis for project management.
Guidance on the SDLCM methodology is found in Draft MD 2.5.13 This draft directive was issued in January 1997 as interim guidance for application system life-cycle management until a final version could be approved.14 The SDLCM methodology is intended to be a structured approach to designing, developing, deploying, maintaining, and decommissioning information systems. Everyone involved in the application system development and maintenance process at NRC is required to use the SDLCM methodology. Similar to the CPIC guidance, the SDLCM methodology emphasizes the importance of using the project action plan as a project management tool and requires that the plan be updated throughout the process as appropriate. Another important SDLCM requirement is that there will be a formal Quality Assurance (QA) Plan describing the QA approach to be followed. The SDLCM guidance points out that the methodology is intended to be a flexible tool that calls for differing levels of adherence based on project scope and cost. However, we were told that there are certain components that must be addressed in a project such as STARFIRE.
While STARFIRE managers followed some of the measures required by the CPIC process and the SDLCM methodology, they did not employ certain fundamental strategies required by the guidance. Despite the unsuccessful attempt to implement the Core Accounting Module - which has necessitated an entirely different approach to the project - STARFIRE managers have never revised their business case documents, although they see the need to do so. It also appears that specific individuals involved with STARFIRE maintain the knowledge-base for the project. Our concern is that costly delays could result if these individuals were to become unavailable for an extended period of time, and there was no up-to-date documented approach for development and implementation. Furthermore, the STARFIRE Project Manager told us that the initial vision for STARFIRE is changing in the sense that the final system may not include an executive information system, a data warehouse, and a Performance Measurement module. Yet, the latest information available on the new STARFIRE web site does not reflect these uncertainties.
With regard to schedule - another basic project management tool - STARFIRE managers stated they do not keep the schedules up to date as a routine matter. Furthermore, there is no concrete schedule information concerning the project as a whole. The STARFIRE web site states only that implementation of 7 of the 12 currently planned STARFIRE components (i.e., 10 modules, a data warehouse, and an executive information system) has "been postponed until Fiscal Year 2002 or later." STARFIRE managers also told us they have no reliable estimate of when the system might be complete.
The project also lacks a written risk mitigation plan for STARFIRE, despite OCIO's specific request to OCFO to provide one. Interestingly, Handbook 2.2, which states that a project management plan15 should include a one- to two-page plan explaining how risks will be managed, refers readers to the NRC web site for a specific example of such a plan. The one-page plan to which it refers is titled, "Illustrative Risk Management Plan for Basi[c] Human Resources Portion of STARFIRE," yet the actual project management plan for Basic HRIS includes only a paragraph on the subject. We also were told by STARFIRE managers that because STARFIRE was relying on COTS software, a large portion of the SDLCM is not applicable. For example, we were told the portion of the SDLCM pertaining to QA did not apply.
With regard to cost, the CPIC guidance requires that the project cost estimates developed as part of the business case allow the sponsor to manage the project within 5 percent of the projected cost and schedule. Yet, we found that STARFIRE managers lack an estimate of overall project cost.
We also found that DCPM has not been reviewing contractor invoices for the project, but relying on the STARFIRE Project Manager to conduct that level of review. DCPM has been primarily depending on the STARFIRE Project Manager to notify them of schedule changes and missed milestones, should they occur, instead of tracking the project in a more proactive manner. We were pleased to learn that DCPM has recently strengthened its approach to monitoring this contract. A contract specialist now will be attending weekly meetings on STARFIRE and reviewing and signing off on all invoices pertaining to STARFIRE. DCPM will also receive monthly technical progress and financial status reports concerning the project.
We believe that the project management concerns identified during this review are, in large part, due to a lack of experience and training necessary to coordinate a project such as STARFIRE. NRC requires that project officers complete the agency-sponsored acquisition training courses, but according to a DCPM staff member, this requirement is not enforced. Furthermore, the project management training provided by the Office of Administration does not address the SDLCM requirements. NRC requires that the SDLCM methodology be followed by everyone involved in the application systems development and maintenance process at NRC. However, specific training on the methodology is required only of NRC employees assigned to be task managers under NRC's Comprehensive Information Systems Support Consolidation (CISSCO) program. Some work on STARFIRE is being performed by the Computer Sciences Corporation under the CISSCO program, but the bulk of the project is being conducted by other contractors. Since NRC has no current documented approach on how to implement the fully integrated system it envisions, it is impossible to identify when the system will be complete or at what cost to the agency.
Our review of STARFIRE is the latest in a series about systems development and implementation projects at the NRC. Our previous work includes a review of agency oversight of information resources management activities (1996), the Pay/Pers system (1996), the M-Cubed contract (1996), the PC Refresh Program (1998), and the Agencywide Documents Access and Management System (ADAMS) project (1999). These reviews have consistently reported issues and problems with NRC's management and oversight of IT projects. While the agency has made improvements in how it budgets for and plans new systems development efforts, we believe the STARFIRE experience demonstrates continuing deficiencies in the management and oversight of such projects.
One common thread that runs through all these projects is that costs seem to increase without limit. In other words, as long as funds are available and oversight is lacking, systems development projects seem free to continue on for indefinite periods of time and at increasing costs. In the immediate experience, STARFIRE started out to be a $6.9 million, 2½ year project, featuring a Core Accounting Module integrated with a suite of other modules. Currently, the agency has spent $6.7 million (with an additional $2 million awarded) and has not deployed any modules, and plans for the Core have been postponed until FY 2002 or later. Pay/Pers, which promised to deliver a labor cost distribution system, was a project that was over budget and schedule, and did not deliver what the agency intended.
STARFIRE is one of the first NRC IT projects that has attempted to employ the SDLCM methodology and to follow the CPIC process. While STARFIRE managers have followed certain aspects of both of these project management strategies, the STARFIRE experience demonstrates that the agency still has not fully implemented the process and methodology for managing its IT projects. Until the agency comes to terms with the need to employ these strategies in a rigorous manner, IT system development efforts will remain dubious. Furthermore, NRC guidance assigns the agency's Executive Council16 responsibility for reviewing "major IT projects that are at risk for a significant variation from their approved cost, schedule, or performance goals [and deciding] whether or not to continue, modify, or terminate such projects." However, we believe the phrase "significant variation" does not provide adequate guidance to facilitate the EC's ability to make fully-informed decisions concerning whether or not to "continue, modify, or terminate major IT projects.
While STARFIRE is still envisioned to become a fully integrated agency-wide financial management system, the effort to implement the cornerstone of the system was unsuccessful. Yet, the agency, while continuing to pursue its final vision, has not updated its plans or reestimated the overall costs and schedule to achieve this vision. Although STARFIRE was to be completed by October 1999 at a cost of $6.9 million, the agency has, to date, spent nearly $7 million on the project, and no modules have been deployed. By failing to follow key aspects of the agency's guidance on developing IT systems, and by failing to provide adequate project oversight, NRC finds itself without a lessons-learned document and without an up-to-date plan for achieving its vision for STARFIRE. We believe the agency runs considerable risk of repeating its own history by spending far more than anticipated on the project over an extended period of time as a result, and not getting what it intends from the effort.
Our recommendations are aimed at ensuring that
STARFIRE and future IT projects are effectively and efficiently managed.
First, we recommend that, with regard to STARFIRE, the Chief
Financial Officer:
Document lessons learned from the unsuccessful attempt to implement the Core Accounting Module
Reassess the overall STARFIRE project expectations, goals, and methodology, and revise appropriate business case documents to reflect the current approach. In doing so, estimate total cost and schedule, while maintaining the original baselines for the project
Adhere to agency guidance on IT system development by updating project action plans and schedules, and including key components of the SDLCM as appropriate (e.g., plans for quality assurance and risk mitigation).
With regard to IT project management at NRC, we recommend that the Chief Information Officer:
Finally, we recommend that the Executive Director for Operations:
Incorporate training on the CPIC process and the SDLCM methodology into project management training, and require the training for all individuals responsible for IT project management
Ensure that DCPM provides necessary oversight of the entire STARFIRE project, including invoice review, monitoring of deliverable dates, and overall project cost.
On June 19, 2000, the Deputy Chief Financial Officer, the Acting Chief Information Officer, and the Deputy Executive Director for Management Services responded to our draft report. They agreed with our recommendations and provided the corrective actions planned to address our concerns and time frames for these actions.
The agency's response also included four comments concerning information presented in the report, and we feel it is important to respond to these comments. Comment number one states that the report "does not acknowledge those planning aspects and project management of STARFIRE that worked well." The agency states, "the draft document does not acknowledge that the system worked effectively to promptly terminate a contract when the vendor failed to perform." From our perspective, it is not possible to categorize the termination as a project management success until all of the factors that preceded the decision to terminate (e.g., development of functional requirements, product testing, deliverable schedule, vendor selection) are assessed in a lessons-learned analysis. Until such analysis is complete, and the entire contract is examined, we cannot agree that this situation demonstrated effective project management.
The agency's second comment implies that our use of $6.9 million as the baseline cost of the system is not accurate. We restate our basis for use of the $6.9 million figure. This is the amount presented in NRC's December 1997 STARFIRE Capital Planning and Investment Control (CPIC) document. Furthermore, our draft (and final) audit report acknowledges the July 1998 addendum to the CPIC, which added $1.35 million to the estimate and which, we were told, was the basis for the increased figure provided to OMB.
In response to the agency's third comment, we agree that the original vision of STARFIRE has not changed. However, the agency does not know when this vision will be achieved or at what cost. In lacking both time and cost estimates for core implementation, the agency has no clear roadmap for achieving its goal. Finally, with regard to the agency's final point concerning the evolving vision for STARFIRE, we note that our report simply reflected the information provided to us on more than one occasion during the course of our audit.
The primary objective of our audit was to determine whether the U.S. Nuclear Regulatory Commission (NRC) has a sound methodology in place for implementing an integrated financial management system (the Standard Financial and Integrated Resource Enterprise [STARFIRE] system) that meets expectations (performance), at cost, and within expected time frames (schedule). To accomplish this objective, we focused on three sub-objectives to determine (1) applicable criteria and methodology NRC is to adhere to in developing and implementing STARFIRE; (2) the cost, schedule, and deliverables to date (in relation to what was planned); and (3) the expected cost, schedule, and performance of STARFIRE as it exists today and what it is expected to be.
We reviewed the agency's management controls and the effectiveness of those controls related to the STARFIRE project. In order to determine the current status of STARFIRE, we interviewed personnel in the Office of the Chief Financial Officer, Office of the Chief Information Officer, Office of Human Resources, and Office of Administration. We also interviewed staff from the Computer Sciences Corporation (contractor for the Comprehensive Information Systems Support Consolidation program) involved in the development of the system. We reviewed guidance documents for information technology project development (Capital Planning and Investment Control (CPIC), System Development and Life-Cycle Management Methodology, and applicable Management Directives), current project plans (STARFIRE CPIC, Project Action Plan, and functional requirements), schedules and status, project costs and budget, and contracting information. Furthermore, we reviewed previous Office of the Inspector General reports related to control over IT projects. We conducted our audit from October 1999 to March 2000 in accordance with generally accepted Government auditing standards.
| MEMORANDUM TO: | Stephen D. Dingbaum Assistant Inspector General for Audits |
| FROM: | Peter J. Rabideau Stuart Reiter Patricia G. Norry |
| SUBJECT: | Draft Audit Report -- Review of the Development and Implementation of STARFIRE |
Thank you for the opportunity to comment on the draft OIG audit report and to respond to the six recommendations. Our responses to the recommendations are as follows:
The CFO should document lessons learned from the unsuccessful attempt to implement the Core Accounting Module.
Response.
Agree. We will formally document lessons learned by the first of the calendar year.
The CFO should reassess the overall STARFIRE project expectations, goals, and methodology, and revise appropriate business case documents to reflect the current approach. In doing so, estimate total cost and schedule, while maintaining the original baselines for the project.
Response.
Agree. We will formally reassess STARFIRE project expectations along with updating the requisite documentation by the end of FY 2000. As the OIG is aware, we intend to complete the STARFIRE project in two phases with the second phase beginning after FY 2002. Therefore, we will be estimating the second phase costs and performing the required CPIC analyses prior to implementing this phase. We will evaluate the need for a revised CPIC on the first phase.
The CFO should adhere to agency guidance on IT system development by updating project action plans and schedules, and including key components of the SDLCM as appropriate (e.g., plans for quality assurance and risk mitigation).
Response.
Agree. We will fully adhere to agency guidance on IT system development and update project action plans and schedules by the first of the calendar year.
The CIO should ensure that the Executive Council is fully aware of the circumstances surrounding and the need for a "significant variation" from approved cost, schedule, and performance goals for major IT projects. As part of this recommendation, define "significant variation" and ensure that this definition accounts for aggregate changes that may total to a "significant" level. Refining this definition should better enable the EC to make fully-informed decisions on whether to continue, modify, or terminate IT projects.
Response.
Agree. OCIO will address the clarity of the definition of "significant variation" from baseline cost and schedule during the CPIC lessons learned review which is currently underway. That review is scheduled to be completed in the fourth quarter of FY 2000.
The EDO should incorporate training on the CPIC process and the SDLCM methodology into project management training for all individuals responsible for IT project management.
Response.
Agree. The NRC has long recognized the importance of developing and maintaining a competent acquisition workforce. In 1991 the EDO directed that project managers take NRC acquisition training courses. A comprehensive five-day training program was in place to meet that requirement.
More recently, HR and ADM replaced the week-long training with a more flexible eleven-module training program covering every aspect of acquisition management including developing an independent cost estimate, preparing statements of work, and contract administration. In May 2000 the EDO established the Agency Acquisition Certification and Training Program which requires project managers to attend five core acquisition modules, and recommends attendance, as needed, at the remaining six modules.
ADM will work with OCIO and HR to revise the current mandatory acquisition modules to include appropriate guidance on the CPIC process and the SDLMC methodology. ADM expects to incorporate necessary changes in the course curricula by September 30, 2000.
Ensure that DCPM provides necessary oversight of the entire STARFIRE project, including invoice review, monitoring deliverable dates, and overall project cost.
Response.
Agree. As acknowledged in the report findings, DCPM has taken steps to assure timely review of invoices and monitoring of contractor conformance with delivery requirements for individual contracts awarded under the STARFIRE project. With regard to oversight for the entire STARFIRE project, DCPM will work with the CFO and CIO as appropriate to monitor this major IT initiative consistent with requirements of the Clinger-Cohen Act.
We have the following comments to ensure completeness of the audit report.
The draft document does not acknowledge that the system worked effectively to promptly terminate a contract when the vendor failed to perform. This follows guidance in the Information Technology Management Reform Act (ITMRA) to assess projects and make a determination of the viability of the projects success. Without effective project management, and without appropriate contracting oversight, the project could have gone on for an extended time without producing the desired system. The project was terminated in a timely manner, cost to NRC was minimized, and other modules of the project are continuing. Additionally, based on contractual arrangements, the vendor was not to be paid until NRC accepted the implemented module. The vendor worked for approximately 18 months without being paid. In return for the $450K that was spent on a negotiated settlement with the terminated contractor, the agency received travel software with a market price of between $150K and $200K and avoided potential expensive litigation. It should also be pointed out that the Core Accounting System that was rejected currently is not operating without major problems in any agency in which it is being implemented.
The audit report should also acknowledge that the modular contracting strategy employed by the project has the following merits:
(a) It allowed other components to continue after the Core Accounting Module contract was terminated.
(b) It allows the agency to realize benefits as modules are implemented.
(c) It is a comparatively low risk approach, promoted by the Clinger-Cohen Act.
The audit report should acknowledge that the baseline "full acquisition" cost reported to OMB in Exhibit 300B of the NRC FY 2000 budget submission was $8.4 million. Exhibit 300B is a formal document reviewed and approved by NRC management as part of its budget request. In addition, the Exhibit 300B of the NRC FY 2001 budget submission identifies an estimated increase in project cost, a revised overall project cost and describes the variation in project approach stemming from the termination of the contract for the core accounting module.
What the STARFIRE project initially desired was to obtain all envisioned modules in one integrated package from one vendor. Because one of the modules was core accounting we were required to obtain the software only from JFMIP approved vendors. When we solicited bids from JFMIP approved vendors, no vendor had all the desired modules in one integrated package. All were required to interface their product with other vendor products to achieve the desired results. Therefore, STARFIRE was going to be a combination of integrated and interfaced modules. Although we are proceeding without the core accounting vendor we selected, the concept of STARFIRE being a combination of integrated and interfaced modules has not changed.
This statement requires clarification. It was initially envisioned that STARFIRE would have to buy hardware and software to provide for an executive information system, a data warehouse and performance measurement. While we still plan for this functionality, STARFIRE may be able to take advantage of existing hardware or software rather than making an additional purchase. For example, the NRC will be studying the possibility of building an agency-wide data warehouse. If this happens, STARFIRE may be able to take advantage of this resource and not purchase its own. Also, if reports from the implemented cost accounting module can satisfy the requirements of a Performance Measurement module it would not be prudent to buy additional software. STARFIRE's vision is to deliver functionality needed for the agency and that vision does not change because we take advantage of other existing hardware or software.
| cc: | W. Travers, EDO |
| J. Funches, CFO |
Anthony Lipuma
Team Leader
Judy Gordon
Senior Management
Analyst
Gina Smith
Senior Auditor
1. INVESTIGATIVE REPORT - WHITE
COVER
An Investigative Report documents pertinent facts of a case and
describes available evidence relevant to allegations against individuals,
including aspects of an allegation not substantiated. Investigative reports do
not recommend disciplinary action against individual employees. Investigative
reports are sensitive documents and contain information subject to the Privacy
Act restrictions. Reports are given to officials and managers who have a need to
know in order to properly determine whether administrative action is warranted.
The agency is expected to advise the OIG within 90 days of receiving the
investigative report as to what disciplinary or other action has been taken in
response to investigative report findings.
2. EVENT INQUIRY - GREEN COVER
The Event
Inquiry is an investigative product that documents the examination of events or
agency actions that do not focus specifically on individual misconduct. These
reports identify institutional weaknesses that led to or allowed a problem to
occur. The agency is requested to advise the OIG of managerial initiatives taken
in response to issues identified in these reports but tracking its
recommendations is not required.
3. MANAGEMENT IMPLICATIONS REPORT (MIR) - MEMORANDUM
MIRs provide a "ROOT CAUSE" analysis sufficient for managers to facilitate correction
of problems and to avoid similar issues in the future. Agency tracking of recommendations
is not required.
4. AUDIT REPORT - BLUE COVER
An
Audit Report is the documentation of the review, recommendations, and findings
resulting from an objective assessment of a program, function, or activity.
Audits follow a defined procedure that allows for agency review and comment on
draft audit reports. The audit results are also reported in the OIG's
"Semiannual Report" to the Congress. Tracking of audit report recommendations
and agency response is required.
5. SPECIAL EVALUATION REPORT - BURGUNDY
COVER
A Special Evaluation Report documents the results of short-term,
limited assessments. It provides an initial, quick response to a question or
issue, and data to determine whether an in-depth independent audit should be
planned. Agency tracking of recommendations is not required.
6. REGULATORY COMMENTARY - BROWN COVER
Regulatory Commentary is the review of existing and proposed legislation, regulations,
and policies so as to assist the agency in preventing and detecting fraud, waste,
and abuse in programs and operations. Commentaries cite the IG Act as authority
for the review, state the specific law, regulation or policy examined, pertinent
background information considered and identifies OIG concerns, observations,
and objections. Significant observations regarding action or inaction by the
agency are reported in the OIG Semiannual Report to Congress. Each report indicates
whether a response is required.
1. Parallel testing actually began on April 4, 2000.
2. The acronym, STARFIRE, stands for Standard Financial and Integrated Resource Enterprise system.
3. The NRC STARFIRE web site was launched in February 2000.
4. A July 1998 addendum to the CPIC added $1.35 million to the estimate for implementation of the Basic Human Resources Information System (HRIS), also referred to as the Human Resources (HR) module.
5. During the course of this audit, we observed that the contracting vehicles used to acquire services for the STARFIRE effort were sometimes referred to as purchase orders and sometimes as delivery orders. For purposes of this report, we are using the term purchase order to refer to these agreements.
6. There are two components to the Human Resources Module: Basic HRIS and Full HRIS. Basic HRIS, necessary for the implementation of STARFIRE, is being paid for by OCFO, while the additional components to Full HRIS will be paid for by the Office of Human Resources (HR). Basic HRIS comprises the functionality needed to support implementation of the Payroll and Time and Labor modules. Full HRIS, which is not considered part of STARFIRE, is expected to consolidate many of the existing HR systems and bring new HR management capabilities to the agency.
7. With the issuance of this purchase order, work performed for Full HRIS will now be completed under a separate purchase order managed by HR.
8. This contract was prepared with a three-phase approach consisting of a phase I period for $416,966 (including $50,000 for travel expenses), and two optional phases.
9. Parallel testing actually began on April 4, 2000. During parallel testing, data is entered into both the new modules and the existing NRC payroll system to determine if both systems produce consistent results.
10. Implementation was to occur in October 1998.
11. OIG/97E-10, Evaluation of Best Practices for Developing and Implementing Integrated Financial Management System, dated June 9, 1997.
12. Capital Planning and Investment Control, May 27, 1999.
13. Application Systems Life Cycle Management.
14. As of March 2000, a final version of MD 2.5 had not been approved.
15. According to the SDLCM guidance the project management plan is a subsection of the Project Action Plan.
16. NRC's Executive Council is composed of the CIO, CFO, and EDO.
17. We are referring to the term, "significant variation," as it is used in MD 2.2 to describe the EC's responsibility in connection with the CPIC process.
|
Privacy Policy |
Site Disclaimer |