Federal Deposit Insurance Corporation
Office of Audits
Office of Inspector General
Washington, D.C. 20434

DATE: March 21, 2003

TO: Steven O. App, Deputy to the Chairman and Chief Financial Officer; and John F. Bovenzi, Deputy to the Chairman and Chief Operating Officer

FROM: Russell A. Rau [Electronically produced version; original signed by Stephen M. Beard for Russell A. Rau], Assistant Inspector General for Audits

SUBJECT: Review of FFIEC Call Report Modernization Cost Benefit Analysis (Audit Report No. 03-018)

At the request of Corporation senior managers, we reviewed the cost benefit analysis (CBA) and assumptions supporting the draft request to the FDIC Board of Directors for funding the Federal Financial Institutions Examination Council (FFIEC) Call Report Processing Central Data Repository, dated January 23, 2003. (Note: The FFIEC includes representatives from the Board of Governors of the Federal Reserve System (FRB), National Credit Union Administration (NCUA), Office of the Comptroller of the Currency (OCC), Office of Thrift Supervision (OTS), and FDIC.) The senior managers requested that we provide them with our analysis prior to the scheduled February 11, 2003, FDIC Board of Directors meeting.

Our objective was to determine whether the cost information contained in the CBA was supported and the assumptions made were reasonable. We reviewed the methodology and supporting documentation used to prepare the CBA. The scope of our review was limited to determining whether the FDIC had adequately supported amounts presented in the business case and the underlying assumptions were reasonable. We did not evaluate the viability or benefits of the alternatives and options in the case. A detailed discussion of our scope and methodology is included as Appendix II. The purpose of this report is to provide you with the results of our review and the slides (Appendix I) we used to brief management on February 7, 2003. (Note: We amended the briefing slides to include technical corrections and clarifications.) We conducted our audit in accordance with generally accepted government auditing standards during the period January 27, 2003 through February 14, 2003.

BACKGROUND

FFIEC Effort to Modernize the Call Report Process. In 1998 the FFIEC’s Task Force on Reports initiated a Call Report modernization effort. An outgrowth of this initiative was the interagency Call Modernization project launched in 2001. The Call Modernization project is a collaborative effort of the FDIC, FRB, and OCC to improve the processes and systems used to collect, validate, store, and distribute Call Report information. (Note: Every national bank, state member bank, and insured state nonmember bank each calendar quarter is required to file Consolidated Reports of Condition and Income (Call Report).) Institutions regulated by the OTS and NCUA are not required to submit Call Reports to the FDIC. Accordingly, the OTS and the NCUA were not included in this project.

The FFIEC Call Modernization Steering Committee also established a Cost Working Group to collect and analyze the cost of the current Call Report processes across the FDIC, FRB, OCC, and the reporting institutions. The FDIC in turn has a subgroup called the Institution Data Management (IDM) project team that performed the study for the FDIC. The study identified current (year 2002) annual operating costs of $10.5 million for the three agencies. Of the $10.5 million, the FDIC incurs $7.6 million, the FRB $2.4 million, and the OCC $0.5 million.

In May 2002, the FFIEC authorized the FDIC to issue a request for proposal to have a contractor design, implement, and host a new facility for Call Report and other bank regulatory reporting data. Bidding closed in October 2002, and the Call Modernization Steering Committee completed its analyses of the proposals in January 2003. The Committee tentatively selected a proposal in the amount of $39 million, which included $14 million in system development costs and $25 million for system operation and maintenance over a 10-year period. Additional funding of $5 million was requested for future contingencies, for a total contract budget of $44 million.

Two Alternatives Considered. The IDM project team prepared a CBA, dated January 23, 2003. The CBA compared two alternatives on the basis of cost, benefit, risk, and sensitivity. The alternatives presented were as follows:

• Alternative 1: Enhancing Existing Agency Systems – This alternative anticipated doing select upgrades of existing FDIC and/or FRB systems to meet minimum business requirements and operational needs for sustaining current Call Report performance levels.
• Alternative 2: Central Data Repository (CDR) Hosted by a Third-Party Vendor – This alternative includes implementation of an interagency CDR that meets the business and technical requirements defined by the FFIEC and is hosted and maintained by a private-sector vendor.

Alternative 1. The current Call Report process uses a 10-year old distributed approach to obtain, validate, and distribute Call Report data. Reporting institutions provide their Call Reports in electronic format to a private vendor, Electronic Data Systems, that consolidates and distributes the Call Reports to the FRB using the 25-year old Bulk Data Transfer format. The FRB parses the consolidated data based on the primary regulator and distributes FRB-supervised institutions’ data among the 12 Federal Reserve District Banks and sends FDIC- and OCC-supervised institutions’ data to the FDIC. The FDIC and the District Banks then edit the data using in-house methods and systems, resolving exceptions based on a combination of common and agency-specific criteria. Once the data are edited, files are exchanged among the regulators, and a consolidated file of all reports is compiled and distributed to internal and external customers.

Alternative 2. The Agencies developed a consensus vision for a new Call Report processing business model that incorporates open data standards, uses a common reporting language, and offers tools to enable banks to submit better reports. The goals of the proposed environment are to create: (1) a single CDR serving as the official source of all the information necessary for collecting, validating, and distributing bank Call Report information; (2) a process driven by uniform business rules leveraging existing standards; (3) a movement of regulators and reporters to a recognizable Internet standard; (4) a common, comprehensive set of FFIEC data quality assurance standards that will be defined and published; and (5) a requirement that only Call Reports that pass all math validation criteria and that provide text explanations for logical/qualitative validation criteria exceptions will be accepted.

Alternative Chosen. The Call Modernization Steering Committee of the FFIEC Task Force on Reports recommended alternative 2, with three actions necessary to implement a CDR contract. The three actions included:

• Funding a contract to be awarded for an initial period of 7 years, with three 1-year option periods, in the amount not to exceed $44 million for the entire 10-year term;
• Entering into a Memorandum of Understanding governing the development and operation of the facility, including business policy and oversight roles, subject to concurrence by the FRB and the OCC; and
• Entering into an agreement for sharing costs of implementing and operating the facility, subject to concurrence by the FRB and the OCC. The recommended formula for cost sharing is 72 percent of the costs to the FDIC, 23 percent to the FRB, and 5 percent to the OCC. (Note: The proposed cost-sharing agreement is based on the current (year 2002) allocation of costs across the Agencies as determined by the Call Modernization Steering Committee.)

The methodology used in comparing the alternatives was generally consistent with FDIC and Office of Management and Budget (OMB) guidance for the preparation of cost benefit analyses. The IDM project team obtained and analyzed cost data from several FDIC divisions, analyzed the benefits and risks associated with the alternatives, and projected the impact that the cost benefit assumptions could have on the recommended alternative in the sensitivity analysis. However, some of the assumptions and rationale used to arrive at the amounts included in the cost analysis were not consistently supported or clearly explained. Specifically, formal management verification, indicating agreement with the costs in the case, was not received from all affected divisions. As a result, there is a risk that the return on investment may not be fully realized. In addition, accountability for the cost benefits may be difficult to establish because the assumptions made were not adequately documented. Nonetheless, adjustments based on our concerns would fall within the range of the CBA sensitivity analysis, which indicates a positive return on investment for the CDR alternative chosen.

The applicable criteria for our review were FDIC Circular 4310.1, Utilizing Cost Benefit Analysis Methodology for the Purchase or Development of Capital Assets, dated July 17, 1998, and OMB Circular No. A-94, Guidelines and Discount Rates for Benefit-Cost Analysis of Federal Programs, both of which provide guidance for preparing CBAs. (Note: Although OMB Circular No. A-94 does not apply specifically to the FDIC, it is used as a prudent business practice in the area of government cost benefit analysis.) According to the circulars, CBAs should be performed to promote efficient resource allocation through well-informed decision-making. The CBA is included as part of the decision-making process when determining whether to purchase or develop a capital asset. The analysis should be explicit about underlying assumptions used to arrive at estimates of future benefits and costs. The analysis should include a statement of the rationale behind the assumptions and a review of their strengths and weaknesses. Key data and results should be reported to promote independent analysis and review. In addition, post implementation verification should be performed to determine whether anticipated benefits and costs have been realized and whether improvements need to be made in future estimates of benefits and costs.

The FDIC and OMB guidance recognize that there are inherent limitations in the estimates of costs and benefits because of imprecision in both the underlying data and the modeling assumptions. These limitations are generally addressed in the sensitivity analysis where the risks associated with the cost projections are addressed.

In preparing the cost analysis, the IDM project team obtained cost data from several divisions in the FDIC as well as the FRB and the OCC. In March 2002, the IDM project team sent each FDIC division involved in the CBA a copy of the current operating cost estimates for review. According to members of the IDM project team, they did not receive any feedback from the divisions on their March 2002 letter. The IDM project team did, however, obtain a subsequent costing letter from the Division of Information Resources Management (DIRM) regarding the DIRM costs included in the cost analysis.

Using this original cost data and a subsequent costing letter from DIRM, the IDM project team prepared two cost alternatives. (Note: Each alternative had two options. The first option related to core (Call Report) processes only, and the second option was for value-added services including Uniform Bank Performance Report processing, referred to as total or full implementation. Our analysis relates only to the full implementation option because that was the business-case-recommended option.) Alternative 1, Enhancing Existing Agency Systems, was derived by using the current (year 2002) estimated operating costs of the reporting processes and adding an additional $500,000 for the migration of one of the systems to a new operating platform. Under Alternative 2, CDR, the cost of current operations, which would no longer be necessary with the CDR, are eliminated as CDR is phased into operations. Further, the cost of the CDR contract is added to the cost of operations. The cost estimates were projected over a 10-year period with a 3-percent annual inflation factor. Using these amounts, alternative 2, CDR, results in a $27 million dollar reduction in costs over the 10-year period. A discount factor of 5.1 percent was applied to the cost reductions to arrive at a return on investment for the CDR of $17.9 million (133 percent) over the 10-year period.

During our review, we attempted to verify the cost data and assumptions used to derive the projections made in the cost analysis. Appendix II describes the procedures we performed to evaluate the CBA. However, in certain instances the data used or assumptions applied could not be traced to information included in the CBA or to additional documentation obtained from the IDM project team. While the DIRM costing letter was useful in validating the information included in the cost analysis and supporting the reasonableness of the assumptions made, the IDM project team did not obtain similar evaluations from the other divisions impacted by the CBA. Also, the IDM project team did not prepare in advance a methodology for ensuring consistent treatment of certain costs. If evaluations had been obtained and cost assumptions defined, several of the cost concerns discussed below may have been resolved.

• Alternative 2 did not clearly identify the staffing cost for the Call Report process or project management. The cost analysis in the CBA identifies a 50-percent reduction to the staffing requirements for the Call Report process for alternative 2. No costs were identified in the cost analysis for project management. However, according to the IDM project team, project management costs for the CDR were in fact included in the Call Report staffing cost for alternative 2. To more accurately reflect the cost associated with alternative 2, project management costs should have been clearly identified in the analysis, and there should have been a reduction to the staffing requirements for the Call Report process. Although there was no effect on the cost benefit, the source of the cost items was not clear and made understanding the cost estimates difficult.



• Alternative 2 transition costs were not clearly defined or sufficiently supported. Under alternative 2, the cost analysis identified legacy costs from the current operations continuing in full through 2003. Beginning in 2004, only the UBPR legacy costs were included and no legacy costs after 2004. Legacy costs are the current costs not required after full CDR implementation for a given phase. The analysis states that the projection of legacy costs assumes functionality of the Call Report process as of year-end 2003 and full CDR functionality as of year end 2004. However, the project plan included in the business case shows that CDR production will not begin until mid-April 2004 with UBPR continuing for another year. This is inconsistent with the legacy cost projections in the cost analysis. We estimated that there will be $1.1 to $1.8 million more in legacy transition costs based on the project implementation schedule included in the business case, than is reflected in the cost analysis.



• Alternative 1 included an additional $500,000 to migrate the current Call Report system to a new operating platform. This amount was in addition to the IDM project team’s determination of the estimated annual cost associated with system upgrades and enhancements. Our review of the historical costs used for determining estimated future costs indicated that system upgrades and enhancements were already included in the historical projections. Accordingly, either this cost should not have been included in alternative 1, or clarification should have been provided as to why this cost was separate from the historical projections.



• Alternative 2 did not include the estimated cost of modifying the current systems to accommodate data coming from the CDR. DIRM estimated a total cost of $1.1 million to modify the current systems to accept data from the CDR. DIRM noted that this was a transitional cost and would not be required if the CDR was not approved. The members of the IDM project team stated that they did not include this cost in the analysis because it was a "downstream" cost and not specifically required of the CDR. As noted by DIRM, the exclusion of this cost from the CBA required additional explanation. Including this cost in alternative 2 would reduce the return on investment.

As identified in the business case to the FDIC Board of Directors, the assumptions related to benefits and risks are highly dependent on the vendor’s successful implementation of the CDR. Because the CDR involves the use of new technology, the risks are greater than those that would be expected with existing proven technology both in terms of the success of the project and the cost in implementing the project. As discussed in the case, the IDM project team believes the potential benefits of the CDR project and new technology outweigh the potential risks. Some benefits and risks were not quantified in dollar terms in the CBA because the project team did not believe it was appropriate to do so, given the uncertainty of their nature. The scope of our review did not include an evaluation of these non-monetary benefits and risks. However, we did note that the business case included a risk mitigation plan to address concerns related to implementation of the new technology.

A sensitivity analysis was prepared to determine how sensitive outcomes are to changes in assumptions. The CBA identified the cost benefit of shutting down current operations as CDR is implemented as the dominant assumption. The sensitivity analysis projected the impact that a reduction in the cost benefit would have on the return on investment over a 10-year period. In preparing the sensitivity analysis, the IDM project team projected cost benefit reductions of 10, 25, 35, and 47 percent (47 percent represented the "break-even" point). If the reduction in cost benefit exceeds 47 percent, then the investment costs would not be recouped in full. The sensitivity analysis was performed properly.

The effect of the cost concerns discussed above would be a reduction in the realizable ROI from $17.9 million (133 percent ROI) to about $14.6 million (109 percent ROI). Consequently, even with the adjustments, alternative 2, CDR, still provides the best-cost alternative given the other data included in the cost analysis. The potential adjustments fall within the 10-percent reduction of the sensitivity analysis included in the CBA. Based on the limited nature of our review, we do not have a sufficient basis to make recommendations related to the cost benefit analysis for the CDR or future investments. However, we suggest that the FDIC consider:

1. Ensuring that assumptions made in preparing a CBA are fully documented to facilitate the post implementation review of benefits and costs, and



2. Including a requirement in FDIC Circular 4310.1 that each division impacted by alternatives in a CBA review concur or non-concur with its contents to establish accountability. Any non-concurrence from an impacted party should also be discussed in the CBA.

We discussed the results of our review and presented the attached slides to officials from the IDM project team on February 7, 2003. The IDM project team generally agreed that the assumptions and rationale used in preparing the cost analysis could have been better supported and more clearly explained. In consideration of the comments received from management, we amended the slides to include technical corrections and clarifications. The CFO and COO agreed with our suggestions and provided joint written comments to a draft of this report. These comments are included as Appendix III.

In response to our suggestions, the Corporation will:

1. Further emphasize in FDIC Circular 4310.1 the importance of fully documenting cost and benefit assumptions in a manner which will facilitate post implementation reviews, and
   
   
2. Add a sample form for detailing cost and benefit assumptions by division for each alternative, requiring concurrence by a senior representative of each affected division.

• On January 23, 2003, OIG was asked to evaluate the Cost Benefit Analysis (CBA)
• The objective was to determine whether (1) the information contained in the CBA was supported and (2) the assumptions made were reasonable.
• Preliminary results needed before February 11, 2003, FDIC Board of Director’s meeting.

• The CBA presents two main cost proposals. The first proposal is for enhancing current systems. The second proposal is for implementation of the Central Data Repository (CDR).

• Cost Analysis
• Benefit Analysis
• Risk Analysis
• Sensitivity Analysis

• FDIC Circular 4310.1 Utilizing Cost Benefit Analysis Methodology for the Purchase or Development of Capital Assets
• OMB Circular A-94 Guidelines and Discount Rates for Benefit-Cost Analysis of Federal Programs

• Methodology for preparing the cost analysis is consistent with FDIC and OMB guidance.
• Exhibits and supporting tables were generally computed accurately and sufficiently explained.
• Inflation and discount factors were appropriate.
• Two main cost proposals: enhancing current systems (alternative 1) or implementing the CDR (alternative 2).
• Cost projections are made over 10 years.

• Some assumptions and rationale used to arrive at costs were not consistently supported or clearly defined. Specifically:
• Project management costs for the CDR were not identified in the CBA.
• Additional system enhancement cost of $500,000 was added to historical data for enhancements. The rationale for the additional cost was not fully supported.
• Transition costs associated with the change to the CDR were not fully factored into the CBA. These are costs that are expected to continue until 2005. The cost analysis shows these costs discontinuing earlier. OIG estimates an additional $1.1 to $1.8 million in transition costs will be incurred.
• A development project to modify current systems for the CDR implementation, with an estimated cost of $1.1 million, was not included in the cost analysis.

• Both quantifiable and non-quantifiable benefits are addressed in the benefit analysis.
• The only quantifiable benefit was based on the cost analysis.
• Non-quantifiable benefits of the CDR option are:
• Eliminate duplicative processes among FFIEC agencies.
• Direct transmission of Call Reports.
• Reject non-conforming reports before processing
• Update technology
• Benefits consistent with Business Case
• FDIC as an innovative leader in comprehensive banking information
• Delivering high-quality data faster
• Creating new value from current and future products
• Benefits to the industry
• Improving Agency operational effectiveness

• Presentation conforms with guidance from OMB for conducting the risk analysis.
• Higher risk associated with CDR.
• Risk mitigation plans were identified for risk related to evolution of XBRL standards, Regulatory Burden, Implementation Risk, and Funding Risk.
• Risks consistent with Business Case
• XBRL new technology
• Size of the expenditure
• Outsourcing the hosting and maintenance
• Operational Control
• Capturing the savings

• The Business Assumptions Benefits and Risks are highly dependent on the success of the new XBRL protocol for financial information reporting
• The case includes a risk mitigation plan to address anticipated issues related to XBRL

• Sensitivity analysis was performed properly.
• Sensitivity analysis identified the dominant benefit as the cost savings resulting from the shutdown of Call Report and other reporting processes and systems. The risk is that FDIC will not be able to shut down operations and realize cost savings as CDR is implemented.

• The DIR generally followed the guidance of FDIC Circular 4310.1 and OMB Circular A-94 in performing the Cost Benefit Analysis.
• "Estimates of benefits and costs are typically uncertain because of imprecision in both underlying data and modeling assumptions." OMB Circular A-94
• Adjustments based on OIG concerns are within the range addressed in the sensitivity analysis.
• Cost Benefit Analysis process could be improved.

• Objective: To determine whether (1) the information contained in the business case is supported and (2) the assumptions made are reasonable.
• Scope: FFIEC Call Report CDR business case dated January 23, 2002
• Methodology: Review the business case, supporting documentation, and FDIC and OMB guidance for case preparation. Interview personnel involved in the preparation of the case.

To accomplish our objective, we interviewed Headquarters Division of Insurance and Research (DIR) officials who were responsible for the preparation of the CBA. We also interviewed DIR, DIRM, and Division of Finance (DOF) officials who were involved in the accumulation of FDIC cost data included in the CBA. We reviewed key documents supporting the CBA, including spreadsheets prepared by the IDM project team, cost assumptions prepared by DIRM, personnel cost figures prepared by DOF, contract costs, and other documents related to specific cost items.

The scope of our audit was limited to determining whether the FDIC had adequately supported amounts presented in the business case and the underlying assumptions were reasonable. Our audit did not evaluate the viability or benefits of the alternatives and options in the case. We did not evaluate the internal control environment over the preparation of the business case. We will assess internal controls in future audits of the IDM project. In addition, our audit did not assess the FDIC’s compliance with applicable laws and regulations because we did not identify specific laws or regulations pertaining to the development of the business case.

Our audit included a determination of whether the FDIC had incorporated performance measures in the business case. We noted that the case describes its alignment with the FDIC strategic vision, mission, and business goals by promoting interagency collaboration, consolidating back office operations, leveraging new technologies for current and future operations, and promoting the FDIC as the source for banking data.

In addition, we relied on computer-processed data without performing tests of system general and application controls to confirm the reliability of the data. We verified that the computer-generated computations were accurate, but we did not review the source data obtained from the systems. However, nothing came to our attention as a result of specified audit procedures that caused us to doubt the reliability of the computer-processed data. Throughout the audit, the auditors were sensitive to the possibility of abuse or illegal acts.

We conducted our audit in accordance with generally accepted government auditing standards during the period January 27, 2003 through February 14, 2003.

Federal Deposit Insurance Corporation
Washington, DC 20429
Deputy to the Chairman

March 17, 2003

MEMORANDUM TO: Russell A. Rau, Assistant Inspector General for Audits

FROM: Steven O. App [Electronically produced version; original signed by Steven O. App], Deputy to the Chairman and CFO; and John F. Bovenzi [Electronically produced version; original signed by John F. Bovenzi], Deputy to the Chairman and COO

SUBJECT: Review of FFIEC Call Modernization Cost Benefit Analysis (Assignment No. 2003-026)

At the request of the Capital Investment Review Committee, the Office of Inspector General (OIG) reviewed the cost benefit analysis (CBA) and the assumptions supporting the draft request to the FDIC Board of Directors for funding the Federal Financial Institutions Examination Council (FFIEC) Call Report Processing Central Data Repository (CDR), dated January 23, 2003. The objective was to determine whether the cost information contained in the CBA was supported and whether the assumptions were reasonable.

The OIG reviewed the methodology and supporting documentation used to prepare the CBA. Based upon the limited nature of the review, the OIG did not have a sufficient basis to make recommendations related to the cost benefit analysis for the CDR or future investments. However, the OIG provided suggestions for improving the FDIC cost benefit analysis directive (FDIC Circular 4310.1).

We concur with and appreciate the OIG suggestions. We are currently in the process of updating or replacing our cost benefit policy to include additional elements such as return on investment (ROI). As part of this effort we will:

1. Further emphasize in the directive the importance of fully documenting cost and benefit assumptions in a manner which will facilitate post implementation reviews, and
2. Add a sample form for detailing cost and benefit assumptions by Division for each alternative, requiring concurrence by a senior representative of each affected division.

Overall, we believe the OIG review presented a fair evaluation of the work performed by the CDR project team.

Cc: Vijay G. Deshpande
Arthur Murton
Fred Selby
Michael MacDermott
James Collins
James E. Crum
Thomas Peddicord
Gail L. Verley