Inside eRA, April 4, 2001

The Good, the Bad, and the Ugly: Migration Post Mortem

On the weekend of February 16th the IMPAC II database underwent a major version migration. The IMPAC II Production (OLTP) and Reporting (IRDB) databases were upgraded to Oracle version 8i, as were the COTS products and tools (software used to build and run applications on your computers). For the most part, the migration was successful.

This major undertaking required hundreds of hours of data base administrator (DBA), developer and tester time to prepare and deploy the database and applications. The upgrade was long overdue. The Oracle Corporation no longer supported the previous versions of the software and the NIH community had requested that support be provided for IMPAC II users on the Windows 2000 platform. The old version of the software did not support Windows 2000. Also, the new version of the database software will enable eRA to move more smoothly to an all WEB environment and enable the development of new functionality.

The result of this effort was for the most part successful. Within a day of deployment most of the ICs upgraded their software and were successfully running the new applications. IMPAC II staff worked with a number of ICs over the weekend and we are thankful for these "early adopters" (NIAID, NIMH, NIGMS and others) who helped fine-tune the deployment software. The issues we were most concerned with -- if the applications would run correctly -- turned out to be non-issues. There were only a few minor problems noted with any of the IMPAC II applications after deployment.

The problems that did surface were associated with the database itself. The first week after migration, users experienced a number of instances of slow performance on the system. Working with Oracle we were able to adjust a number of system parameters and performance steadily increased over the first week. The other major database problem was the inability to bridge IMPAC II data to the IRDB database. After over a week of struggling with the bridge program we were forced to "manually" bridge the data to the IRDB. As of this writing, a solution has been identified and implemented to restore the automated bridging process between the IRDB and the OLTP. The nightly updates will take place on their regular schedule, after the IMPAC I bridge is complete. While the IRDB does not affect most of the IMPAC business processes, users who use the IRDB database for reports found the data over a week out of date. Users of the Electronic Council Book (ECB) were also unable to get up-to-date information.

Some tuning of the database after the migration was expected, however, the full extent of the problems that surfaced was not. Part of the reason is that the available testing environment is inadequate to simulate the production database environment. While the current test environment is adequate for application testing and for some database testing, it is not adequate for testing the system under a full load (over 500 simultaneous users) nor is it large enough to simulate the real world environment. Resources were requested in the FY 2001 budget to purchase equipment that would provide an adequate test environment, but as the eRA budget was not approved until February, the equipment is not yet available.

While not exactly a "lesson learned", one thing the migration demonstrated is the need for the IMPAC II/eRA project to move away from a Client Server Architecture and move to a full Web based environment. Steps are underway to move the NIH Commons architecture to an environment where the only software needed by the client (user) would be a web browser.

The migration was carefully scheduled around council dates and a three-day weekend in order to minimize potential disruptions to NIH business. In the future, OER will continue to alert technical staff in advance of any system changes and Inside eRA will alert the broader NIH community. Major database migrations like this do not happen very often. Hopefully with the lessons learned, an adequate test environment, and better communications, future migrations will be less burdensome to the user community.

Professional Profile Data: The Cost of Flexibility

The eRA project team recently released funds to implement a $2 million per year effort to cleanup personal data in the IMPAC II database. A contract will be awarded that will: 1) identify existing data conflicts; 2) manually correct any errors; and 3) develop and implement proposals to increase accuracy of the data. This cost will be ongoing until business rules and algorithms are created to reduce the possibility of redundant data entries, and the redesign of the NIH Commons enables grantees to verify their own data.

Data Integrity -- the accuracy or correctness of the data in a database -- is not a significant problem for most of the eRA data. But in terms of personal data, it is a major issue indeed. A recent analysis showed that less than 1 percent of overall data in IMPAC II was inaccurate, whereas the error rate in personal data was roughly 8 percent. These errors create problems for reporting accurate statistics (e.g., support for new investigators, physician scientists, etc.) and tracking grantee award history (mistaken identity of an investigator could pose serious problems).

The problem of conflicting data goes back many years. Typos in names, social security numbers and degrees entered into the legacy system were never systematically corrected. The user community chose not to impose rigor in the IMPAC II system that would slow down production work. Therefore, the system allows users to enter a duplicate social security number or duplicate record when entering personal data. This practice exacerbated the inadvertent creation of duplicate profiles and conflicting data on existing profiles. One alternative is to require the user to investigate and verify any conflicting information before proceeding. The adoption of an appropriate algorithm could assist in data verification.

Background information and further explanation of data integrity issues is available from eRA.

ALERT -- Changes in Research Contract DATA for FY 2001

Users need to be aware that FY 2001 contract data may not be accessible in IMPAC this fiscal year. The NIH Office of Acquisitions Management and Policy (OAMP), Acquisition Management Committee (AMC), CIT, and eRA have worked together to find a solution, however, it requires several steps that may not be complete by October. The following provides background information to put the issue in perspective.

The Federal Acquisition Regulations require that every agency submit all acquisition data to the Federal Procurement Data System (FPDS) monthly. From 1969 through September 2000, NIH data was captured in the IMPAC I system, converted to the FPDS format, and transmitted to the FPDS via the Department Contract Information System (DCIS).

OMB levied some additional FY 2001 reporting requirements. It was decided that it would be cost prohibitive to modify IMPAC I to accept the new OMB requirements, and determined by the AMC Working Group that the best short term solution for NIH was to use the DCIS directly for all contract reporting until the New Business System is in place. Operating under the existing circumstances, a memo dated September of 2000 was issued by OAMP indicating that there would be no further entry of Research Contract information into IMPAC I. While this ensured that contracts was compliant with FPDS, it did leave two remaining items for the NIH to decide: how to get the data into IMPAC I or II for internal reporting purposes; and how to add the other fields required by the NIH and IC's for their reports.

At the request of OAMP, DCIS is being modified to accommodate over 40 additional data elements that need to be captured for NIH. The expanded set of contract data, including mandated information, will be housed in DCIS and made available to the NIH Data Warehouse and IMPAC II. The first step is for the contractors working on DCIS to complete their work. Subsequently, CIT and eRA staff will build the links needed to complete the process. Everyone is working hard to have this aspect of the process completed by the time the books are closed on this fiscal year. Upcoming editions of Inside eRA will keep you apprised of status and progress.

Also, this responsibility is moving to the New Business System (NBS). They will develop the tools needed by the community for contracts. For further information on the NBS please reference the NBS home page.

IMPAC II Support for Windows 2000

Windows 2000 became the "official" IMPAC II client development and production platform with the February IMPAC II Migration to Oracle 8i. All IMPAC II applications development and testing is now done in the Win 2000 environment. ICs are encouraged to upgrade IMPAC II users to the Win 2000 operating system as soon as conveniently possible. While there are no immediate issues known, future difficulties may arise if working with older operating systems.

In keeping with the migration plan agreed upon by CIT and other information technology management groups at NIH, OER supported the Win 95 platform until Win NT 5.0 (as Win 2000) proved sufficiently stable for NIH to move to that platform as an enterprise solution. While there is only one official platform, there are many machines on other platforms that still need support. For now, OER will work with ICs and attempt to support alternate platforms, however, the project cannot continue to support multiple platforms indefinitely.

New Architecture for NIH Commons

Plans for a NIH Commons Architecture Tool were recently vetted through the shared design team, Information Technology Management Board (ITMC), and the Extramural Program Management Committee (EPMC) and the eRA Project Team. The draft calls for a migration from a standard client/server technology to an N- tiered application architecture, and is due to be finalized soon. This multi-tiered architecture includes a Java-based platform and technologies utilizing Java 2, Enterprise Edition (J2EE). Two draft documents are available online: the J2EE technology overview and the platform and tools recommendations.

The draft plan was recently shared with the Commons Work Group. Comments from the Commons Work Group have been supportive and favorable:

"The two documents regarding NIH's proposed J2EE architecture are very good documents. I learned a thing or two by reading them."

"I have read the documents and think that you are on a good path."

"We have read them over and agree... trying to use vendor independent solutions and modularizing the components offers the most flexibility as we move forward."

"The Platform and Tool Recommendations paper made reference to trying to keep things 'vendor independent'. I believe that this is very important..."

Additional comments should be directed to Dr. George Stone, stoneg@od.nih.gov.