The QVR System—An Example of CIT–eRA Cooperation

How familiar are you with the Query/View/Response (QVR) System at NIH? Most likely you’ve heard the name in connection with grants but are perhaps not sure what this system encompasses. Then keep reading: The QVR system is a successful information technology tool supported by both CIT and the eRA project working together, and this article not only summarizes its development and its features but also offers a special focus on how the QVR team identified and responded to the needs for information relating to the grant applications in the IMPAC system.

The grants process and the IMPAC system—background in a nutshell

NIH has a budget in excess of 28 billion dollars per fiscal year. By far the largest portion of that budget is obligated in the form of grants for biomedical research conducted at institutions across the United States and around the world. NIH receives and processes over 88,000 grant applications per year.

These grant applications are complicated, detailed, and extensive documents. They contain administrative information, budgetary information, details on research protocols, diagrams, photographs, charts, and graphs. They vary in length from around fifty pages to upward of several hundred pages or more. Prior to 2006, all of this documentation was submitted in the form of paper documents. Now the NIH grant application process is transitioning from paper to electronic submission.

Most of the applications are for NIH funding but there are other agencies that use the NIH system, including AHRQ (Agency for Health Care Research and Quality), FDA (Food and Drug Administration), SAMHSA (Substance Abuse and Mental Health Services Administration), and the CDC (Center for Disease Control). Almost all of the applications go through extensive receipt, referral, and review processes, with only the best ones eventually being funded.

The IMPAC system is the repository for data on ALL applications submitted to NIH for processing. In fact, the IMPAC system has data on applications and funded grants stretching from the present all the way back to 1970. As indicated above, NIH provides the receipt, referral, and review processes for several other government agencies. In fact, it is certain this service will be expanding in future years to other participating federal agencies. As the grants business process moves forward, numerous data fields are populated and, in many cases, modified. One can only imagine the vastness of the data available in this system.

The key question is how to access the data that provides the knowledge for the business processes to move forward for each application and grant. It involves much more than simply retrieval of information on individual projects. Because the grants process is the major part of the budget of NIH and because NIH is the focal point for public health research in the United States and, indeed in the world, there is a tremendous need to be able to analyze the grant portfolio to identify breakthroughs in science, as well as gaps that may need to be filled by targeted research.

How does NIH handle all of these myriad requirements? The answer is vastly more complicated than the extent of this article but, put simply, the answer is the IMPAC system and the various information tools that are available to staff. Read on.

The QVR system

One current example—the training profile:

The following example typifies the teamwork, cooperation with the user community, and rapid development process that is the QVR system. In the fall of 2006, there were discussions at the QVR Steering Committee meeting about the need for a retrieval and analysis function for trainees on NIH training grants. As the idea was further explored, it was discovered there existed a substantial number of records in the IMPAC database on the trainees on training grants. At the same time, there were discussions between the QVR team and NIH staff involved in “program evaluation” within the ICs about creating an IT tool to capture information on these training records. Taking this nascent concept, the QVR team moved ahead to explore how this could be done.

Concurrently, activists in the program evaluation community—Sarah Glavin and Janet Guthrie—went ahead with an application for evaluation funds for the project. Shortly after funding was secured, the QVR team produced a prototype retrieval and analysis tool—the Training Profile. It was presented to the newly formed Training Steering Committee for their comments and suggestions. Within weeks it was presented to a wider audience of training program directors for their input. Again, within weeks, the Training Profile-beta version was put into production. Since that time, formal training classes have been offered on this new QVR feature, and the user reception has been tremendously positive. Throughout this brief development cycle many features were changed and improved based on user suggestions.

What do we have? The Training Profile has access to over 450,000 records on trainees. It provides the ability to assess the training history of current applicants and grantees. It also offers the ability to evaluate the application and grants success of trainees from as far back as 1980. In addition, there were also three Standard Reports created that allow for detailed analysis of the trainees on various types of training programs.

The QVR developmental paradigm

Development of the Training Profile has by no means been deemed complete at this point. Nor has the QVR team ever really finished with development and refinements of the other user profiles. The QVR development process takes advantage of every-day users to continually refine and expand existing systems. In fact, in addition to the direct feedback from the steering committee and users, QVR’s extensive user logs are also utilized. The QVR team tracks ALL users of the system. Special attention is paid to power users—those who use the system frequently or submit complex searches. In addition, all error messages are tracked.

QVR team members frequently contact these individuals to help them solve the search problem they have encountered but, more significantly, we discuss with them how they are using the system: What about the search screen is confusing? What was the cause of error in their search? Team members then often steer the conversation to other topics that help them further refine the system. What are the users trying to do? What will these data be used for? Where are they positioned in the grants business process? The QVR team also takes the opportunity to point out features of the system that may relate directly to the part of the business process users are involved in. The model for these conversations is to solve the problem and learn from the user.

It could be argued that this is a very inefficient use of developer time and effort. However, this approach accomplishes two very important things: First, it brings the developers into a direct relationship with the end-users of the system. There is no one trying to “interpret” user needs to the developers. From these direct contacts the team has received invaluable user guidance for further development of the system.

Secondly, there is a less visible but more important outcome. Each member of the development team REALLY understands the details of the business process used in conjunction with the IMPAC database. This allows them to use their creativity more effectively in future development efforts. Indeed, many of the enhancements to the system have come directly from the creative minds of the development team members—often based on a few, disconnected suggestions from users. It brings real excitement to the system.

The QVR system—the nuts and bolts of the system:

The QVR system has a wide range of features. Here is a brief summary of the major features:

User Profiles: There are four different user profiles in the system. Each profile provides for retrieval of information that relates to business processes at NIH and the other federal organizations that use the system. They allow users to create and retrieve information in a variety of formats.

Main Profiles: The main QVR user profiles allow access to information on all application/grant records in the IMPAC system for the last seven years (currently that includes fiscal years 2007 to 2000). The Main profile has three search screens available. The system started with what is now called the Main (Original) search screen. That screen has over 70 search items organized into logical sections (PI, Budget, Review/Program, Output/Sort, etc.). In response to customer needs, the Main (New) search screen, which brought the most frequently used search items to the top of the search screen, was developed. In addition, a Quick Search screen that is designed for new users who need to have only a limited number of search items available was added to the system. As the system developed and as users suggested new ideas, three new user profiles have been developed. Each of these is briefly highlighted below.

Person Profile: This profile is designed to allow staff to identify scientists in various fields of research. It has a database of over 275,000 records and is searchable on the basis of a wide range of criteria, including name, country, scientific expertise, previous grant history, degree, etc. The hitlist that is retrieved allows the user to examine more detailed information through links to the selected record. For example, one can retrieve the entire grant history for a person, their degree/expertise, committee history, etc.

History Profile: This profile allows users to access IMPAC data from 1970 to the present. There are two main reasons that this is a separate profile:

1. As the IMPAC system evolved over the years more and more data items were incorporated into it. Thus, many of the items do not “track” completely through all years of data.
2. The entire IMPAC database is massive. In order to make the QVR system respond to searches quickly, it is necessary to partition along logical search lines.

It was determined that most users needed information from the most recent seven years of data. For those needing to search deep into IMPAC history this profile provided that capability.

Training Profile: This is the most recent addition to the QVR system. It is mostly covered in the example above but it is worth reinforcing the scope of this single undertaking. It provides access to a database of over 450,000 records on trainees (most of the “T”, “K” and “F” funding activities). One may search current applicants/grantees and retrieve their previous training experience. Also, users may search previous trainees and determine their subsequent application/grant activity. In addition, it has three standard reports that allow for detailed analysis of data relating to the trainees.

Hitlists: The system offers several hitlists that are customized to user needs. The basic hitlist provides one line of data. The Basic Hitlist (Detailed) provides much more information and gives several options for sorting the data. These hitlists have links to the application image, Notice of Grant Award, summary statement, snapshot, grant abstract, grant history, and PI history. In addition, there are links that open an email screen to either the PI or Program Manager on the project. There is also a “Summary” hitlist that is designed to provide several options for downloading and printing summary statements.

In addition to the hitlists, the system also offers a custom download and a Standard Report output that offers a wide range of formatted reports.

Custom Download: QVR has the ability to download data to a spreadsheet or local database from a list of over 300 data items sourced from IMPAC and the Data Warehouse. This feature is used extensively by extramural staff.

Standard Reports: QVR provides 48 Standard Reports to users. These reports are designed for Scientific Program managers, Grants Management staff, Review staff, Budget Offices—essentially every category of position among extramural staff. All of these reports are the product of suggestions by QVR users.

PubMed: The system has a link to PubMed based on the PI name or the grant number. This allows staff to evaluate the productivity of particular grantees as well as the productivity of particular projects.

IC-Specific codes: The QVR team has developed a production system in conjunction with the SCS On Demand project, which allows institute staff to query on the basis of that ICs own scientific codes as well as the full range of search parameters available through the QVR screens. This greatly expands the capability of QVR and is being demonstrated to other interested ICs.

Initiative Tracking and Checkbook System: The QVR team has worked with NIDDK to provide a series of additional features that are IC-specific. The initiative Tracking and Checkbook System accesses data stored in a NIDDK local system and merges it with IMPAC and Data Warehouse data available through QVR. It allows NIDDK staff to track their scientific initiatives, create payplans, and generate reports that facilitate managing their scientific portfolio.

Pre-processed grant applications: The QVR system pre-processes the scanned image of the grant application and converts it to a text file. This procedure allows the system to extract specific sections of the grant application for use by staff.

One transforming event and what the future holds

It seems the sky is the limit in many respects. There are new opportunities available wherever you look. Here is just one of them that has already happened. There are many others and more are being realized as we move forward.

Transforming event— A look back at a seemingly simple event that vastly changed the grants process illustrates the power of transforming events and suggests future IT applications. In the beginning of 2002, it was decided to capture the scanned image of all grant applications, bookmark the file accordingly, and make it available through the IMPAC system.

Prior to that time, if a staff person from an IC that did not have primary assignment of a particular application wanted a copy of the application, they would have to request a copy from the primary IC. This was all too often a drawn-out process because the time-consuming work of copying and sending the requested applications created quite a large backlog of requests for the ICs involved. With the scanned image (and the QVR system) all IC staffers now had to do was click on the “IMG” link and the full text of the applications appeared in a PDF format—information was immediately available throughout the NIH and among all other agencies using the QVR system.

The eRA is on the verge of completing the transformation from paper applications to electronic submissions. Having access to all applications in electronic format will open many doors for information and analysis.

In summary, we are in a very exciting time in information technology at the NIH. The fusion of the grants process and IT has huge potential. Stay tuned.