Detailed Information on the
Veterans Health Research and Development Assessment

Continue to refine meaningful and useful performance measures to assist VA in management.

Assess the physical condition of VA medical research infrastructure to determine its adequacy to support high-quality veteran-centric research.

Increase the number of research projects related to OEF/OIF veterans

Continue to emphasize the implementation of research findings by the Veterans Health Administration and publish significant findings in peer-reviewed scientific journals so that they can be adopted by other Federal and private health care systems.

Continue to monitor and ensure that research does not inappropriately duplicate other non-VA research activities.

Develop a streamlined process for collecting and analyzing regulatory approval information prior to release of research funds.

Measure: Program Assessment by the National Research Council

Explanation:The National Research Advisory Council (NRAC) is a distinguished advisory committee that consists of VA and non-VA clinicians and scientists who assess the likely effectiveness of the VA Research program in positively affecting the health status of veterans. The NRAC is charged with assessment of this program by ORD, consonant with the charter mandate that it "conduct analyses and develop reports or other materials as necessary." This assessment is based upon guidelines related to the scientific importance of the research, on veterans' health, and its impact on the public health of the general population. The NRAC is instructed to consider the appropriateness of the research to the VA healthcare mission, the balance of this research in terms of the burden of disease and the special responsibilities of VA in the areas of mental health, central nervous system injury, and deployment health. The program may be given evaluations of "Fully Successful," "Substantially Successful," "Partially Successful," or "Unsuccessful."

Measure: Percentage of research program costs allocated for administrative purposes.

Explanation:Administrative expenses include both the costs of operating the central Office of Research and Development (ORD) in Washington, DC, and the cost of operating Research Offices at VA medical centers. The administrative allocation to medical centers is provided by ORD through a specific budget allocation. Expenses incurred in Washington for operating ORD include salaries of program and review staff, travel expenses for members of scientific review panels and other advisory committees, building rental, and contracts for services provided by the Institute of Medicine, Association for Assessment and Accreditation of Laboratory Animal Care (AALAC) and the National Committee for Quality Assurance (NCQA). Although the cost of research compliance activities is increasing as a result of increasing regulation, ORD strives to reduce administrative expenses over time.

Measure: For multi-site clinical trials, the percentage of sites that achieve 100% of the number of patients needed per year according to the study design.

Explanation:When clinical trials are designed, statistical methods are used to determine the number of patients that must be included in the study in order to ensure useful results. Studies that require large numbers of patients typically recruit them from many different sites so that the studies can be completed in a reasonable amount of time, and so that results reflect the effects of many different patient care environments. VA, NIH, and other research organizations have found that multi-site clinical trials frequently do not recruit the planned number of patients within the time allotted each year. VA is not unusual in this regard. The study period must often be extended until enough patients have been recruited. This has two adverse effects: 1) studies that take longer than expected to complete cost more, since salaries for study staff must be paid over longer periods and costs of supplies, equipment and services typically increase, and 2) clinical benefits are postponed. In addition, if a substantial number of the clinical trial sites recruit far fewer patients than expected, the validity of the study results may be compromised. Although VA research findings are not compromised by this problem, since patient recruitment is extended as necessary to assure validity of results for all large VA clinical trials, the increase in expense results in funding of fewer clinical trials. Factors such as trial design, patient population, and disease influence the cost and duration of clinical trials. Thus, it is impossible to develop meaningful measures by which to simply compare the costs of conducting multiple large clinical trials with the costs that were expected (direct cost-efficiency measure). Measures that simply look at how well trials meet their patient recruitment targets overall do not make good tools for improving patient recruitment, because the number of patients recruited is reviewed as a total and the focus is not on the specific sites that fail to meet their annual targets. A surrogate efficiency measure that assesses recruitment for each trial at each study site is proposed. This will allow VA to intervene and take corrective measures, possibly terminating participation at a delinquent study site. VA data shows that while most of our sites contribute meaningfully to recruitment (giving VA an overall recruitment rate of approximately 80% of target), only a minority of individual sites actually reach 100% of the recruitment target. We will measure the percentage of study sites that reach 100% of the recruitment target for each year of each study. Each site for each study is counted separately for this measure. The measure is novel, and expected to be an effective management tool for improving recruitment and reducing clinical trial expenses.

Measure: Define characteristics associated with onset of post-traumatic stress disorder (PTSD), by characterizing by 2012 the changes that occur over time in a person's emotional, mental, and physical health after deployment to Iraq. Annual Targets: 2002--Plan, review and obtain approvals (VA and DoD); 2003--40% of baseline assessments of completed;2004--100% of baseline assessments completed; 2005--50% of first post-deployment assessments completed; 2006--100% of first post-deployment assessments completed.

Explanation:Prior to deployment with the Army to Iraq, baseline measures of health, emotional status, memory and thinking will be obtained from various types of units.. The baseline information will include demographic and medical history, stress exposure history, emotional distress and health perception, and objective neurocognitive performance. Neurocognitive performance will be assessed using validated tests from the Army's Automated Neuropsychological Assessment and the Neurobehavioral Evaluation System. Following deployment to Iraq, these same individuals will be retested on the same measures at different repeated long-term intervals. The repeated testing of these subjects provides a powerful study design to identify the changes over time. This is especially important because adverse consequences of deployment may not present for months following the deployment. Additionally, health utilization records and self-reported and objective stress records will be gathered. Together with available environmental exposure data, scientists will determine the long-term effects of the military and environmental consequences resulting from deployment to Iraq. The data will provide insight into the onset of PTSD, as well as other mental and physical health consequences that occur as these individuals move from the military to civilian/veteran status. It provides an excellent opportunity for VA scientists to work directly with DoD personnel in identifying the effects of war, which may last for decades. Results will inform new intervention and prevention strategies.

Measure: By 2012, integrate evidence-based collaborative care ("team treatment") models for common mental health disorders nationally in VA's care delivery system. Annual Targets: 2005--Initiate funding and steering committee for regional spread of depression collaborative care; 2006--Complete cost-effectiveness and business case analysis for collaborative care of depression; develop and test informatics tools to support collaborative care; 2007--Complete evaluation of collaborative care in 4 VISNs; identify tools needed to sustain and spread intervention.

Explanation:Mental health disorders such as depression, substance abuse, and stress disorders are highly prevalent and disabling in the VA population. Patients often resist treatment by mental health professionals, yet primary care clinicians are not equipped to manage their problems unassisted. Randomized trials have demonstrated the effectiveness of collaboration, including screening and assessment, structured symptom monitoring, and care management, between primary care and mental health providers. However, experience shows that changes to the healthcare delivery system are necessary to change treatment patterns. VA is conducting studies in which researchers partner with administrators and clinicians to implement and evaluate evidence-based collaborative care. Studies aimed at determining how system improvements in the treatment of mental disorders can be sustained and spread throughout VA are currently underway. Expansion of this research into other settings (e.g., VA contract clinic sites) as well as generalizing the approach for other mental disorders (e.g., stress disorders and substance abuse) is anticipated. The major questions in this initiative are: 1) what are the barriers and facilitators to implementing the collaborative care model? (This includes development of informatics tools as well as business case analysis.); 2) what factors drive sustainability and spread?; and 3) how can the partnership between researchers and VA decision-makers be made more effective? This work will expand VA's capacity to deliver the most effective mental health treatment to veterans and to enlarge our knowledge of how best to translate research findings into "real world" quality improvement. Although the goal for 2012 is to have the collaborative care model in place in all 21 VISNs, we envision ongoing monitoring and continuous quality improvement beyond that date based on evolving evidence that derive from VA's broad mental health research portfolio.

Measure: By 2012, develop and implement robot-assisted treatments/interventions as standard clinical practice for patients who have suffered neurological injury due to conditions such as stroke, spinal cord injury, or multiple sclerosis. Annual targets: 2004 - Review of BWS treadmill (S) study (BWSTT-S); review of BWS treadmill study (SCI) study (BWSTT-SCI); review of BWS treadmill (MS) study (BWSTT-MS). 2005 - Review of RULT multi-site Clinical Trial (RULT-S); initiate funding of BWSTT-S; initiate funding of BSWTT-SCI; Initiate funding of BSWTT-MS; initiate funding of RULT-S. 2006 - Begin Patient Enrollment - all studies. 2007 - 40% of required patients enrolled in RULT-S; 25% of required patients enrolled in BWSTT-S, BSWTT-SCI and BSWTT-MS. 2008 - 65% of required patients enrolled in RULT-S; BWSTT-MS study completed, final report prepared.

Explanation:

Measure: By 2011, develop treatments/interventions as standard clinical practice that prevent, and/or lessen the duration and severity of pressure ulcers that develop in veteran patients that are paralyzed or have limited mobility due to conditions such as spinal cord injury, limb loss, stroke, or multiple sclerosis. Research into spinal cord injury as well as other diseases that lead to paralysis of the lower extremities is a statutorily mandated function of VA research. Annual Targets: 2004--Anabolic steroid feasibility study initiated; Bion study funding initiated; anabolic steroid feasibility study completed; biomechanical testing of Bions in animals; 33% patients enrolled for Lavage Study.

Explanation:Many chronic diseases/conditions severely compromise tissue health, leading to tissue breakdown and the development of pressure ulcers (bedsores), which is a common complication of paralysis. Pressure ulcers can become infected and lead to infection, possibly resulting in death. The veteran population is at particular risk complication because of the high incidence of immobilizing conditions such as spinal cord injury. In addition to the decrement in quality of life of the patient, pressure ulcers are very costly. It has been estimated to cost in excess of $1.3 billion per annum to treat pressure ulcers in the USA, primarily due to the need for prolonged periods of hospitalization associated with many methods of treatment. Many medical treatment products have been developed that attempt to treat this problem by reducing applied pressure; however, the incidence of pressure ulcers remains unacceptably high. VA research is funding several studies looking at new approaches to mitigate the occurrence and severity of bedsores in the veteran population: 1) use of anabolic steroids to heal pressure ulcers, 2) use of a special washing (pulsatile lavage) treatment to heal pressure ulcers, and 3) the use of devices (Bion??s) placed into the muscle so that the same part of the body is not subjected to constant pressure. It is likely that one or more of these approaches will lead to better treatments for this disfiguring condition.

Measure: By 2011, develop and disseminate at least one new effective treatment for post-traumatic stress disorder (PTSD). Annual Targets: 2004--Planning and approval for large multi-site clinical trial of the drug risperidone (RT). 2005--Final patient evaluations completed for the cognitive-behavioral therapy (CBT) clinical trial; RT enrollment begins. 2006--CBT data analyzed; enrollment in the drug prazosin (PT) clinical trial. 2007--Findings from the drug divalproex sodium trial (DST) evaluated for large, multi-site trial planning

Explanation:PTSD is an anxiety disorder that can develop after a person has been exposed to a terrifying event or ordeal in which physical harm occurred or was threatened. PTSD related to combat exposure is a major concern i the health of the veteran population. In cases where veterans do not respond to initial treatment, symptoms (including nightmares, disturbing memories during the day, sleep problems and aggressive behavior) may persist for years. Therefore, effective relief of symptoms is needed. Veterans with military service related PTSD will be treated using at least four new approaches, including cognitive-behavioral therapy (CBT), the drug risperidone (RT),the drug prazosin (PT), and the drug divalproex sodium (DST). The well-validated Clinician-Administered PTSD Scale (CAPS) will be used as an outcome measure, and success will be defined as a statistically significant improvement in the CAPS score in the treatment group.

Is the program purpose clear?

Explanation: By statute, the Veterans Health Administration (VHA) has three core missions: healthcare, education, and research. Congress has mandated the Secretary of Veterans Affairs (VA) to "carry out a program of medical research in connection with the provision of medical care and treatment to veterans." In concert with the authorizing statute, the current mission statement of the VHA Office of Research and Development (ORD) is to "discover the knowledge and create innovations that advance the health and care of veterans and the nation." Furthermore, the VA research program is instrumental in helping VA attract high-quality physician specialists and nurses. This was documented in the 2002 survey of VA researchers (the most recent survey) in which a majority of researcher-clinician respondents indicated that they would not work for VA without research opportunities.

Evidence: (1) 38 USC Section 7303--Functions of VHA: Research programs. (2) The FY06 VA Research and Development (R&D) budget presents the research mission. (3) VHA Directive 1200, "Veterans Health Administration Research and Development" cites the authorizing legislation and mission. (4) VHA Vision 2020 (issued 4/03) cites research as a key mission of VHA and describes the program's focus on ensuring that the best science-based information is used to improve healthcare for veterans. (5) The National Survey of VA Researchers, 2002 contains survey data reflecting the importance to VA clinicians on their ability to conduct research in VA facilities.

Does the program address a specific and existing problem, interest, or need?

Explanation: VA conducts research in areas targeted to the unique health, treatment, and rehabilitation needs of the veteran population, as required by statute. ORD is organized into four services (Biomedical Laboratory R&D, Clinical Science R&D, Health Services R&D, and Rehabilitation R&D) that fund VA investigators for projects that address prioritized, veteran-focused research areas such as military and environmental exposures, prosthetics, and spinal cord injury research. Research is focused on improving the quality of life of veterans as identified by the Research Realignment Advisory Committee. This Committee established Designated Research Areas (DRAs) based on an evaluation of the types of diseases and conditions most important to veterans, as well as the importance of questions regarding healthcare delivery within the VA. DRAs are re-evaluated periodically by the ORD to assess their appropriateness.

Evidence: (1) 38 USC Section 7303--Functions of Veterans Health Administration: Research Program identifies the need for the VA research program and its link to healthcare for veterans. (2) The FY06 VA R&D budget describes the need for VA research in the context of its focus on health conditions prevalent in the veteran population. (3) The R&D Web site (www.va.gov/resdev) includes additional information regarding the focus of the research program, as do the Office of Research and Development Annual Report for 2003, and "Impacts, 2002-2003."

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: VA research focuses on the high-priority healthcare needs of veterans, based on the broad statutory mandate. The program targets veteran-specific issues such as Gulf War Illness, post-traumatic stress disorder, traumatic brain injury, and amputation. However, since VA also supports research on conditions in the aging veteran population, there is overlap with other Federal and private medical research efforts. For example, there is both overlap and collaboration with NIH on hepatitis C; head and neck cancer; the causes, distribution, and control of prostate cancer, and diabetes. However, some overlap is appropriate. Since researchers build upon previous findings, it is important to determine if those findings can be repeated. Studying the same health conditions and issues using different research approaches and different data sets helps validate the findings and provides additional insights. Overlap also provides competition and recognizes the fact that some findings can be more readily translated into patient care than others. For example, there are many studies on congestive heart failure and high blood pressure using the same groups of drugs in different ways and on different populations. In studies supported by VA, industry and/or NIH, ACE inhibitors, diuretics, A2 blockers, calcium channel blockers a and b blockers, vasodilators and/or nitrates have been compared to each other in various combinations or to "usual care". Out of these studies comes a syntheses of how to treat patients, individualize care (e.g. on the basis of race) and what to do if a patient can't tolerate a particular drug. Results of research sponsored by one organization or agency do not necessarily meet the needs of all other agencies. Direct and systematic replication of research is a crucial element of the research enterprise and critical to effective delivery of health care in VA.

Evidence: (1) Federal law (38USC7303) states that VA "carry out a program of medical research in connection with the provision of medical care and treatment to veterans??. Such program of medical research shall include biomedical research, mental illness research, prosthetic and other rehabilitative research, and health-care-services research." (http://frwebgate.access.gpo.gov/cgi-bin/getdoc.cgi?dbname=browse_usc&docid=Cite:+38USC7303). (2) NIH-funded research projects can be found at http://crisp.cit.nih.gov. (3) Research Design Explained: Lecture 3.1 at http:psy1.clarion.edu/mm/teach RDE/Lecture 31.html (4) John P. A. Ioannidis "Contradicted and Initially Stronger Effects in Highly Cited Clinical Research" JAMA 2005;294:218-228.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: VA conducts an intramural research program and does not have statutory authority to fund research outside VA facilities. All research proposals must address relevance to veterans' health issues to qualify for funding. The research program uses a multi-level approach that relies on VA and external, non-VA peer reviewers to ensure that the research program is efficient, effective, and scientifically meritorious. VA funds only about 20% of the proposals evaluated. Thus only the most qualified proposals are funded. The majority of review panel members are researchers from outside VA who represent major research universities. Scientific review assesses the quality of proposal, significance of research, and the principal investigator's past accomplishments. This process is consistent with the NIH and other scientific peer-review processes. Funding of a project is determined by peer review and availability of funding. These multiple layers of checks ensure scientific quality and financial soundness of funding decisions. The National Research Advisory Council (NRAC) provides advice to the Secretary on the nature and scope of research, including the scientific merit review process. Two thirds of VA investigators??most of whom hold appointments at highly respected universities??perform research that is supported by NIH or other non-VA source. Thus, the VA investigators who conduct research under this program are of an exceptional caliber.

Evidence: Federal law (38USC7303) establishes VA research as an intramural program which is to be conducted in connection with the provision of medical care and treatment to veterans. 38USC7303 further requires that this research includes biomedical research, mental illness research, prosthetic and other rehabilitative research, and health-care-services research. The FY06 VA budget (volume 2 of 4, pages 9-5) identifies the requirement for scientific peer review of all VA-funded proposals. Documents that provide detailed information about the peer review committees and processes include: (1) Overview of Scientific Technical Program Advisory Boards, Office of Research and Development (prepared for the Secretary of Veterans Affairs); (2) Federally chartered committee charters (http://www.facadatabase.gov); and (3) information about peer review meetings in FY04 (http://www.facadatabase.gov). NRAC charter and meeting

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: The VA research program makes a significant impact by focusing all VA research resources on veteran-related health issues and illnesses. This is accomplished through several mechanisms. First, as part of the scientific peer review process, proposals are evaluated for clinical relevance specifically to VA healthcare and veterans' healthcare needs. Second, the research portfolio is evaluated annually to ensure that funded research is meeting the needs of veterans. Analyses are performed using Designated Research Areas based on prevalence of medical conditions within the VA patient population and the disease burden to VA healthcare. These analyses have consistently demonstrated that the VA research appropriation is targeting veteran-specific health/healthcare issues. Third, the research portfolio is also reviewed by the National Research Advisory Council to ensure consistent relevance to veterans' healthcare needs.

Evidence: Research program activities and their benefit to the veteran population are discussed in a number of sources: the FY06 VA R&D budget; the R&D Web site (www.va.gov/resdev); the Office of Research and Development Annual Report for 2003; and "Impacts, 2002-2003."

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The VHA R&D program has several long-term measures that seek to further the purpose of the program and specifically to develop new knowledge and translate this knowledge into effective clinical practice in VA medical care. First, VA proposes to develop more effective treatments for post-traumatic stress disorder (PTSD) by 2012. PTSD is an important and costly veterans' health issue. Improved treatment will improve quality of life and societal function while reducing health care costs. Annual progress will be tracked by following initiation, patient enrollment, and completion of clinical investigations required to develop and validate these new treatment methods, and development of treatment guidelines. Second, implementation research based on new mental health treatment paradigms will explore more effective methods by which to implement new evidence-based treatment models throughout VA. Annual targets will track research on implementing new mental health treatment approaches initially in smaller numbers of VA hospitals and outpatient clinics, then in progressively larger groups. Development of these new implementation paradigms in parallel with the development of new treatments for PTSD will assure that the benefits of the PTSD research rapidly change clinical care. Third, VA will develop more effective methods for restoring limb function in those with brain and spinal cord injuries. This goal is to improve the lives of our most severely injured veterans. This is an ambitious goal for achieving significant advances to dealing with a nearly intractable problem. Annual progress will be tracked by following initiation, patient enrollment, and completion of clinical investigations required to develop and validate these new treatment methods, and development of treatment guidelines. Fourth, the goal of developing effective new interventions to decrease the number of "pressure ulcers" that bed and wheelchair bound veterans experience represents a commitment to improve treatment for a debilitating, expensive and sometimes lethal, but common, condition affecting millions of Americans and costing over a billion dollars annually. Annual progress will be tracked by following initiation, patient enrollment, and completion of clinical investigations required to develop and validate these new treatment methods, and development of treatment guidelines. These performance measures reflect the veteran-centric nature of VA research and represent a challenging standard for evaluation. The presentations and publications that result from this work, as for all VA research, will be widely disseminated through scientific and clinical meetings, publication in high-quality peer-reviewed scientific journals, and as appropriate into VHA Practice Guidelines that affect care provided to veterans nationwide.

Evidence: The FY07 VA budget identifies specific measures. The outcomes and the strategic targets reflect 4 to 6 year timeframes. Background material is found in research proposals and progress reports maintained by the Office of Research and Development.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The ORD performance measures reflect an aggressive program for improving the understanding and treatment of post-traumatic stress disorder, together with the implementation of an evidence-based framework allowing the results of this research, and other new evidence-based mental health treatment paradigms, to be effectively implemented in the VA system. The timeframe for improving limb-function following nervous system injury is highly ambitious and aggressive. Finally, finding more effective measures by which to prevent or treat pressure ulcers has been a goal that has invited repeated failures throughout the worldwide medical research community for many years. The timeframe for changing this history is extremely ambitious.

Evidence: The FY07 VA budget, and research proposals for the projects upon which the performance goals are based, identify targets that will require highly effective program management to achieve within the proposed timetable.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: VA has three annual performance measures. The first measure is based on the program assessment by the National Research Advisory Council (NRAC), an advisory committee of VA and non-VA clinicians and scientists who assess the effectiveness of the VA Research program. The second measure is an efficiency measure for multi-site clinical trials of the percentage of sites that achieve 100% of the number of patients needed per year according to the study design. The third annual measure is an efficiency measure of the ratio of research program costs for administrative purposes to total program costs. In addition, each long-term goal has specific and measurable annual targets that represent incremental steps toward accomplishing the long-term goal. The annual targets represent building blocks for achieving a goal within a specific timeframe. For example, clinical trials are designed by scientific and statistical experts based on knowledge and principles of clinical research methodology, together with deep understanding of both the condition to be treated and the care system in which treatments will be delivered. Clinical trial design thus considers the number of patients which must be recruited to complete the trial, and the timeframe over which recruitment can be accomplished in the VA research system. Patient recruitment and the completion of individual research studies provide excellent benchmarks by which to track progress towards achieving the long-term goals.

Evidence: FY07 VA budget. VA project proposals; Reviewer Reports; Investigator progress reports from FY04 and FY05; Minutes of Cooperative Studies Program Merit Review Board, May 11, 2005; Cooperative study profiles; Cooperative study pharmacy coordinating center annual report, July 2004; ReTIDES Initial Steering Committee Minutes, December 2004.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The program has three annual measures: all have baselines and ambitious targets. Baseline performance for clinical trials is assessed as part of "usual care" arms during clinical trials. Use of historical baselines for the assessment of clinical trials is not considered acceptable by the scientific community, and is inferior to properly designed controlled clinical trials. Thus, VA has used the appropriate methodology for assessing baseline treatment performance. The annual performance measures that provide a framework for tracking long-term performance are appropriate and aggressive.

Evidence: VA project proposals; Reviewer Reports; Investigator progress reports from FY04 and FY05; Minutes of Cooperative Studies Program Merit Review Board, May 11, 2005; Cooperative study profiles; Cooperative study pharmacy coordinating center annual report, July 2004; ReTIDES Initial Steering Committee Minutes, December 2004.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: VHA policies and solicitations convey the goals of the program to research investigators. Long-term and annual performance goals are reflected in objectives for individual Merit Review, Research Enhancement Award Program (REAP), and Career Development projects. Investigators who successfully compete for Career Development Awards must sign a statement of agreement with the conditions of the award. The expectations for REAPs are provided to awardees and posted on the VA Web site. Governance training for VA Non-Profit Corporation (NPC) board members provides individual board members roles, responsibilities, and functions. Cooperative Technology Administration Agreements (CTAAs) describe the mutual expectations of VA and the University in inventions resulting from research. As cited in VHA policy, research productivity (in the form of scientific publications) is a major criterion for ongoing project funding.

Evidence: (1) VHA Handbook 1202.1 para. 3.b. (www1.va.gov/resdev/resources/policies/docs/1202_Merit_Review_Handbook_JIT.doc), (2) VHA Handbook 1202.3 para 1 (vaww1.va.gov/resdev/resources/policies/docs/Edited-Career-Development-Handbook.pdf), (3) VHA Handbook 1204.1 Appendix A.6.c(6) (www1.va.gov/resdev/resources/policies/docs/Handbook-1204.1-draft.doc), (4) VHA Handbook 1203.1 Appendix B.11 (www1.va.gov/resdev/resources/policies/docs/1203-1.pdf), (5) VHA Handbook 1200.18 para. 5 (www1.va.gov/vhapublications/ViewPublication.asp?pub_ID=403), (6) VHA Handbook 1200.19 para. 2 (www1.va.gov/vhapublications/ViewPublication.asp?pub_ID=361). (7) REAP Program solicitation (vaww1.va.gov/resdev/funding/solicitations/docs/blrd-csrd-reap-0304.pdf). (8) Governance Training for NPC Board Members, (9) Career Development working group minutes, (10) CTAA template (www.vard.org/tts/iia/iia.htm), (11) Executive Order 12591

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: "Independent evaluations are provided by several Federally chartered advisory committees: (1) The National Research Advisory Council (NRAC) makes recommendations on the nature and scope of VHA's R&D program. The NRAC is made up of eminent clinicians and biomedical scientists (overwhelmingly non-Government); it meets at least twice a year. (2) Scientific Technical Program Advisory Boards consist of subject-matter experts who review research proposals to determine scientific merit. (3) The Research Advisory Committee on Gulf War Veterans' Illnesses reviews research on Gulf War-associated illnesses. These committees operate as another component of the ""people's voice"" in our government; they are often created by Congress to ensure that legislative mandates are meeting their goals. (4) Other organizations that provide independent evaluations are the GAO, OIG, the National Committee for Quality Assurance (NCQA), and the American Association on Accreditation of Laboratory Animal Care (AAALAC). (5) The latest Congressional submission (FY06) states that ""the VA Office of Policy, Planning, and Preparedness is working with the Office of the Secretary and the VA Administraions to clarify which programs will be evaluated over the next several years."""

Evidence: (1) Federally chartered advisory committees (www.fido.gov/facadatabase/default.asp). (2) Minutes of the NRAC. (3) Government Accountability Office (GAO) reports (www.gao.gov). (4) Office of Inspector General (OIG) report (www.va.gov/oig/51/FY2004rpts/VAOIG-03-03053-115.pdf). (5) NCQA (www.ncqa.org/programs/qsg/vahrpap/vahrpap.htm). (6) AAALAC (www.aaalac.org).

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The budget request reflects the broad scope of all VA research. The request presents the budget in two ways: First, the budget request is tied to four broad programmatic areas of research: biomedical laboratory, health services, rehabilitation, and clinical research. Second, the budget presents the funding request by designated research areas such as mental illness, cardiovascular disease, aging or trauma. Funds for individual research projects or groups of projects are not explicitly outlined in the budget request. Within the programmatic areas outlined above, funds are allocated to individual research activities through a peer-reviewed competition based upon VA-directed calls for study and investigator-initiated requests. This process assures that the programmatic goals, including the performance goals, are addressed in the most scientifically sound manner by the most qualified investigators within the VA system. The budget request does not link to the performance goals for funding.

Evidence: FY06 President's Budget Submission, VHA Directive 2001-014, and GAO Report 03-10.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: Research strategic planning is an ongoing activity, which is integrated with the strategic planning efforts of the VA and the Veterans Health Administration (VHA). The 2007 VHA strategic plan, which is in the process of finalization, calls upon VHA to increase collaboration between research and clinical care delivery. The Health Services Research and the Quality Enhancement Research Initiative Model (QUERI) will be used to improve VA health care delivery. In addition, the plan calls upon VA to increase its involvement in the research and practice of genomic medicine, in collaboration with other government activities. Finally, the plan calls for VA to improve knowledge of the effects of military service on health care. Some of the performance measures implemented by the research program are linked with these VHA strategic goals. Strategic planning is an ongoing activity with operational consequences. A previous Chief R&D Officer convened several "blue ribbon panels" to review various aspects of the program in 2003, and some recommendations of those panels have already been carried out: reorganization of the career development program, splitting of Medical Research Service into Biomedical Laboratory R&D and Clinical Science R&D services, and release of QUERI "Special Solicitation for Service Directed Projects on Implementation of Research into Practice." Additionally, the Chief R&D Officer and the research service directors have collaborated to identify their "top ten action items" for the VA research office, with the intent of laying the groundwork for the new Chief R&D Officer to engage in formal strategic long-term planning. These items have been linked to the goals set forth in VHA strategic planning efforts, in which VA leadership has taken an active role, and with objectives set by the National Leadership Board. The result of these efforts will be to strengthen the important role that VA research has played in putting VA "ahead of the curve" in providing state of the art medical treatment to veterans. Just as in the past VA research has led to VA leadership in effective treatment of tuberculosis, diabetes, depression, bipolar disorder and prostate disease, to name a few, current research is expected to lead to exceptional care for veterans suffering from PTSD, limb loss, central nervous system injury and other diseases associated with military service today. Research aimed at improving implementation of new research findings will lead to even more rapid adoption of new clinical tools, with commensurate benefit for the veteran and the American public.

Evidence: VHA Strategic Goal 4 (draft 7/8/05). Final Reports of the VA Office of Research & Development Blue Ribbon National Advisory Panels, August 2003. PowerPoint presentation: "Regaining the High Ground, Strategy to Reinvigorate ORD." Moving from Strategy to Action "12 Priorities - 12 Months" (VHA Logistics Action Plan). VHA National Leadership Board Strategic Plan, 2003-2008. Three new R&D performance measures.

Has the agency/program conducted a recent, meaningful, credible analysis of alternatives that includes trade-offs between cost, schedule, risk, and performance goals, and used the results to guide the resulting activity?

Explanation: ....

Evidence: ....

If applicable, does the program assess and compare the potential benefits of efforts within the program and (if relevant) to other efforts in other programs that have similar goals?

Explanation: VA research projects are funded by a peer-review and approval process. Each merit review committee receives multiple applications and ranks them on scientific merit and relevance to the veteran population. The committees only approve a portion of these proposals, and they consider alternative methods and objectives for each study. Once studies are approved for funding, VA staff review each approved study in the context of similar studies in that research area, with general guidance from advisory panels and in light of available funding. Similar competitive processes occur for capacity development initiatives, such as research centers and training awards to individual researchers. All of these processes are highly competitive, and most projects are only funded after multiple submissions. In response to the question about research data, several VA databases have been made available to the public.

Evidence: "VHA Directive 1200 provides information on the mission and common policies of the four research services. In addition, it provides information on shared principles among the services, including prioritization of research proposals on the basis of scientific merit, fiscal responsibility, and high standards of scientific integrity. Information on the Vietnam Era Twin Registry can be found at www.eric.seattle.med.va.gov. Information on the ALS registry is available at www.va.gov/durham/alsregistry.asp. Information on VA Health Economics Resource Center can be found at www.herc.research.med.va.gov. Information on VA Information Resource Center is available at www.virec.research.med.va.gov."

Does the program use a prioritization process to guide budget requests and funding decisions?

Explanation: In accordance with the statutory mandate of 38USC7303 highest priority is given to research into spinal-cord injuries, other diseases that lead to paralysis of the lower extremities, and injuries and illnesses particularly related to service. High priority is also given to biomedical, clinical and health services research directed towards maladies afflicting the VA patient population, as reflected in the seventeen Designated Research Areas (DRAs) approved by the Research Realignment Advisory Committee. On a year-to-year basis, specific budget priorities are set as a part of the budgeting and strategic planning processes. These funding priorities are highlighted in the budget request, and are also reflected by specific requests for application (RFAs) that are issued to stimulate research in focused areas, such as Gulf War Illnesses. After specific projects have been proposed by VA investigators, the priority for funding each proposed project also reflects the assessment of scientific merit by review committees. Research is funded in a manner that maximizes both research productivity and impact on clinical care, as intended by the statutory requirement (38USC7303) that the VA research program be conducted in connection with the provision of medical care and treatment to veterans. Only projects of the highest merit are approved by the research director for new/continued funding. All VA project applications are competitively reviewed by external and VA experts for quality, program relevance, and potential performance outcome. Promising concepts for new areas of research are reviewed and approved by the National Research Advisory Council. Projects are assigned a priority score based on scientific merit to guide funding decisions. Depending on the specific field of study, different funding mechanisms are used. For instance, if a field is newly emerging, VA might fund a research Center of Excellence to jumpstart the field. A VHA Government Performance and Results Act (GPRA) goal sets the percentage of research projects devoted to the 17 DRAs at a target level of 100%. These DRAs are considered the high priority research areas for the veteran population. Budget submissions are guided by the GPRA goal, but focused by the need to more tightly address emerging veterans' needs in such areas as post-traumatic stress disorder and influenza.

Evidence: The FY06 VA budget, VHA R&D Strategic planning framework and research portfolio documents. National Research Advisory Council charter and minutes. 38USC7303. Designated Research Areas working paper dated February 1, 1998. These designated research areas were revised in the FY02 budget submission to more completely reflect the nature of VA research.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: Performance data from key partners are collected by means of annual research project reports and applications for competitive renewal, which are reviewed by VA portfolio managers and Scientific Review Committees, respectively. Funding is terminated if progress is unsatisfactory. Annual reports from VA Non-Profit Corporations are used to inform oversight by the Non-Profit Oversight Board. Patent disclosures are monitored at the VISN and facility level. VA research findings are discussed by the Research Subcommittee of the Health Systems Committee of the National Leadership Board (NLB). Internal and external oversight committees that review performance data and recommend program changes include the ORD Directors' staff meeting, the Career Development Working Group, the Field Research Advisory Committee (FRAC), the National Research Advisory Council (NRAC) and the Business Oversight Board (BoB).

Evidence: Annual reports of projects and programs, research portfolio databases, the Consolidated Annual Report, Guidelines for Scoring of Proposals for Merit Review; Outline of Scientific Review, QUERI Review Instructions and Checklists for QUERI Center Strategic Plans; Evaluation Guidelines for Research Career Scientists; and Career Development Program Evaluation Criteria provide evidence of the collection and use of performance data. Non-Profit Oversight Board meeting minutes, NLB Subcommittee meeting minutes, ORD Directors' staff meeting agendas, FRAC minutes, NRAC minutes, and Career Development Working Group minutes all provide evidence of the use of performance data to manage the program and improve performance.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: Portfolio managers review progress reports submitted by investigators. Project expenditures are monitored by research fiscal staff. The VHA Chief Financial Officer reviews monthly reports. The awardee and local Associate Chief of Staff for Research are held accountable for expenditure and progress reporting on a schedule cited in VHA policies. Annual reports are required for each VA-funded study. Continuation of funding is dependent on satisfactory annual progress reports. In addition, each award has a specific duration; thus, the investigator must go through the merit review process again to receive renewal funding. Through this process, the performance results of the previously funded study are evaluated.

Evidence: Research Principal Investigators' performance agreements are used to measure individual researchers' results. Progress reports and the feedback sent from ORD to researchers describe necessary action to be taken to address shortcomings. Some examples that document performance criteria include: Guidelines for Scoring of Proposals for Merit Review; Outline of Scientific Review, QUERI Review Instructions and Checklists for QUERI Center Strategic Plans; Evaluation Guidelines for Research Career Scientists; and Career Development Program Evaluation Criteria. Reviewer Reports, Merit Review Entry Program Midterm Review, and Research Enhancement Award Program (REAP)/Center Annual Progress Reports are documents that provide feedback on progress of individual awards.

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: All VHA R&D funds are consistently obligated for items and services that fall within the intent of the appropriation or fund. Funds are obligated in a timely manner. No Anti-Deficiency Act violations have been reported.

Evidence: Financial data are used to assure that funds are obligated. Examples of financial reports include: End-of-Month Status of Allowance, Monthly Activity by Account Classification Code, and Biweekly Unobligated Balances by Veterans Integrated SErvice Network (VISN) and by Station.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: Efficiency of a medical research program can be measured by its ability to achieve results economically. The scientific value of the individual research projects is evaluated at initiation and at time of competitive renewal by using expert review panels consisting primarily of non-VA scientists. These panels may recommend that an investigator's project be discontinued, or that it be funded at a reduced level. They may also recommend that a project be redesigned to assure effectiveness in obtaining the outcome design, then to evaluate the new design prior to recommending funding. High quality science is rewarded; the funding of poor quality science is reduced or eliminated. For example, in the last year the Health Services Research and Development Service eliminated one of its Centers of Excellence, while the Biomedical Laboratory Research and Development Service reduced the number of ongoing Research Enhancement Award Programs from 34 to 19. Together, these actions saved upwards of three million dollars annually, for investment in more promising projects. These decisions are based upon assessments of the productivity and scientific quality of the work produced - the "bang for the buck." The VA also increased the efficiency of the research services. For example, the Biomedical Laboratory and Clinical Science Research Services have consolidated their review activities to increase efficiency, and studies are underway to determine if the remaining review activities can also be consolidated. Research services assess the productivity of review staff (proposals/evaluated per staff member), the length of time between receipt of a proposal and a decision on funding, and the overall administrative overhead required to sustain high quality evaluation and execution of the research program. Clerical support activities in support of these review activities have largely been outsourced through FedSource, a Treasury Department activity that assures efficient competitive sourcing through competitive mechanisms. Finally, the VA has adopted an approach to funding individual hospital research offices in a manner that has reduced their cost from 7.8% of overall research expenses in FY00 to 6.5% in FY04. This has enabled VA to increase the resources devoted to compliance activities, such as human subjects protection, without increasing the overall costs of program operation. VA also assesses other measures associated with clinical trial execution, such as the "overhead costs" for the clinical trial and success or failure of individual facilities in meeting research participant recruitment targets. Results of these assessments result in changes in program execution. For example, VA has determined that many protocols do not reach the patient recruitment targets, extending the time for project completion and increasing the cost. VA has begun to fund studies VA has determined that many protocols do not reach the patient recruitment targets, extending the time for project completion and increasing the cost. VA has begun to fund studies conducted through this program via a "capitation" mechanism in which funding is provided as individual patients are recruited into a study. Studies which have been funded through this "capitation" mechanism have been shown to more frequently meet patient recruitment targets than studies in which funding is not tied to meeting recruitment goals. Therefore, new CSP studies will be funded through the capitation mechanism, and additional measures will be developed to improve trial planning and patient recruitment.

Evidence: The FY06 R&D Budget; VHA Handbooks, 1200 Series; OMB and VHA Office of Policy and Planning Web sites; E-PROMISE Project Management Information System Requirements Specification; and VA R&D Merit Review Process Task Order Report.

Does the program collaborate and coordinate effectively with related programs?

Explanation: VHA's R&D program has collaborative arrangements with diverse organizations that share its research interests, including government institutions (NIH Institutes and Centers, Department of Defense, Centers for Disease Control and Prevention, Centers for Medicare and Medicaid Services, Dept. of Health and Human Services (HHS), Agency for Healthcare Research and Quality (AHRQ), Medical Research Council of the UK, Canadian Institutes of Health Research), non-profit organizations (including 92 VA-affiliated non-profit research and education corporations), and private industries (leading pharmaceutical and biotechnology companies). For example, the TRIAD study, a joint initiative of the Department of Health and Human Services and the VA, documented that veterans received more evidence-based interventions compared with commercial managed-care patients. In addition, joint studies of the VA and National Institute on Drug Abuse have resulted in FDA approval of two drugs for the treatment of opiate addiction.

Evidence: (1) Partnerships are listed in the Office of Research & Development Annual Report for 2003, pages 9-12 (www1.va.gov/resdev/resources/pubs/docs/annrpt03.pdf). Hard copies of information about collaborations with HHS and AHRQ are available. (2) Cosponsored clinical trials can be found on ClinicalTrials.gov (www.clinicaltrials.gov). (3) Information about VA-affiliated non-profit research and education corporations is available at www.vanpc.org/accomplishments.aspx and links from that page. (4) Results of the TRIAD study were published in the Annals of Internal Medicine (www.annals.org/cgi/reprint/141/4/272.pdf). (5) Joint solicitations include an RFA on amyotrophic lateral sclerosis (grants.nih.gov/grants/guide/rfa-files/RFA-NS-04-003.html) and the VET-HEAL RFP (www1.va.gov/resdev/funding/solicitations/docs/VET_HEAL_RFA_020705.doc). (6) The Annual Report to Congress on Federally Sponsored Research on Gulf War Veterans' Illnesses 2002 can be found at www1.va.gov/resdev/resources/pubs/docs/GulfWarRpt02.pdf.

Does the program use strong financial management practices?

Explanation: VA's strong financial management is demonstrated by unqualified audits over the past 6 years. Few federal agencies have obtained unqualified audits. VA's innovative financial management has saved millions through initiatives such as the purchase card program for small purchases and the prime vendor system for pharmaceuticals. VA's strong fiscal controls allocate appropriations and reimbursable Federal funding down to the lowest operational levels (e.g., funding for career development is determined, resources are allocated to each facility, and costs are captured against this same category). The new VHA cost accounting system--Decision Support System--allows VHA to more accurately track indirect research support by narrowly categorizing programs when tracking expenditures. ORD improved financial management by placing an administrative officer in each research service. ORD is planning a financial oversight discussion for the upcoming Society of Research Administrators (SRA) conference.

Evidence: ORD has implemented weekly status of funds reviews at each station and nationally to monitor the effective use of appropriated funds. Annual Performance and Accountability Report 2004 is accessible at www.va.gov/budget/report. Report of Audit of VA Consolidated Financial Statements for Fiscal Year 2002 - 1999. The Non-Profit Oversight Board meets quarterly. At its last meeting, the board approved the framework of a strategy for increased non-profit oversight. VA Directives, Handbooks, and manuals can be accessed at www.va.gov/publ/direc/finance/finance.htm#han.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: Deficiencies have been identified through past GAO reports (such as 2003 report on VA's need to improve human subject protection) and reports from the Office of the Inspector General (OIG). In response to the OIG report of March 2004, steps have been taken to correct deficiencies in finance and personnel practices as well as use of non-profit organizations to administer extra-VA research funding. The proposed reorganization plan currently under review is designed to correct human resources issues identified by the OIG. VHA established a Non-Profit Oversight Board to ensure appropriate management of the VA research and education corporations. ORD also initiated non-profit board member governance training. In the area of human subjects protection, the accreditation program with the National Committee for Quality Assurance (NCQA) is running smoothly, and field research programs are meeting requirements. There is also an active educational program underway in human subjects protection.

Evidence: Response to OIG report (confidential file). Non-Profit Oversight Board meeting minutes (July 2004, November 2004) and meeting agenda (March 2005). A training manual for non-profit governance is available on the VA R&D Web site. List of VA field research programs accredited by the VA/NCQA Accreditation Program. Memorandum from Under Secretary for Health, subj: VHA Office of Research and Development Reorganization.

Is the program managed by maintaining clearly defined deliverables, capability/performance characteristics, and appropriate, credible cost and schedule goals?

Explanation: R&D needs are unique, specific, and clearly defined, as are the deliverables, the performance characteristics, costs, and schedule goals. R&D projects are funded using competitive peer review. Deliverables, schedules, performance characteristics, etc. are detailed as part of the research protocol. All VA ORD research awards and capacity-building awards such as the Career Development Award have specific durations. Awardees and program staff agree on milestones that must be achieved on an annual basis. Portfolio managers review progress reports. Awardees are notified if progress is unsatisfactory, and funding is terminated in the absence of satisfactory improvement. Contracts are used to obtain goods or services are monitored by a Contracting Officer's Technical Representative in accordance with Federal Acquisition Regulation Part 4 and VA Acquisition Regulation Part 801.

Evidence: Documentation is included in Federal Acquisition Regulation Part 4 and VA Acquisition Regulation Part 801 and VHA 1200 series Handbooks. Examples that document performance criteria include: Guidelines for Scoring of Proposals for Merit Review; Outline of Scientific Review; Scientific Peer Review Process in Research; QUERI Review Instructions and Checklists for QUERI Center Strategic Plans; Evaluation Guidelines for Research Career Scientists; and Career Development Program Evaluation Criteria. Reviewer Reports, Merit Review Entry Program Midterm Review, and REAP/Center Annual Progress Report are documents that provide feedback on progress of individual awards.

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: The Merit Review process uses a formal and highly competitive peer-review process in which the proposed work is reviewed by subject-matter experts. Investigators must use a standardized application, with standardized detailed written instructions on merit scoring, budgeting, and application timelines. In addition, there is advanced training on research design that VA research funds at national meetings, and through formal mentorship programs and Career Development Awards. The peer-review committee votes whether to approve or disapprove the proposal and a priority score is assigned to each approved proposal based on criteria, including scientific merit, originality, significance of the research, feasibility, and contribution to the health needs of veterans. Proposals are funded based on the priority score, the available funding, and programmatic needs.

Evidence: VHA Directive 1200 provides for prioritization of research proposals on the basis of scientific merit, fiscal responsibility, and high standards of scientific integrity. VA publishes directives and guides to provide guidance on the merit review process, the standardized proposal format and standard procedures.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: All research activities are conducted by VA employees, resulting in a better ability to monitor research activities. VA researchers must submit an annual report detailing scientific progress and financial activity and progress during the past year. All VA ORD research awards and research capacity-building awards, such as the Career Development Awards, have specific durations. For the research awards, investigators must provide milestones that are expected to be achieved on an annual basis in the research proposal. For capacity-building awards, midterm reports are required to determine if satisfactory progress has been made.

Evidence: Information is released through VA fact sheets and VHA Handbook 1200.19, the FY06 VA R&D budget, the Research and Development Information System (RDIS) Annual Report. The Office of Research and Development Annual Report for 2003 (www.va.gov/resdev).

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: An annual report of research activities is required of all medical care facilities where research activities are conducted. Performance data are collected on both VA and non-VA funded research. The Research Program disperses information through the media, newsletters, special publications, information campaigns, information for VA leaders and Congress, and special events. VHA Handbook 1200.19 establishes procedures for presentation of research results in peer-reviewed journals and other national and international venues, and assigns responsibilities and specifies authority for ensuring that the contributions of the VA to the research community are acknowledged and publicly disclosed. An annual report of research accomplishments is published, widely distributed, and is posted to the VA internet site. The publication of research findings provides the best way to share important information with the public and scientific community and allows for an independent assessment of researchers' findings and importance of the research.

Evidence: Information is released through VA R&D Communications fact sheets and VHA Handbook 1200.19, the FY06 VA R&D budget, the Research and Development Information System (RDIS) Annual Report. Office of Research and Development Annual Report for 2003 and VA R&D internet site.

For R&D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain program quality?

Explanation:  

Evidence:  

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: VA has demonstrated significant progress towards its long-term goals. Although the performance goals in this document are new this year, they reflect ongoing programs with previously established goals and annual targets. The neurocognitive evaluation of soldiers deployed to the Persian Gulf is meeting its deadlines, as are the studies of new therapies for post-traumatic stress disorder and for the treatment of pressure ulcers. The initial results for therapies aimed at improving limb function in those with central nervous system damage are equally encouraging. These results are not surprising given the success of VA research in meeting challenging long-term goals in the past. For example, large multi-site clinical studies of drugs to treat heart rate irregularities, of drugs to treat schizophrenia, of laparoscopic hernia repair, and of coronary artery surgery which were recently published in the New England Journal of Medicine and other major medical journals were each originally planned in the 1990s, as was a study of a new vaccine for shingles. The smallest of these studies involved a few hundred patients, while the largest required following nearly 40,000 patients for 3 years.

Evidence: VA project proposals; Reviewer Reports; Investigator progress reports from FY04 and FY05; Minutes of Cooperative Studies Program Merit Review Board, May 11, 2005; Cooperative study profiles; Cooperative study pharmacy coordinating center annual report, July 2004; ReTIDES Initial Steering Committee Minutes, December 2004. R. Rosenheck et al. "Effectiveness and Cost of Olanzapine and Haloperidol in the Treatment of Schizophrenia," JAMA 2003; 290: 2693-2702. EO McFalls et al. "Coronary-Artery Revascularization before Elective Major Vascular Surgery," New England Journal of Medicine 2004; 351: 2795-2804. L Neumayer et al, "Open Mesh versus Laparoscopic Mesh Repair of Inguinal Hernia," New England Journal of Medicine 2004; 350: 1819-1827. M.N. Oxman et al, "A Vaccine to Prevent Herpes Zoster and Postherpetic Neuralgia in Older Adults" New England Journal of Medicine 2005; 352:2271-2284

Does the program (including program partners) achieve its annual performance goals?

Explanation: Goal #1: PTSD Treatment - met goals of beginning risperidone treatment trial and completing final evaluations of cognitive-behavioral therapy trial patients. Goal #2: Define characteristics associated with onset of PTSD - met goal of completing 50% of first post-baseline assessments. Goal #3: Integration of Research Evidence into VA Mental Health Delivery - Met goal of initiating funding and creating steering committee for regional spread of depression collaborative care. Goal #4: Develop improved treatments for restoring limb function in those who have suffered nervous system injury - met goal of restoration of limb function following nervous system injury. Goal #5: Develop improved treatments for pressure ulcers - met goal of initiating a large study of anabolic steroid use, developing an implant procedure for Bion?? technology, and enrolling 70% of required patients for lavage study.

Evidence: VA project proposals and progress reports; Reviewer Reports; Investigator progress reports from FY04 and FY05; Minutes of Cooperative Studies Program Merit Review Board, May 11, 2005; Cooperative study profiles; Cooperative study pharmacy coordinating center annual report, July 2004; ReTIDES Initial Steering Committee Minutes, December 2004.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The program has demonstrated a decrease in the average time between proposal submission and decision making in the medical research services, from 163 days in FY01 to 143 days in FY04. At the same time, the number of proposals evaluated has increased from 935 proposals in FY01 to 1,353 in FY04, with no change in staffing. The program has adopted measures that focus on improving the efficiency of central R&D management, as well as management of the research offices at individual VA medical treatment facilities. The fraction of the research allocation (direct appropriation plus transfer from medical care) that goes to support the general administration of research offices throughout the system has been steadily reduced, from 7.8% in FY00 to 6.5% in FY04. This has enabled direction of increased resources to human subjects protection education and oversight.

Evidence: Quarterly VHA Financial reports; Cooperative study profiles; FY05 and FY06 Cooperative Study Coordinating Center Budgets; Overview of Scientific Technical Program Advisory Boards, Office of Research and Development (prepared for the Secretary of Veterans Affairs).

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: VA's R&D program compares favorably to NIH's intramural research program. For FY04, the number of publications per doctoral-level researcher was 1.2 for VA and 1.3 for the NIH intramural program. This is important as a measure of disseminating research results and findings to the field and likelihood of impact on population health status and new discoveries. The number of new inventions per researcher was 0.09 for VA and 0.06 for NIH. The VA investigator base of 2020 doctoral researchers has produced two Nobel laureates - a ratio similar to that of the NIH intramural program's 6000 investigators giving rise to 5 Nobel prizes. However, NIH's intramural program has a relatively higher number of National Academy of Sciences and Institute of Medicine members. Because the process for reviewing VA intramural research proposals is similar to that for reviewing NIH extramural proposals, we have also compared VA research with this program. VA's program also compares favorably to NIH's extramural research program. For FY04, the number of publications per project (research, center, and career development grants) was 1.3 for VA and less than 1.1 for NIH's extramural program. An attempt was made to compare the VA intramural research program with various other programs. While a number of U.S. agencies (e.g., CDC, DOD, NSF) and non-profit organizations (e.g., Howard Hughes Medical Institute, Gates Foundation) and some foreign agencies (e.g., Medical Research Council, Karolinska Institutet) have similar missions, we have been unable to obtain performance metrics to enable comparison.

Evidence: Some of VA's accomplishments are noted in the R&D and QUERI Web sites (http://vaww1.va.gov/resdevand www.hsrd.research.va.gov/queri), in VA R&D impacts documents (www1.va.gov/resdev/resources/pubs/pubs_individual.cfm?Category=Impacts) and annual reports (www1.va.gov/resdev/resources/pubs/pubs_individual.cfm?Category=Annual%20Reports), and independent GAO, OIG, American Association on Accreditation of Laboratory Animal Care, financial data budget submissions and other reports. Internal report produced in collaboration with the NIH intramural program.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: A report by the RAND Corporation concluded that VHA patients received higher quality care (research report published in Annals of Internal Medicine). An article in the July 18, 2005, issue of U.S. News & World Report entitled "America's Best Hospitals" praises the VA quality of care while showing that the high quality results from VA research, which is the underpinning of this clinical excellence. Indeed, health services research is the basis of the "constant examination of the processes and procedures that go into caregiving," systems, and rehabilitation and clinical research systematically examine specific interventions. Examination of that issue of U.S. News & World Reports (July 18, 2005) underscores the importance of research in providing top-quality patient care. Virtually every hospital listed as having high quality in both the general and specialty lists is the site of a highly active program encompassing both clinical and biomedical research. The research program uses the National Research Advisory Council (NRAC), which reports to the Secretary and is composed of eminent clinicians and biomedical scientists external to the VA., to provide an external evaluation of VA research efforts. At the June 2005 meeting, the NRAC expressed confidence in the peer-review system used by VA to evaluate research proposals and applications for continuation. The NRAC also commented favorably on the quality of research conducted by VA. An evaluation of the VA Non-Profit Corporations by the GAO cites many significant accomplishments of VA research that have had wide-ranging public impact. Finally, the impact of the program is evident from the many awards and recognitions received by VA investigators, publications in high-impact journals, and positive reports on inspections of VA facilities by the National Committee for Quality Assurance (NCQA) and the American Association on Accreditation of Laboratory Animal Care. The NCQA program for accrediting VA human subjects protection programs has itself been commended by the Institute of Medicine. VA research has created the standard of care for many diseases. A few examples: VA research revolutionized the treatment of tuberculosis in the 1940s, leading to the disappearance of once-ubiquitous TB sanitariums. In the 1950s and 1960s, the VA cooperative studies program developed the essentials of the randomized controlled clinical trial that is the standard for clinical research today. In the 1960s the VA invented the immunoassay, a type of laboratory test that is used to detect proteins such as prostate-specific antigen (PSA) and is now a mainstay of clinical laboratory testing through the world. VA research proved the value of the coronary artery bypass operation in the 1960s and early 1970s. In the 1960s and 1970s, VA also proved the value of lithium and antidepressants in the management of depression and bipolar disorder. VA proved the value of one-a-day aspirin in patients with atherosclerosis during the 1980s, and demonstrated the importance of maintaining control of blood sugar, together with the safety and efficacy of the implantable insulin pump in adult-onset Type II diabetes mellitus during the 1990s. Recent advances have included demonstrations of the effectiveness of non-surgical therapy for gastroesophageal reflux disease and prostate enlargement, demonstration of the value of advanced cochlear implants in veterans with profound hearing loss, development of new treatments for post-traumatic stress disorder which are useful in veterans and non-veterans alike, and has developed a cognitive-behavioral therapy that improves symptoms of those with Gulf War Illnesses. In recent months, VA showed that an experimental vaccine for shingles cuts its incidence in half and dramatically reduces severity and complications in those that develop the disease.

Evidence: RAND Report: http://www.rand.org/publications/RB/RB9100 (2005). Research Report: Annals of Internal Medicine 141: 938-945, December 2004. GAO report: VA Health Care: Nonprofit Corporations Enhance VA Research, but Would Benefit from Increased Oversight. GAO-02-1103T September 19, 2002 http://www.gao.gov/new.items/d021103t.pdf. NRAC charter: http://www.facadatabase.gov. IOM report on Gulf War (1999): http://www.iom.edu/report.asp?id=5600. Cooperative Studies Program web site: http://www1.va.gov/resdev/programs/blrd-csrd/csp.cfm Experimental vaccine for shingles: New England Journal of Medicine, June 2, 2005. http://content.nejm.org/cgi/reprint/352/22/2271.pdf Institute of Medicine Report: Preserving Public Trust: Accreditation and Human Research Participant Protection Programs (HRPPP) April 2001. http://www.iom.edu/report.asp?id=4872 U.S. News & World Report July 18, 2005 issue, "America's Top Hospitals": http://www.usnews.com/usnews/health/best-hospitals/tophosp.htm. NIH Awards Trends: http://grants1.nih.gov/grants/award/trends/rnk04all1to500.htm There is strong evidence that quality of care (assessed by a variety of measures) correlates with research activity in a hospital setting. For example, 13 of the 16 "top hospitals" in the US News report are among the top 20 institutions in terms of NIH research funding. Conversely, 13 of the 20 institutions that receive the most NIH funding are to be found in the US News top 16. Given that there are more than 7500 hospitals in the United States (http://answers.google.com/answers/threadview?id=516668)and 3181 institutions receiving NIH funding (http://grants1.nih.gov/grants/award/awardtr.htm), the association is striking (p<0.0001). There is also evidence that care conducted in teaching and research hospitals is substantially better than ordinary care using a variety of individual specific indicators. Kupersmith J, "Quality of care in teaching hospitals: a literature review." Academic Medicine 80:458-66, May 2005.

Were program goals achieved within budgeted costs and established schedules?

Explanation: VA sets quarterly goals for personal services, equipment and all other categories of expenditures for the research appropriation. Actual expenditures versus planned expenditures are reviewed monthly. VA research activities typically meet budget and time schedules. Capital assets for the VA R&D program are primarily capitalized research equipment. Capitalized equipment requests are subject to scientific peer review prior to purchase. Capital asset management following purchase is under the purview of VHA as documented under VHA Memorandum 10-2002-01. The R&D program capital asset management is achieved within budgeted costs and established schedules.

Evidence: Financial data. Performance Plans and reports. VHA Memorandum 10-2002-01 (February 26, 2002) "VHA Consolidated Memorandum Receipt (CMR) Accountability Property Process" outlines frequency of inventory, assignment of designated officials and disposition requirements for VHA equipment. VHA Handbook 1202.6 describes the Shared Equipment Evaluation Program.