Draft Review Criteria for Competing Renewal Applications - National Center for Research Resources

The goals of NIH-supported research are to advance our understanding of biological systems, to improve the control of disease, and to enhance human health. In their written critiques, reviewers will be asked to comment on each of the following criteria in order to judge the likelihood that the proposal will have a substantial impact on the pursuit of these goals. Each of these criteria will be addressed and considered in assigning the overall score, weighting them as appropriate for each application. Note that an application does not need to be strong in all categories to be judged likely to have major scientific impact and, thus, deserve a high priority score. For example, an investigator may propose to carry out important work that — by its nature — is not innovative but is essential to move a field forward. All components of the unified CTSA application, (i.e., all related sections of the single overall application that are received as a unit) will be awarded a single overall priority score determined after evaluation of all components.

Performance in Prior Project Period

Competing renewal applicants should include a "Background" section of up to 20 pages of text and 20 pages of Tables (40 pages in all) summarizing the achievements of the CTSA in the previous project period. Reviewers will be asked to assess the impact that the CTSA has had on the quality and extent of clinical and translational science at the applicant institution and within the local and regional communities. Were the resources of the CTSA distributed equitably amongst specialties and NIH-funded researchers at the applicant institution in the previous project period? Was Pilot Project funding, if offered, accessible and effective? Did progress toward measurable goals and milestones, outlined in the previously funded grant application, meet prior expectations? Did the applicant identify potential problem areas and avoid them when necessary or apply timely corrective measures where avoidance was not possible? Was the institutional commitment appropriate to create an integrated "home" for clinical and translational research? Was the environment favorable to attract new clinical and translational researchers, assist their transition to independent funding, and ensure that their contributions were recognized within the institution's promotions and tenure program? Was the CTSA able to assist researchers in bringing projects from the bench to the bedside, and from proof of principle to clinical application? To what degree has the translated research been accepted and disseminated by the community? Does the research being translated address the needs of the community?

Reviewers will also be asked to assess the impact of the CTSA in the previous project period on health research communities outside the awardee institution. These communities could include local organizations and regional affiliations as well as national clinical and translational research communities. Impact in this context will include research priorities identified by the CTSA Consortium Oversight Committee and other CTSA-wide Steering Committees, workgroups, and taskforces that have positively affected health or healthcare delivery.

Plans for Next Project Period

In developing plans for the next project period, applicants are encouraged to demonstrate continuity in their programs, such that strengths in the previous project period are identified and facilitated while challenges are recognized and addressed. Reviewers will be asked to assess four overall areas:

Significance: Will the CTSA significantly impact the overall quality of clinical and translational science at the applicant institution? Will the overall program vision and strategy sustain a home for clinical and translational science that incorporates a wide range of clinical disciplines, specialties, and sub-specialties? Will the CTSA have potential to contribute significantly to a national consortium of CTSAs?
Approach: Will the CTSA program enhance, complement, or extend the applicant's current resources for clinical and translational science research? Will the Center, Department, or Institute (C/D/I) include relevant scientific disciplines to maximize productivity? Does the application make efficient use of potentially unique resources, such as access to certain human subject populations or the provision of pre-clinical resources? Does the applicant indicate how the organization will be adapted to respond to changes in translational focus? Will opportunities for careers in clinical research be sustained across the spectrum of clinical and translational science? If significant changes have been proposed from the focus of the previously funded period, are the changes justified and has a plan for transition been included with this application? Are plans for participation in the national consortium clearly addressed and appropriate to the CTSA program? Are there adequate resources identified for participation in the national consortium?
CTSA Governance: Have the applicants described an effective administration and governance structure that will promote the discipline of clinical and translational science? For applications designating multiple PD/PIs, is the leadership approach, including the designated roles and responsibilities, governance, and organizational structure consistent with — and justified by — the aims of the CTSA and the expertise of each of the PD/PIs? Is the governance structure designed to ensure both accountability of multiple PIs and integration of the components of the C/D/I into a coherent program? Will the leadership and governance plans accommodate changes in the direction of research and allow for the efficient use of funds? Is there an Advisory Committee to provide critical, stimulating, and thoughtful advice for the overall CTSA performance and CTSA key functions? Are there plans to resolve conflicts and implement recommendations? Is the institutional support clearly demonstrated in the governance structure? Has the administration and governance structure been significantly restructured from the previously funded program? If so, has this restructuring been justified and are adequate transition plans included with this application?
Implementation Plans: Does the implementation plan adequately address the transition from the previously funded grant period? Is the timeline for implementation feasible and are specific goals and milestones set? Are alternatives proposed should the goals and milestones not be reached in a timely manner? Is there a feasible time line for integrating CTSA resources with other complementary resources available to the institution?
Integration: How well have the components of the CTSA been integrated with each other? Are there plans to further integrate CTSA activities into all the relevant schools and clinical research sites that participate in clinical and translational science in the applicant institution? Is there a commitment to integrate the CTSA into the institution and into a national network of CTSAs and also to reach out to the local community? Will this integration be reflected in the senior leadership and decision-making processes of the CTSA?
Local and National Collaboration, Data Sharing, and Dissemination: How adequately will the institution and its researchers collaborate, share, and disseminate resource tools and resources at institutional, community, and national levels? Are plans included to address regulatory hurdles locally? Is there a commitment to and plans for adopting and implementing national standards?
Innovation: Will the CTSA program create new and innovative opportunities? Are approaches proposed that would develop the integration of clinical, basic, and other relevant (e.g., public health, bioinformatics) disciplines? Does the program develop or employ novel concepts, approaches, methodologies, tools, or technologies that will improve the discipline? Is the program likely to develop or sustain novel approaches to increasing the ease and efficiency of clinical and translational research, allowing research results to move from patient observations and laboratory discoveries to the bedside and to clinical practice?
Investigators: Have there been changes in the PD/PI(s) or key personnel during the first project period? Are the PD/PI(s) and other key personnel appropriately trained and well suited to carry out this work? Is the work proposed appropriate to the experience level of the PD/PI(s)? Do the PIs have the experience and authority and committed time to administer the institutional home for clinical and translational research? Does the PI have sufficient authority and credibility in the institution to work across institutional boundaries? Does the PI have the environment and institutional support necessary to be responsible for the resources committed by the institution(s) for the C/D/I? Do the program leadership and management team bring complementary and integrated expertise to the project? Does the C/D/I have the professional staffing to impact significantly the overall quality of clinical and translational science at the institution? For key functions that are included in the proposal, do key function Directors have the appropriate training, experience, and resources for their leadership roles? Have the Directors of the key functions committed sufficient time to this program? Will the Directors have the authority at their institution to implement best practices identified by National Consortium Steering Committees? Are the administrative and professional staff appropriately trained and well suited to carry out this work?
Environment: Does the applicant have a track record of awarding higher degrees in clinical research? Will the academic and scientific environments sustain a home for clinical and translational science? Is there a strong training record at both institutional and faculty levels? Does the proposal provide strong evidence that the CTSA will provide resources to the institution that would not otherwise be possible? Has there been an effort made for inclusion of all health related schools and scientific areas? If applicable, are there adequate cooperative arrangements between affiliated institutions to ensure that the CTSA program performs effectively as one activity across institutional boundaries? Are there unique features of the scientific environment or in the available human subject populations or collaborative arrangements?

Institutional Commitment: What was institutional commitment to the CTSA program during the first project period? Will the institution align or adjust incentives and rewards to promote the academic mission and new modes of team-based research? Is the institutional leadership committed to protect the time of the investigators to pursue clinical and translational research and mitigate the demands of providing patient care? Will clinical research be supported in terms of a specific tenure process for clinical researchers at the institution? Is the institutional leadership committed to this program and its goals in terms of providing specific assets for the program (e.g., financial support, faculty support, specific equipment, dedicated space, or tuition rebates)? Will existing NIH-supported cores be appropriately shared with the CTSA program?

Key Function Areas

In addition to the above criteria, the following components of the CTSA application will be considered in the determination of the overall priority score for the entire U54 application. Where significant changes are proposed to key functions developed during the previously funded period, reviewers will be asked to address the proposed changes with respect to the local environment and to consider the plans for transition from the previous program. Has the selection of key functions and their scope for the next project period been well justified?

Development of Novel Clinical and Translational Methodologies: Is there an active program of research in novel methodologies? Is the outcome likely to benefit the C/D/I? Is there a plan to involve new investigators? Will these activities be integrated with the CTSA as a whole?

Pilot and Collaborative Translational and Clinical Studies: Is there an adequate plan to solicit proposals, to prioritize the projects, and to review their methodology and research performance? Will the expected benefits to the CTSA and to the wider research community be measured and tracked? Will lessons learned be shared?

Biomedical Informatics: Will the biomedical informatics resources offered be commensurate with the breadth of the CTSA program? Will data security and privacy be safeguarded? Are assessments of performance of this resource included? Will the Biomedical Informatics Director have the necessary authority to the ensure implementation of best practices as adopted by the Biomedical Informatics Steering Committee? As applicable, will this resource be sufficient for intra- and inter-institutional operations? Will the institution be willing to work toward interoperability of the informatics systems and adopting national data standards?

Design, Biostatistics, and Clinical Research Ethics: What types of support and resources will be in place to ensure all clinical and translational research designs are sound and that statistical analyses are appropriate and rigorous? Will this training include conflict of interest, federal codes requirements, guaranteeing privacy and safety of research participants, especially as pertaining to vulnerable populations? Are there plans for creation and innovation in developing the application of these topics to clinical research? As applicable, will this resource be sufficient for intra- and inter-institutional operations?

Regulatory Knowledge and Support: Will this resource provide "researcher-focused" support for regulatory compliance and management? Is the resource well integrated with biomedical informatics and participant and clinical interactions? Is there duplication of IRB responsibilities? Does the institution have experience in working with multi-site trials and with the FDA with respect to studies involving investigational new drug application procedures? Will CTSA staff members be available with the necessary experience in working with the FDA and in ensuring that standards for reporting adverse events are met? Are criteria for the research subject advocate functions sound?

Participant and Clinical Interactions (PCI): Will human subject participation in clinical research protocols be encouraged? Will the institution work with underserved populations in clinical research? Has the applicant adequately described and justified the resources to be provided? Will PCI resources meet the highest standards for subject safety, quality of science, and statistical and ethical design? Is the application of Good Clinical Practice guidelines appropriate? Will resource utilization be tracked, and are mechanisms proposed to adapt resources to the needs of investigators? Will the resources provided serve small — as well as large — studies or trials?

Community Engagement and Research: Will this effectively involve the community in which the CTSA institution resides, both the public and practitioners, in clinical and translational research priority setting, participation, and follow-up? Are there adequate plans to train researchers, trainees, and scholars in the methodology of community/population-based research and outreach? Will the resource foster long-term bidirectional relationships between the CTSA institution and the community for their mutual benefit? Will the research interests of C/D/I faculty contribute to an intellectually stimulating environment?

Translational Technologies and Resources: Is the plan to identify technologies to be offered appropriate? Will resource utilization and evaluation be adequate? Is their flexibility in types of resources to be offered? Will faculty members be encouraged to pursue research in areas that develop translational methodologies?

Research Education, Research Training and Research Career Development: Do the research education, training, and career development components strengthen the training and career pathways for all clinical and translational research professionals and team members? Can increased efficiency shorten the period of training? Does the institution have a sufficient pool of academically strong trainees and commensurate experienced and well-qualified mentors to justify the career development pathways that are proposed? Will the curricula and courses proposed provide appropriate training in clinical and translational research relevant to a broad range of specialties? How will trainee registration for higher degrees in clinical research be encouraged? Has the Program Director committed adequate time to program administration?

Research Education Component: Review considerations include the quality, innovation, and content of courses and adequacy of the syllabus; the scientific qualifications and experience of the faculty; the criteria for selecting participants; efforts to publicize the availability of the program to potential participants.

Research Training Component (TL1):

Research Training and Career Development Record: This criterion evaluates the past research training and career development record (both prior to the CTSA and during the first CTSA project period) of both the T and K programs and the designated mentors. How successful are former trainees in seeking further career development and in establishing productive scientific careers? Evidence of further career development can include successful completion of the Ph.D., receipt of fellowships, career development awards, additional training appointments, and similar accomplishments. Evidence of a productive scientific career can include a record of successful competition for research grants, receipt of special honors or awards, a record of publications, receipt of patents, promotion to scientific positions, and any other measure of success consistent with the nature and duration of the training received. What is the track record of the mentors in directing training and career development or the potential of those mentors lacking a track record?
Mentored Career Development (KL2) and Research Program Design: This criterion will assess the likelihood that the career development and research programs will contribute significantly to the scientific development of the candidate scholars to successfully pursue clinical and translational research careers. Reviewers will evaluate whether the plans include appropriate course work and/or activities to achieve program goals. The scientific quality, technical merit, and degree of relevance of the research projects the scholars will be pursuing will be evaluated.
Training (TL1) and Career Development (KL2): In addition to the above criteria, reviewers will be asked to comment on the following criteria for both the TL1 and the career development (KL2) programs.
Preceptors/Mentors: This criterion assesses the caliber of mentors as researchers, including successful competition for research support in areas directly related to the proposed research training or career development program. What is the overall quality of mentors' research, their publication record, and their successful competition for research support in areas directly related to the proposed training and career development programs? How strong is their record as mentors?
Institutional Environment and Commitment to the Program: For the training program, this criterion assesses the quality of the institutional training environment for NRSA-supported trainees and the relationship of the NRSA program to the broader training program (if appropriate). What is the level of institutional commitment, quality of the facilities, availability of appropriate courses, and the availability of research and research training support? Does the environment in which the training program will be conducted (i.e., the quality of the participating departments and the extent of their participation) contribute to the probability of success? Is there evidence of adequate institutional commitment? For the career development program, this criterion will assess the applicant institution's commitment to the program, such as recruitment efforts, necessary educational resources and equipment, available established investigators who will serve as mentors, and evidence that scholars will have sufficient "protected" time to devote to the program.
Trainee Recruitment, Selection, and Retention: This criterion evaluates the quality of the applicant pool and the plan for selection of individuals for appointment to the training or career development program. Specifically, what is the quality and size of the applicant pool? Are the recruiting procedures, trainee selection criteria, and retention strategies appropriate and well defined?
Evaluation and Tracking of Research Education, Research Training, and Research Career Development: Is the plan adequate to determine progress and outcome measures for each of the research education, training, and career development components? Does it include a system for tracking participants following program completion to determine success or failure of the program? The tracking would include information on program publications, grant proposals and awards, and career trajectory of the trainees that were supported in the program. If an external advisory committee is proposed, are plans adequate and appropriate to ensure proper monitoring of the research education, training, and career development components? Are there means to modify the research education, training, and career development components based on appropriate recommendations from the external advisory committee?
Training in the Responsible Conduct of Research: Peer reviewers will assess the applicant's plans for training in the responsible conduct of research on the basis of the appropriateness of topics, format, amount and nature of faculty participation, and the frequency and duration of instruction for the research education, training, and career development components. Plans will be discussed after the overall determination of merit, and the review panel's evaluation of the plan will not be a factor in the determination of the priority score. Plans will be judged as acceptable or unacceptable. The acceptability of the plan will be described in an administrative note on the summary statement. Regardless of the priority score, applications with unacceptable plans will not be funded until the applicant provides a revised, acceptable plan. Program staff will judge the acceptability of the revised plan.
Recruitment and Retention Plan to Enhance Diversity: The NIH recognizes a unique and compelling need to promote diversity in the biomedical, behavioral, clinical, and social sciences workforce. The NIH expects efforts to diversify the workforce to lead to the recruitment of the most talented researchers from all groups; to improve the quality of the educational and training environment; to balance and broaden the perspective in setting research priorities; to improve the ability to recruit subjects from diverse backgrounds into clinical research protocols; and to improve the Nation's capacity to address and eliminate health disparities. Accordingly, the NIH continues to encourage institutions to diversify their trainee and faculty populations and, thus, to increase the participation of individuals currently underrepresented in the biomedical, clinical, behavioral, and social sciences such as: individuals from underrepresented racial and ethnic groups; individuals with disabilities; and individuals from socially, culturally, economically, or educationally disadvantaged backgrounds that have inhibited their ability to pursue a career in health-related research. Institutions are encouraged to identify candidates who will increase diversity on a national or institutional basis. Peer reviewers will separately evaluate the recruitment and retention plan to enhance diversity after the overall score has been determined. Reviewers will examine the strategies to be used in the recruitment and retention of individuals from underrepresented groups. The review panel's evaluation will be included in an administrative note in the summary statement. If the diversity recruitment and retention plan is judged to be unacceptable, funding will be withheld until a revised plan (and report) that addresses the deficiencies is received. Staff within the NIH awarding component, with guidance from the appropriate national advisory committee or council, will determine whether amended plans and reports submitted after the initial review are acceptable.
Evaluation Plan: Is the plan adequate to evaluate the short and long-term goals for each of the key proposed functions? Are the measures valid for the programs' goals to be assessed and how accessible and practical are the available data sources? Does the plan make sufficient resources available for participation in the national CTSA programs If necessary, is the plan to obtain IRB approval and informed consent from program participants adequate for self-evaluation activities and the national program evaluation?

Contact Information

For further information about the Clinical and Translational Science Awards and the CTSA Consortium, please contact:

Anthony R. Hayward, M.D., Ph.D.
Director
Division of Clinical Research Resources
National Center for Research Resources
National Institutes of Health
One Democracy Plaza, Room 906
6701 Democracy Boulevard, MSC 4874
Bethesda, Maryland 20892-4874 (20817 for express mail)
Telephone: 301-435-0790
Fax: 301-480-3661
HaywardA@mail.nih.gov