Clinical and Translational Science Awards (CTSA) Applications - Peer Review Approach

This description is provided to inform the extramural community of the approach used for the initial peer review of the CTSA applications. See Funding Opportunity Announcement (FOA) RFA-RM-08-002.

Structure of Critique

The criteria comprising each review section are taken directly from the published FOA and are organized to facilitate reviewer assignments.

• At least five reviewers are assigned to each application; one reviewer may have one or two assignments on an application, but typically no reviewer has more than three assignments on an application. Two reviewers are assigned to provide a critique on Integration and Overall; typically these two reviewers are drawn from those assigned to other components of the application.
• If an application contains components not identified in the following table, the component will be assigned and reviewed according to standard NIH review criteria: Significance, Approach, Innovation, Investigators, and Environment.

Reviewer	Review Section Heading	Brief Description of Approach
#1	Significance, Approach, Innovation, Environment, and Implementation Plans	CTSA structure
#2	CTSA Staffing, Governance, Institutional Commitment, Local and National Collaboration, Data Sharing, Dissemination and Evaluation Plan	CTSA personnel, collaboration, sharing and evaluation plans
#3	CTSA Biomedical Informatics, Investigators and Staffing; also includes Human Subjects Issues, such as Data Confidentiality/Security	Self-explanatory
#4	CTSA Clinical Research Design, Biostatistics, Community Engagement, Clinical Research Ethics, Participant and Clinical Research Interactions, and Regulatory Knowledge and Support; Relevant Investigators and Staffing	Infrastructure elements related to clinical research design and implementation
#5	CTSA Translational Technologies and Resources; Pilot Studies; Relevant Investigators and Staffing	Cores, pilots and translational tools, defined in application
#6	CTSA Training (All applications require K12, other components are optional), Research Education and Career Development; Relevant Investigators and Staffing	Self-explanatory
#7/8	Integration and Overall	Self-explanatory

Critique 1: Significance, Approach, Innovation, Environment, and Implementation Plans

Significance: Will the proposed CTSA significantly impact the overall quality of clinical and translational science at the applicant institution? Is the overall program vision and strategy adequate to satisfy the intent of this initiative to facilitate and sustain a home for clinical and translational science that incorporates a wide range of clinical disciplines, specialties, and sub-specialties? Will the proposed CTSA have potential to make significant contributions to a national consortium of CTSAs?

Approach: Will the CTSA program enhance, complement, or extend the applicant's current resources for clinical and translational science research? Does the application identify key obstacles to the performance of translational and clinical research and then propose plans or means to overcome these? Will the proposed C/D/I include relevant scientific disciplines to maximize productivity? Does the application make efficient use of potentially unique resources, such as access to certain human subject populations or the provision of pre-clinical resources? Does the applicant indicate how the organization will be adapted to respond to changes in translational focus? Will new opportunities for careers in clinical research arise across the spectrum of clinical and translational science?

Innovation: Is the CTSA program original and innovative? Are new approaches proposed that would integrate clinical, basic and other relevant (e.g. public health, bioinformatics) disciplines? Does the program develop or employ novel concepts, approaches, methodologies, tools, or technologies that will improve the discipline? Is the program likely to develop novel approaches to increasing the ease and efficiency of clinical and translational research, allowing research results to move from patient observations and laboratory discoveries to the bedside and to clinical practice?

Is there an active program of research in novel methodologies? Is the outcome likely to benefit the C/D/I? Is there a plan to involve new investigators? Will these activities be integrated with the CTSA as a whole? Are there unique features of the scientific environment or in the available human subject populations or collaborative arrangements?

Environment: Do the academic and scientific environments contribute to the probability of success in establishing a home for clinical and translational science? Does the proposal provide strong evidence that the addition of the CTSA will provide resources that would not otherwise be possible? If applicable, are there adequate cooperative arrangements between affiliated institutions to ensure that the CTSA program performs effectively as one activity across institutional boundaries?

Implementation Plans: Is an implementation phase well described? Is the timeline for implementation feasible and are specific goals and milestones set? Are alternatives proposed should the goals and milestones not be reached in a timely manner? Is there a feasible time line for integrating CTSA resources with other complementary resources available to the institution?

Strengths and Weaknesses: Briefly summarize the most important points of the Critique, addressing the strengths and weaknesses of this part of the application in terms of the review criteria.

Score (Critique 1): This score is informational only and is not to be averaged with other critique scores to reach the final official score (one number) for this application; the official, final score may be more or less than the average of the scores. Final scores are based on the written reviews and the review committee discussion.

Critique 2: CTSA Staffing, Governance, Institutional Commitment, Local and National Collaboration, Data Sharing, Dissemination and Evaluation Plan

Investigators: Are the PD/PI(s) and other key personnel appropriately trained and well suited to carry out this work? Is the work proposed appropriate to the experience level of the PD/PI(s)? Do the PIs have the experience and authority and committed time to administer the proposed institutional home for clinical and translational research? Will the PI have sufficient authority and credibility in the institution to work across institutional boundaries? Will the PI have the environment and institutional support necessary to be responsible for the resources committed by the institution(s) for the center, department, or institute (C/D/I)? Do the program leadership and management team bring complementary and integrated expertise to the project?

Will the proposed C/D/I have the professional staffing to impact significantly the overall quality of clinical and translational science at the institution? If part of the proposal, do the Key Function Directors have the appropriate training, experience and resources to assume leadership roles? Have the Directors of the Key Functions committed sufficient time to this Program? Will the Directors have the authority to implement best practices identified at Steering Committees at their Institution? Are the administrative and professional staff appropriately trained and well suited to carry out this work?

CTSA Governance: Have the applicants described an effective administration and governance structure that will promote the discipline of clinical and translational science? For applications designating multiple PD/PIs, is the leadership approach, including the designated roles and responsibilities, governance and organizational structure consistent with and justified by the aims of the CTSA and the expertise of each of the PD/PIs? Is the governance structure designed to ensure both accountability of multiple PIs, and integration of the components of the C/D/I into a coherent program? Will the leadership and governance plans accommodate changes in the direction of research and allow for the efficient use of funds? Will an Advisory Committee be constituted to provide critical, stimulating, and thoughtful advice for the overall CTSA performance and CTSA Key functions? Are there plans to resolve conflicts and implement recommendations?

Institutional Commitment: Is there institutional commitment to establishing the CTSA program as an integral part of its overall clinical research environment? Will the institution align or adjust incentives and rewards to promote the academic mission and new modes of team-based research? Is there substantial commitment from the institutional leadership to protect the time of the investigators to pursue clinical and translational research and mitigate the demands of providing patient care? Will clinical researchers/trainees career development be supported in terms of a specific tenure process for clinical researchers at the institution? Is the institutional leadership committed to this program and its goals in terms of providing specific assets for the program, such as financial support, faculty support, specific equipment, dedicated space, or tuition rebates, as a few examples? Will existing NIH-supported Cores be appropriately shared with the CTSA program?

For the training program, this criterion assesses the quality of the institutional training environment for NRSA supported trainees and the relationship of the NRSA program to the broader training program (if appropriate). What is the level of institutional commitment, quality of the facilities, availability of appropriate courses, and the availability of research and research training support? Does the environment in which the training program will be conducted, i.e. the quality of the participating departments and the extent of their participation, contribute to the probability of success? Is there evidence of adequate institutional commitment?

For the career development program, this criterion will assess the applicant institution's commitment to the program, such as recruitment efforts, necessary educational resources and equipment, and available established investigators who will serve as mentors, evidence that scholars will have sufficient "protected" time to devote to the program.

Local and National Collaboration, Data Sharing, and Dissemination: How adequately will the institution and its researchers collaborate, share and disseminate resource tools and resources at institutional, community, and national levels? Are plans included to address regulatory hurdles locally? Is there a commitment to and plans for adopting and implementing national standards?

Evaluation Plan: Is the plan adequate to evaluate the short and long-term goals for each of the key proposed functions? Are the measures valid for the programs' goals to be assessed and how accessible and practical are the available data sources? Does the plan make sufficient resources available for participation in the national CTSA programs If necessary, is the plan to obtain IRB approval and informed consent from program participants adequate for self-evaluation activities and the national program evaluation?

Strengths and Weaknesses: Briefly summarize the most important points of the Critique, addressing the strengths and weaknesses of this part of the application in terms of the review criteria.

Score (Critique 2): This score is informational only and is not to be averaged with other critique scores to reach the final official score (one number) for this application; the official, final score may be more or less than the average of the scores. Final scores are based on the written reviews and the review committee discussion.

Critique 3: CTSA Biomedical Informatics

Investigators: If part of the proposal, does the Biomedical Informatics Director(s) have the appropriate training, experience and resources to assume leadership roles of their respective units (if more than one unit)? Have the Director(s) and any co-director(s) of this key resource committed sufficient time to devote to this Program? Will the Directors have the authority to implement best practices identified at Steering Committees at their Institution? Are the administrative and professional staff appropriately trained and well suited to carry out this work?

Biomedical Informatics: Will the biomedical informatics resources offered be commensurate with the breadth of the CTSA program? Will data security and privacy be safeguarded? Are assessments of performance of this resource included? Will the Biomedical Informatics Director have the necessary authority to the ensure implementation of best practices as adopted by the Biomedical Informatics Steering Committee? As applicable, will this resource be sufficient for intra- and inter-institutional operations? Will the institution be willing to work toward interoperability of the informatics systems and adopting national data standards?

Strengths and Weaknesses: Briefly summarize the most important points of the Critique, addressing the strengths and weaknesses of this part of the application in terms of the review criteria.

Score (Critique 3): This score is informational only and is not to be averaged with other critique scores to reach the final official score (one number) for this application; the official, final score may be more or less than the average of the scores. Final scores are based on the written reviews and the review committee discussion.

Critique 4: CTSA Clinical Research Design, Biostatistics, Clinical Research Ethics, Participant and Clinical Research Interactions, Community Engagement and Research, and Regulatory Knowledge and Support

Investigators: If part of the proposal, do core/resource Directors have the appropriate training, experience and resources to assume leadership roles of their respective units? Have the Program Director, co-Program Director(s) and Director(s) and co-Director(s) of these key resources committed sufficient time to devote to this Program? Will the Directors have the authority to implement best practices identified at Steering Committees at their Institution? Are the administrative and professional staff appropriately trained and well suited to carry out this work?

Clinical Research Design and Biostatistics: What types of support and resources will be in place to ensure all clinical and translational research designs are sound and that statistical analyses are appropriate and rigorous? Will this training include conflict of interest, federal codes requirements, guaranteeing privacy and safety of research participants, especially as pertaining to vulnerable populations? Are there plans for creation and innovation in developing the application of these topics to clinical research? As applicable, will this resource be sufficient for intra- and inter-institutional operations?

Participant and Clinical Interactions, and Ethics: Will human subject participation in clinical research protocols be encouraged? Will the institution work with underserved populations in clinical research? Has the applicant adequately described and justified the resources to be provided? Will PCI resources meet the highest standards for subject safety, quality of science and statistical and ethical design? Is the application of Good Clinical Practice guidelines appropriate? Will resource utilization be tracked and are mechanisms proposed to adapt resources to the needs of investigators? Will the resources provided serve small as well as large studies or trials?

Community Engagement and Research: Will this effectively involve the community in which the CTSA institution resides, both the public and practitioners, in clinical and translational research priority setting, participation, and follow-up? Are there adequate plans to train researchers, trainees, and scholars in the methodology of community/population-based research and outreach? Will the resource foster long-term bidirectional relationships between the CTSA institution and the community for their mutual benefit? Will the research interests of center, department, or institute (C/D/I) faculty contribute to an intellectually stimulating environment?

Regulatory Knowledge and Support: Will this resource provide "researcher-focused" support for regulatory compliance and management? Is the resource well integrated with biomedical informatics and participant and clinical interactions? Is there duplication of IRB responsibilities? Does the institution have experience in working with multi-site trials and with the FDA with respect to studies involving investigational new drug application procedures? Will CTSA staff members be available with the necessary experience in working with the FDA and in ensuring that standards for reporting adverse events are met? Are criteria for identifying a research participant advocate sound?

Strengths and Weaknesses: Briefly summarize the most important points of the Critique, addressing the strengths and weaknesses of this part of the application in terms of the review criteria.

Score (Critique 4): This score is informational only and is not to be averaged with other critique scores to reach the final official score (one number) for this application; the official, final score may be more or less than the average of the scores. Final scores are based on the written reviews and the review committee discussion.

Critique 5: CTSA Translational Technologies and Resources; Pilot Studies

Investigators: If part of the proposal, do core/resource Directors have the appropriate training, experience and resources to assume leadership roles of their respective units? Have the Program Director, co-Program Director(s) and Director(s) and co-Director(s) of these key resources committed sufficient time to devote to this Program? Will the Directors have the authority to implement best practices identified at Steering Committees at their Institution? Are the administrative and professional staff appropriately trained and well suited to carry out this work?

Development of Novel Clinical and Translational Methodologies: Is there an active program of research in novel methodologies? Is the outcome likely to benefit the center, department, or institute (C/D/I)? Is there a plan to involve new investigators? Will these activities be integrated with the CTSA as a whole?

Translational Technologies and Resources: Is the plan to identify technologies to be offered appropriate? Will resource utilization and evaluation be adequate? Is their flexibility in types of resources to be offered? Will faculty members be encouraged to pursue research in areas that develop translational methodologies?

Pilot and Collaborative Translational and Clinical Studies: Is there an adequate plan to solicit proposals, to prioritize the projects and to review their methodology and research performance? Will the expected benefits to the CTSA and to the wider research community be measured and tracked? Will lessons learned be shared?

Strengths and Weaknesses: Briefly summarize the most important points of the Critique, addressing the strengths and weaknesses of this part of the application in terms of the review criteria.

Score (Critique 5): This score is informational only and is not to be averaged with other critique scores to reach the final official score (one number) for this application; the official, final score may be more or less than the average of the scores. Final scores are based on the written reviews and the review committee discussion.

Critique 6: CTSA Training (all applications require K-12, other components optional)

Research Education and Career Development: Do the Research Education, Training and Career Development components strengthen the training and career pathways for all clinical and translational research professionals and team members? Can increased efficiency shorten the period of training? Does the institution have a sufficient pool of academically strong trainees and commensurate experienced and well-qualified mentors to justify the career development pathways that are proposed? Will the curricula and courses proposed provide appropriate training in clinical and translational research relevant to a broad range of specialties? How will trainee registration for higher degrees in clinical research be encouraged? Has the Program Director committed adequate time to program administration?

Preceptors/Mentors: This criterion assesses the caliber of mentors as researchers, including successful competition for research support in areas directly related to the proposed research training or career development program. What is the overall quality of mentors' research, their publication record, and their successful competition for research support in areas directly related to the proposed training program? How strong is their record as mentors?

Trainee Recruitment, Selection and Retention: This criterion evaluates the quality of the applicant pool and the plan for selection of individuals for appointment to the training or career development program. Specifically, what is the quality and size of the applicant pool? Are the recruiting procedures, trainee selection criteria, and retention strategies appropriate and well defined?

Mentored Career Development (K12) and Research Program Design: This criterion will assess the likelihood that the proposed career development and research plans will contribute significantly to the scientific development of the candidate scholars to successfully pursue clinical and translational research careers. Reviewers will evaluate whether the proposed plans include appropriate course work and/or activities to achieve program goals. The scientific quality, technical merit and degree of relevance of the proposed research projects the scholars will be pursuing will be evaluated.

Pre-doctoral Research Training Component (T32): The quality of the proposed training program will be considered as well as whether the requested number of trainee positions is appropriate for the planned research training program. Does the proposed training program provide appropriate courses for clinical and translational science research? Are appropriate programmatic activities incorporated into the training program?

Research Training Record: This criterion evaluates the past research training record of both the program and the designated mentors. How successful are former trainees in seeking further career development and in establishing productive scientific careers? Evidence of further career development can include successful completion of the Ph.D., receipt of fellowships, career development awards, additional training appointments, and similar accomplishments. Evidence of a productive scientific career can include a record of successful competition for research grants, receipt of special honors or awards, a record of publications, receipt of patents, promotion to scientific positions, and any other measure of success consistent with the nature and duration of the training received. What is the track record of the mentors in directing pre-doctoral training or the potential of those mentors lacking a track record?

Research Education Component: Review considerations include the quality, innovation and content of courses and adequacy of the syllabus; the scientific qualifications and experience of the faculty; the criteria for selecting participants; efforts to publicize the availability of the program to potential participants.

Training in the Responsible Conduct of Research: Assess the applicant's plans for training in the responsible conduct of research on the basis of the appropriateness of topics, format, amount and nature of faculty participation, and the frequency and duration of instruction for the research education, training and career development components

Evaluation and Tracking of Research Education, Training, and Career Development: Is the plan adequate to determine progress and outcome measures for each of the research education, training and career development components? Does it include a system for tracking participants following program completion to determine success or failure of the program? The tracking would include information on program publications; grant proposals and awards, and career trajectory of the trainees that were supported in the program. If an Advisory Committee is proposed, are plans adequate and appropriate to ensure proper monitoring of the research education, training and career development components? Are there means to modify the research education, training and career development components based on appropriate recommendations from the Advisory Committee?

Training Environment: Does the applicant adequately demonstrate that a program for awarding higher degrees in clinical research is in place? Is there a strong training record at both institutional and faculty levels?

Strengths and Weaknesses: Briefly summarize the most important points of the Critique, addressing the strengths and weaknesses of this part of the application in terms of the review criteria.

Score (Critique 6): This score is informational only and is not to be averaged with other critique scores to reach the final official score (one number) for this application; the official, final score may be more or less than the average of the scores. Final scores are based on the written reviews and the review committee discussion.

Critique 7/8: CTSA Integration and Overall (Note: critiques to be amended as needed based on discussion at the review meeting)

Integration: How well will the components of the CTSA be integrated with each other? Are there plans to integrate CTSA activities into all the relevant schools and clinical research sites that participate in clinical and translational science in the applicant institution? Is there a commitment to integrate the CTSA into the institution and into a national network of CTSAs and also to reach out to the local community? Will this integration be reflected in the senior leadership and decision-making processes of the CTSA? Is the CTSA program integrated, cohesive, synergistic, adaptable, and potentially more effective than what currently exists?

Overall: Will the proposed CTSA significantly impact the overall quality of clinical and translational science at the proposed center, department, or institute (C/D/I)? Does the program make only modest incremental improvements to an already strong program? Will significant value be added to existing resources? Will the proposed program significantly enhance/improve patient interaction resources, career development and training, informatics, research design, ethics, and interactions with human subjects? Does the applicant acknowledge potential problem areas and consider alternative tactics?

Is the Institutional commitment sufficient? Is the training program appropriate and well integrated with the goals of the CTSA? Will the proposed program provide adequate support for clinical and translational investigators as well as for the development of new knowledge that drives the discipline of clinical and translational research? Are the key components of the CTSA adequately described? Does the program make only modest incremental improvements to previously-supported programs, or will significant value be added? Will the resources be equitably distributed among different disciplines (e.g., pediatrics, medicine, pre-clinical research)?

Strengths and Weaknesses: Briefly summarize the most important points of the Critique, addressing the strengths and weaknesses of the application in terms of the review criteria.

Score (Critique 7/8): This score is informational only and is not to be averaged with other critique scores to reach the final official score (one number) for this application; the official, final score may be more or less than the average of the scores. Final scores are based on the written reviews and the review committee discussion.

Contact Information

For further information about the Clinical and Translational Science Awards and the CTSA Consortium, please contact:

Anthony R. Hayward, M.D., Ph.D.
Director
Division of Clinical Research Resources
National Center for Research Resources
National Institutes of Health
One Democracy Plaza, Room 906
6701 Democracy Boulevard, MSC 4874
Bethesda, Maryland 20892-4874 (20817 for express mail)
Telephone: 301-435-0790
Fax: 301-480-3661
HaywardA@mail.nih.gov

For information related to Peer Review, please contact:

Mohan Viswanathan, Ph.D.
Scientific Review Administrator
Office of Review
National Center for Research Resources
National Institutes of Health
One Democracy Plaza, Room 1084
6701 Democracy Boulevard, MSC 4874
Bethesda, Maryland 20892-4874 (20817 for express mail)
Telephone: 301-435-0829
Fax: 301-480-3660
mv10f@nih.gov