Step 3: For the Evaluator(s) That Pass the Step 2 Vetting Process, Request a Proposal.

Your request would presumably include a number of items specific to the evaluation you are sponsoring.

For example, your request might include general parameters of the study that you, as the sponsor, wish to specify, such as: the program to be evaluated, the setting for the study, the main questions you seek to answer, and your timeline and available funding. Within these general parameters, your request might then invite the applicant to describe its proposed research questions, research methods, data analysis plan, management and staffing plan, timeline, and budget.

A full discussion of project-specific items to include in your request is beyond the scope of this Guide.

To help ensure that the randomized controlled trial will be well-designed and implemented, we suggest that you also request and review the following:

• The applicant’s plan for addressing each item in the What Works Clearinghouse’s Key Items To Get Right When Conducting A Randomized Controlled Trial in Education.⁷

This document is a user-friendly checklist of items that are often critical to the success of a randomized controlled trial, a significant departure from any one of which may undermine the validity of the study’s results. As part of your proposal review process, we suggest you ask 1-2 experts (such as those you identified in step 2) to review the applicant’s plan for addressing these key items to determine whether it constitutes a sound approach.

• The curricula vitae of key staff and their proposed roles (including time commitments), which should demonstrate that the study team has the core capabilities you seek:

(i) A demonstrated track record in conducting well-designed randomized controlled trials⁸ in program areas similar to yours; and

(ii) Expertise in the program area being evaluated, or closely-related areas.

Ideally, the proposed principal investigator will have both of these core capabilities. But often that may not be the case, and instead different individuals on the team will have these respective capabilities. In this circumstance, we suggest you verify that at a minimum: (i) the individual(s) with a demonstrated track record in conducting well-designed randomized controlled trials will play a hands-on role in all main aspects of study design and implementation; and (ii) the individual(s) with substantive expertise in the program area will play a key consulting role in study design and implementation, and in interpretation and communication of the study’s findings.

• A list of 3-4 references, so you can verify that the proposed key staff did indeed play a central role in the applicant’s earlier well-designed randomized controlled trials.

Useful references would be other researchers or school officials who participated in the earlier trials. We suggest you ask these references to confirm whether the proposed key staff: (i) played a central role in the earlier trials; and (ii) successfully handled that role, demonstrating the organizational and interpersonal skills needed to carry out the trial within budget and schedule. You may wish to assure the references that their answers will be kept confidential, so as to facilitate an open and complete conversation.

The above process should enable you to choose a highly-capable evaluator—one with (i) a demonstrated track record in high-quality evaluations, (ii) substantive expertise in your program area, or related areas, and (iii) a sound plan for conducting your study so as to produce scientifically valid results.