Notes

1. A randomized controlled trial is a study that randomly assigns individuals (or other units such as classrooms or schools) to a program group or to a control group, in order the measure the program’s impact on educational or other outcomes.

2. Other examples of common flaws include randomizing groups (e.g., classrooms or schools) but conducting the statistical analysis as if individual students had been randomized; and reporting only the program’s positive effects, rather than its effects on all outcomes that were measured.

3. If a randomized controlled trial is not feasible and you plan to use another study design, such as a matched comparisongroup study, this step would involve identifying candidates with experience in that design.

4. If you plan to use a high-quality study design other than a randomized controlled trial, you should ask for two study reports on evaluations using that design.

5. If you plan to use a high-quality study design other than a randomized controlled trial, we suggest you use an expert review process parallel to that outlined in this section, but with experts who have a strong understanding of that specific design.

6. We do not mean to suggest that all of the evaluator’s randomized controlled trials need be well-designed and implemented, because sometimes studies fail for reasons outside an evaluator’s control. Rather, we are suggesting that you look for evidence that the evaluator has successfully carried out a trial in at least two instances.

7. Many of the items in this checklist would apply to high-quality designs other than a randomized controlled trial, such as a well-matched comparison-group study, if that is what you plan to sponsor. However, you would need to add a few items—e.g., on effective matching technique—and modify a couple others. To do so, we suggest you seek the advice of someone with expertise in that specific design.

8. If you plan to use a high-quality study design other than a randomized controlled trial, the study team should have a demonstrated track record in conducting studies using that design.

This publication was produced by the Coalition for Evidence-Based Policy, in partnership with the What Works Clearinghouse (a joint venture of the American Institutes for Research and the Campbell Collaboration). It was produced under a contract with the U.S. Education Department’s Institute of Education Sciences (Contract #ED-02-CO-0022). The views expressed herein do not necessarily reflect the views of the Institute of Education Sciences.

This publication is in the public domain. Authorization to reproduce it in whole or in part for educational purposes is granted.