Justice Outcome Evaluations
Design and Implementation of Studies Require More NIJ Attention
GAO-03-1091, Sep 24, 2003
Additional Materials:
Share This:
Contact:
(202) 512-2758
contact@gao.gov
Office of Public Affairs
(202) 512-4800
youngc1@gao.gov
Policy makers need valid, reliable, and timely information on the outcomes of criminal justice programs to help them decide how to set criminal justice funding priorities. In view of previously reported problems with selected outcome evaluations managed by the National Institute of Justice (NIJ), GAO assessed the methodological quality of a sample of completed and ongoing NIJ outcome evaluation grants.
From 1992 through 2002, NIJ managed 96 evaluation studies that sought to measure the outcomes of criminal justice programs. Spending on these evaluations totaled about $37 million. Our methodological review of 15 of the 96 studies, totaling about $15 million and covering a broad range of criminal justice issues, showed that sufficiently sound information about program effects could not be obtained from 10 of the 15. Five studies, totaling about $7.5 million (or 48 percent of the funds spent on the studies we reviewed), appeared to be methodologically rigorous in both design and implementation, enabling meaningful conclusions to be drawn about program effects. Six studies, totaling about $3.3 million (or 21 percent of the funds spent on the studies we reviewed), began with sound designs but encountered implementation problems that would render their results inconclusive. An additional 4 studies, totaling about $4.7 million (or 30 percent of the funds spent on the studies we reviewed), had serious methodological limitations that from the start limited their ability to produce reliable and valid results. Although results from 5 completed studies were inconclusive, DOJ program administrators said that they found some of the process and implementation findings from them to be useful. We recognize that optimal conditions for the scientific study of complex social programs almost never exist, making it difficult to design and execute outcome evaluations that produce definitive results. However, the methodological adequacy of NIJ studies can be improved, and NIJ has taken several steps--including the formation of an evaluation division and funding feasibility studies--in this direction. It is too soon to tell whether these changes will lead to evaluations that will better inform policy makers about the effectiveness of criminal justice programs.
Status Legend:
Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
- In Process
- Open
- Closed - implemented
- Closed - not implemented
Recommendations for Executive Action
Recommendation: The Attorney General should instruct the Director of NIJ to conduct a review of its ongoing outcome evaluation grants--including those discussed in this report--and develop appropriate strategies and corrective measures to ensure that methodological design and implementation problems are overcome so the evaluations can produce more conclusive results. Such a review should consider the design and implementation issues we identified in our assessment in order to decide whether and what type of intervention may be appropriate. If, based on NIJ's review, it appears that the methodological problems cannot be overcome, NIJ should consider refocusing the studies' objectives and/or limiting funding.
Agency Affected: Department of Justice
Status: Closed - Implemented
Comments: NIJ developed and implemented an Evaluation Research Oversight Plan to ensure that evaluations stay on track, issues are highlighted, and solutions, where feasible, are proposed. To be responsive to GAO's recommendation, NIJ now identifies evaluation research projects that require monitoring at a high intensity level, as opposed to normal. The oversight plan calls for project monitors to discuss the high intensity projects at semi-annual meetings. Among the topics for discussion are changes in evaluation plan, methodology, or design, and problems in implementation. Corrective actions taken are to be incorporated into an annual report. NIJ provided information on each of its high intensity evaluation research grants for the period January 2005 through June 2005.
Recommendation: The Attorney General should instruct the Director of NIJ to continue efforts to respond to our March 2002 recommendation that NIJ assess its evaluation process with the purpose of developing approaches to ensure that future outcome evaluation studies are funded only when they are effectively designed and implemented. The assessment could consider the feasibility of such steps as obtaining more information about the availability of outcome data prior to developing a solicitation for research; requiring that outcome evaluation proposals contain more detailed design specifications before funding decisions are made regarding these proposals; and more carefully calibrating NIJ monitoring procedures to the cost of the grant, the risks inherent in the proposed methodology, and the extent of knowledge in the area under investigation.
Agency Affected: Department of Justice
Status: Closed - Implemented
Comments: We recommended that NIJ continue efforts to assess its evaluation process with the purpose of developing approaches to ensure that future outcome evaluation studies are funded only when they are effectively designed and implemented. NIJ has reviewed its guidance for grant applications and added selection criteria and reporting requirements, as well as stated that funding priority would be given to rigorous evaluation research designs. NIJ also developed a multi-part corrective action plan to implement our recommendation which, among other things, calls for assessing the feasibility of proposed outcome evaluations when appropriate and puts in place a mechanism for monitoring all current and future evaluation research grants. In addition, NIJ prepared an Evaluation Research Oversight Plan that included nine steps to be taken by NIJ personnel (specifically, Evaluation Division staff, project monitors, division chiefs, and the Acting director of the Office of Research and Evaluation) to monitor NIJ outcome evaluations and help ensure that they are effectively designed and implemented. We believe the steps taken by NIJ will facilitate program planning and provide the Congress and others with more meaningful information based on better designed and implemented evaluation studies.