Department of Education
Improved Dissemination and Timely Product Release Would Enhance the Usefulness of the What Works Clearinghouse
GAO-10-644, Jul 23, 2010
Additional Materials:
Share This:
Contact:
(202) 512-8403
contact@gao.gov
Office of Public Affairs
(202) 512-4800
youngc1@gao.gov
In connection with the Omnibus Appropriations Act, 2009, GAO was required to study the What Works Clearinghouse (WWC), a federal source of evidence about effective education practices. Operating through a 5-year contract awarded by the U.S. Department of Education's Institute of Education Sciences (IES), the WWC reviews education research and disseminates its findings. GAO examined: (1) the extent to which the WWC review process meets accepted standards for research evaluation and how the WWC has responded to recommendations and criticism, (2) how WWC output and costs have changed over time and how its performance is measured, and (3) how WWC products are disseminated and how useful educators find them to be. To conduct its work, GAO reviewed WWC-related documents, analyzed cost and performance data, surveyed all states and a nationally representative sample of school districts, and interviewed IES officials, WWC contractors, researchers, and others.
GAO as well as a congressionallymandated panel of experts, found that the WWC's review process, which includes screening studies to determine if they meet WWC criteria, follows accepted standards for evaluating research on the effectiveness of education interventions. WWC is responding to recommendations made by the expert panel to further improve its review and reporting processes. For example, the panel recommended improvements in the way the WWC presents information to readers on the reasons why studies do not qualify for review. The WWC is revising a report template to include a table summarizing which studies met or did not meet WWC criteria for evaluating research. The WWC has also responded to researchers who have criticized the WWC for presenting limited information because its screening criteria exclude some rigorous research designs that may be appropriate for evaluating certain education programs, such as special education. The WWC responded to this criticism by creating new standards that include two additional study designs and by creating a new product, called a practice guide, which includes a wider range of research. WWC's report output and scope increased under the current contract. For example, the WWC increased its production of various reports, introduced new products, and developed new processes for evaluating research. However, IES had a substantial backlog in its product review process from January 2009 to May 2010. The backlog generally decreased the timeliness of WWC reports, with 20 reports being delayed by up to 6 months. To support the increases in output and scope, WWC's costs doubled from the previous contract to the current one. Both contracts designated about 60 percent of costs to production, while the other 40 percent of costs support other tasks, such as communications, dissemination, and process development. IES' performance goals for the WWC primarily relate to the number of reports produced. However, IES has not developed performance measures related to the cost or usefulness of WWC products. Education uses WWC contractors, Regional Educational Laboratories (RELS) and the Doing What Works (DWW) Web site to disseminate information about WWC products; however, awareness and use of the WWC varies among states, districts, teachers, and principals. WWC contractors disseminate product information in various ways including email alerts and presentations. The RELs host events featuring WWC products for state, district, and school officials and DWW provides resources to educators based on WWC products. Based on our survey, officials from 33 of 38 state education agencies that responded to our survey and an estimated 42 percent of school districts have heard of the WWC. Those states and school districts generally used the WWC to a small or moderate extent to inform decisions on effective practices. Based on our survey, states and school districts reported that they would likely increase their use of the WWC if it included a broader array of information or more timely information. GAO recommends that IES: develop and implement strategies to avoid backlogs in WWC product reviews; establish performance measures related to costs and usefulness; and improve dissemination efforts to promote awareness and use of the WWC. Education generally agreed with GAO's recommendations.
Status Legend:
Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
- In Process
- Open
- Closed - implemented
- Closed - not implemented
Recommendations for Executive Action
Recommendation: To better track the costs and usefulness of the WWC, the Secretary of Education should direct IES to develop performance measures related to product usefulness and periodically assess whether WWC products are meeting the needs of target audiences by gathering information on product usefulness in the proposed survey or through other means.
Agency Affected: Department of Education
Status: Closed - Implemented
Comments: Department of Education officials reported that it uses a wide variety of tools to gather consumer input, such as a Help Desk and online voting for future report topics. GAO noted that this relies on existing users and reflects the views of those users who provide feedback, rather than those of the broader population. According to Education officials, the agency developed a plan for obtaining feedback on What Works Clearinghouse (WWC) products from attendees at major educational conferences. The agency e-mailed all conference participants and provided them with a link to their web-based customer satisfaction survey, which they shared with GAO. The agency provided GAO with some of their survey data. Education plans to conduct a more extensive review in 2012 that requires OMB clearance, including surveying potential customers about satisfaction, relevance and usefulness of WWC products. Education officials confirmed that data on product usefulness were being used in the evaluation criteria for the upcoming WWC contract bidding process in 2012, and that ED's Institute of Education Sciences will evaluate the next contractor based on these findings.
Recommendation: To better track the costs and usefulness of the WWC, the Secretary of Education should direct IES to incorporate findings from its cost studies to develop performance measures related to costs, such as identifying a range of acceptable costs per product and using that information to monitor contractor spending.
Agency Affected: Department of Education
Status: Closed - Implemented
Comments: Education concurred, and noted it had already taken some steps that address the recommendation previously, and has now fully implemented this recommendation. According to Education officials, in 2010, Education completed cost studies on WWC products. The agency shared additional findings from these studies with GAO in 2011, including cost ranges established for WWC products. Education officials confirmed that these findings were being used in the evaluation criteria for the upcoming WWC contract bidding process in 2012, and that IES plans to evaluate the next contractor based on these findings.
Recommendation: To consistently release WWC products in a timely manner, the Secretary of Education should direct IES to develop and implement strategies that help avoid future backlogs and ensure that IES's review and approval processes keep pace with increased contractor production. Strategies could include shifting IES resources to ensure sufficient staff time for managing the peer review process and streamlining its approval processes.
Agency Affected: Department of Education
Status: Closed - Implemented
Comments: According to Department of Education officials, in 2011, the agency reviewed and revised their policies and procedures for reviewing and approving What Works Clearinghouse products. They indicated that these revisions will ensure the process is scientifically sound, efficient, and results in a timely release of documents. For example, they have automated portions of the template used by reviewers to expedite their work. They have also expanded their pool of experienced peer reviewers and internal staff support, increasing their capacity to complete reviews, and provided product release schedules that demonstrate improved timeliness.
Recommendation: To reach more members of the target audience, the Secretary of Education should direct IES to assess and improve its dissemination efforts to promote greater awareness and use of the WWC, for example, by developing a way to inform school districts of new products or encouraging educator professional development programs to focus on research-based practices such as those discussed in practice guides.
Agency Affected: Department of Education
Status: Closed - Implemented
Comments: According to Department of Education officials, in 2010, Education updated its outreach plan for the What Works Clearinghouse (WWC). For example, Education held department-wide briefings on WWC products intended to strengthen collaboration and awareness within the agency and improve promotion of WWC efforts. The agency also increased outreach efforts to key external educational associations in an attempt to further improve dissemination efforts. Education has also initiated several efforts to assess its dissemination efforts, including tracking of WWC product awareness and use at outreach events focused on Practice Guides and analyzing product website visits and referrals from other websites.