NCEE Blog

EDITOR'S NOTE: Dr. Steve Graham was the head of a panel of experts that assisted the What Works Clearinghouse in developing recommendations for its practice guide on effective writing for secondary students. We invited Dr. Graham to author this blog about the guide and a January 18 webinar on its recommendations. 

By Steve Graham, Warner Professor in the Division of Leadership and Innovation, Arizona State University

Effective writing is a vital component of students’ literacy achievement and a life-long skill that plays a key role in postsecondary success. For more than 30 years, I’ve focused my research on how teachers can help students become strong writers, how writing develops, and how writing can be used to support reading and learning. Much progress has been made in the field of writing instruction, and summarizing and sharing these findings will help teachers implement evidence-based practices. Using effective instructional practices will help ensure our students become adept at using writing to support and extend learning, argue effectively and fairly, connect and communicate with others, tell captivating stories, and explore who they are as well as reflect on their experiences. 

Recently, the What Works Clearinghouse (WWC) released a new practice guide to address the challenges of teaching writing to secondary students. Teaching Secondary Students to Write Effectively offers three evidence-based recommendations for helping students in grades 6–12 develop effective writing skills. The first recommendation focuses on teaching students to use writing strategies to plan, think critically, and effectively convey their ideas. The second recommendation suggests integrating reading and writing to emphasize key features of text. Finally, the third recommendation describes how to use a formative assessment cycle to inform writing instruction.

The guide includes practical instructional tips and strategies for each recommendation that teachers can use to help students improve their writing. You’ll find over 30 examples to use in the classroom, including sample writing strategies and prompts and activities that incorporate writing and reading.

I’d like to invite teachers, administrators, and others to join me for a webinar on the recommendations in this practice guide, Wednesday, January 18, at 3 p.m. (ET). During the webinar, we will discuss the guide’s three recommendations and give teachers in all disciplines usable guidance on how to implement them in the classroom. We will also discuss potential challenges educators may face when implementing the recommended practices and provide advice on how to overcome those challenges.

Developing the Practice Guide

The WWC develops practice guides with the support of an expert panel. The panelists combine their expertise with the findings of rigorous research to produce specific recommendations. I was honored to chair this panel, which also included Jill Fitzgerald, from the University of North Carolina at Chapel Hill and MetaMetrics; Linda D. Friedrich, from the National Writing Project; Katie Greene, from Forsyth County (Ga.) Schools; James S. Kim, from Harvard University; and Carol Booth Olson, from the University of California, Irvine. 

For this practice guide, WWC staff conducted a systematic review of the research—a thorough literature search identified more than 3,700 relevant studies. After screening each study, 55 studies were found to use eligible research designs and examine the effec­tiveness of the practices found in this guide’s recommendations. The recommendations are based on the 15 studies that meet the WWC’s rigorous standards. For each of the recommendations, the WWC and the panel rate the strength of the evidence that supports it.  Appendix D in the guide presents a thorough summary of the evidence supporting each recommendation. 

By Christopher Weiss, Senior Education Research Scientist, WWC

Many individuals and organizations have special ways of doing things, specific procedures that make them unique —Coca-Cola has its formula; sports teams have their playbooks; and grandparents have their secret recipes for biscuits, barbecue, and other family favorites.

It’s the same for the What Works Clearinghouse (WWC). Our “special sauce” is in how we review effectiveness research to help determine what is working in education. But unlike Coke, coaches, and grandma, the WWC doesn’t keep it a secret.

On December 15, the Institute of Education Sciences (IES) launched a set of video training modules – the WWC Group Design Standards Online Training – to share our procedures. These modules are designed to help you learn more about the elements that go into a WWC rating and the features of a research study that WWC examines during evaluation.  The online training will help education decision-makers and researchers better understand key elements of the WWC review process. These modules describe and explain key topics and concepts of the WWC’s Group Design standards and how the WWC uses these standards to identify and evaluate high quality, rigorous research.

The series is designed to address the needs of both consumers and future producers of the WWC’s reviews of educational effectiveness research. Whether you’re a researcher who’s hoping your study will meet the WWC’s standards or someone trying to make an evidence-based decision related to education, this training series will help! And no background in research is needed –we’ve also developed an extensive set of materials to support you as you learn.

Each of the five modules follows a similar structure, including an overview of module objectives, detailed information about the topic, examples, and knowledge checks to reinforce what you’ve learned. (We've embedded the first video in the series at the end of this post, but if you are going to take the training, start it through the WWC website.)  

Each module focuses on a specific aspect of the standards.

• Group designs – or overall research designs – and the types of research that can be reviewed using the WWC Group Design Standards;
• Attrition, or loss of participants in a research study, and why this is important;
• Baseline Equivalence, which assesses how similar two groups are at the beginning of a research study;
• Confounding Factors, which are study components that make it difficult or impossible to distinguish the effect of an educational intervention from the effect of that component; and
• Outcome Measures, or what is measured to assess the effectiveness of an intervention.

If you view all five of the training modules, you will earn a certificate of completion. Details about how to view the session and earn this certificate are available on the What Works Clearinghouse website.

The online training takes about seven hours to complete, but the modules are designed so that you can complete them at your own pace. We’ve included a feature that allows you to take a break from the training at any point – then pick it up again where you left off when you’re ready to continue.

These modules cover the same material that WWC reviewers learn through their in-person certification training – and completion of the online training course is one step toward becoming a certified reviewer in WWC Group Design Standards. Certification also requires completing WWC Procedures training and successful completion of a certification exam. We expect to be able to offer online versions of the WWC Procedures training and the certification exam later in 2017.

We hope this online training brings more transparency and understanding to the WWC review process. Then we can work on that secret biscuit recipe. 

By Erin Pollard, ERIC Project Officer, NCEE

ERIC has recently added several new fields to our database that will make it easier for researchers to find relevant studies. These are changes we've been working on for a while and we are excited that they are finally live. 

By Thomas Wei, Evaluation Team Leader

As researchers, we take little pleasure when the programs and policies we study do not find positive effects on student outcomes. But as J.K. Rowling, author of the Harry Potter series, put it: there are “fringe benefits of failure.” In education research, studies that find no effects can still reveal important lessons and inspire new ideas that drive scientific progress.

On November 2, The Institute of Education Sciences (IES) released a new brief synthesizing three recent large-scale random assignment studies of teacher professional development (PD). As a nation we invest billions of dollars in PD every year, so it is important to assess the impact of those dollars on teaching and learning. These studies are part of an evaluation agenda that IES has developed to advance understanding of how to help teachers improve.

One of the studies focused on second-grade reading teachers, one on fourth-grade math teachers, and one on seventh-grade math teachers. The PD programs in each study emphasized building teachers’ knowledge of content or content-specific pedagogy. The programs combined summer institutes with teacher meetings and coaching during the school year. These programs were compared to the substantially less intensive PD that teachers typically received in study districts.

All three studies found that the PD did not have positive impacts on student achievement. Disappointing? Certainly. But have we at least learned something useful? Absolutely.

For example, the studies found that the PD did have positive impacts on teachers’ knowledge and some instructional practices. This tells us that intensive summer institutes with periodic meetings and coaching during the school year may be a promising format for this kind of professional development. (See the graphic above and chart below, which are from a snapshot of one of the studies.)

But why didn’t the improved knowledge and practice translate to improved student achievement? Educators and researchers have long argued that effective teachers need to have strong knowledge of the content they teach and know how best to convey the content to their students. The basic logic behind the content-focused PD we studied is to boost both of these skills, which were expected to translate to better student outcomes. But the findings suggest that this translation is actually very complex. For example, at what level do teachers need to know their subjects? Does a third-grade math teacher need to be a mathematician, just really good at third-grade math, or somewhere in between? What knowledge and practices are most important for conveying the content to students? How do we structure PD to ensure that teachers can master the knowledge and practice they need?

When we looked at correlational data from these three studies, we consistently found that most of the measured aspects of teachers’ knowledge and practice were not strongly related to student achievement. This reinforces the idea that we may need to find formats for PD that can boost knowledge and practice to an even larger degree, or we may need to find other aspects of knowledge and practice for PD to focus on that are more strongly related to student achievement. This is a critical lesson that we hope will inspire researchers, developers, and providers to think more carefully about the logic and design of content-focused PD.

Scientific progress is often incremental and painstaking. It requires continual testing and re-testing of interventions, which sometimes will not have impacts on the outcomes we care most about. But if we are willing to step back and try to connect the dots from various studies, we can still learn a great deal that will help drive progress forward.

By Dr. Barbara Foorman, Director Emeritus, Florida Center for Reading Research, Florida State University

The Institute of Education Sciences recently released a What Works Clearinghouse (WWC) educator’s practice guide that has four recommendations to support the development of foundational reading skills that are critically important to every student’s success. The recommendations in Foundational Skills to Support Reading for Understanding in Kindergarten Through 3^rd Grade are based on a comprehensive review of 15 years of research on reading, and guidance from a national panel of reading experts, of which I was the chair.

Recently, the Regional Educational Laboratory (REL) Southeast at Florida State University has developed a set of professional learning community (PLC) materials and videos to help teachers and other practitioners implement the guide’s recommendations in classrooms.

Over the past few months, REL Southeast has shared the practice guide and PLC materials with practitioners and policymakers in two states – North Carolina and Mississippi, which both have K-3 reading initiatives and reading coaches who assist with implementation. I’m excited by the feedback we are getting.

During these presentations, we shared the format of the ten 75-min PLC sessions and accompanying videos that demonstrate the recommendations and action steps in actual classrooms. We filmed the videos in partnership with Dr. Lynda Hayes, Director of the PK Yonge Developmental Research School at the University of Florida, and her primary grade teachers.

In North Carolina, we trained K–3 regional literacy consultants, elementary teachers and reading coaches, and higher education faculty on the PLC Facilitator’s Guide in Charlotte and Raleigh. The K-3 regional literacy consultants are organized by the North Carolina Department of Public Instruction.

In Mississippi, we trained the 90 Mississippi Department of Education reading coaches and district-supported special education specialists in Jackson. In turn, the state coaches will train the K–3 teachers who are a part of the reading initiative in the practice guide recommendations and action steps. Additionally, the coaches will work with the primary grade teachers in each of their assigned schools to implement the PLC. Having the state coaches oversee the implementation of the PLC will help ensure commitment and instill depth to the PLC sessions.

Also present at the training in Mississippi were faculty members from the University of Mississippi and Belhaven University. I accepted an invitation from the Mississippi Institutions of Higher Learning Literacy Council to speak to higher education faculty about the guide and PLC materials. The invitation is timely because Mississippi recently completed a study of teacher preparation for early literacy instruction.

I hope you will download the practice guide and PLC materials. If you have any thoughts, comments, or questions, please email Contact.IES@ed.gov. You can learn more about the work of the Regional Educational Laboratories program and REL Southeast on the IES website.  

Dr. Foorman is the Director of REL Southeast, located at Florida State University

By Erin Pollard, ERIC Project Officer, NCEE

Over the past several years, the staff of ERIC has worked hard to get to know its users—we want to understand who uses ERIC and how they are using it. We found that the most common users are college undergraduates using the ERIC electronic library to write research papers as part of their school work. We also learned that they often weren’t searching for the best resources to meet their needs.

For example, some users told the ERIC help desk they were writing a paper for an introductory “contemporary issues in education” course, but they were requesting documents from over 20 years ago. Also, some of the documents they wanted were not peer-reviewed or particularly relevant to their topic. It became clear that we could do a better job addressing our most frequent users’ needs.

First, we redesigned the ERIC website to be more friendly and intuitive for those who are not familiar with traditional search methods, making it more likely they would get the best results (learn more about searching ERIC). Second, we worked to develop tools to help novice users, including a new series of how-to videos.

The first video in this new series is aimed at students—Using ERIC to Write a Research Paper. The ERIC team worked with undergraduate faculty members to outline a step-by-step approach to describe how a student should begin a search process, refine their results, and use different fields to find the resources they need. The video describes how to use ERIC to find full-text resources and how users can know that they found right resources.

We asked school of education faculty and librarians for their thoughts on this video and received a lot of positive feedback. Some professors said that they would link to this video on their syllabi and post it on their course management systems, while librarians have posted it on their content management system, called LibGuides. Take a look and see what you think.  And if you have feedback on this or any other ERIC videos, or suggestions for new videos and tools, let us know through the ERIC Help Desk.

By Elizabeth Warner, Team Leader for Teacher Evaluations, NCEE

These days, people want answers to their questions faster than ever, and education research and evaluation is no exception. In an ideal world, evaluating the impact of a program or policy would be done quickly and at a minimal expense. There is increasing interest in quicker-turnaround, low-cost studies – and IES offers grants specifically for these types of evaluations.

But in education research, quicker isn’t always better. It depends, in part, on the nature of the program you want to study and what you want to learn.

Consider the case of complex, multi-faceted education programs.  By their very nature, complex programs may require several years for all of the pieces to be fully implemented. Some programs also may take time to influence behavior and desired outcomes.  A careful and thorough assessment of these programs may require an evaluation that draws on a large amount of data, often from multiple sources over an extended period of time. Though such studies can require substantial resources, they are important for understanding whether and why an investment had an intended effect. This is especially true for Federal programs that involved millions of dollars of taxpayer money.

On August 24, IES plans to release a new report from a large-scale, multi-year study of a complex Federal initiative: the Teacher Incentive Fund (TIF) [1] This report – the third from this evaluation -- will provide estimated program impacts on student achievement after three years of implementation. The Impact Evaluation of the Teacher Incentive Fund is a $13.7 million study that stretches for more than six years and has reported interim findings annually since 2014. (The graphic above is from the a study snapshot of the first TIF evaluation report.) 

A “big study” of TIF was important to do, and here’s why.

Learning from the Teacher Incentive Fund

TIF is a Federal program that provides grants to districts that want to implement performance pay with the goal of improving teacher quality.  TIF grants awarded in 2010 included the requirement that performance pay be based on an educator evaluation system with multiple performance measures consistent with recent research. The grantees were also expected to use the performance measures to guide educator improvement.

The TIF evaluation provides an opportunity not only to learn about impacts on student achievement over time as the grant activities mature but also to get good answers to implementation questions such as:

• How do districts structure the pay-for-performance bonus component of TIF?
• Are educators even aware of the TIF pay-for-performance bonuses?
• Do educators report having opportunities for professional development to learn about the measures and to improve their performance?
• Do educators change their practice in ways that improve their performance measures? 
• Are principals able to use their ability to offer pay-for-performance bonuses to hire or retain more effective teachers?  

Analyses to address these questions can suggest avenues for program improvement that a small-scale impact evaluation with limited data collection would miss. 

Studying the Initiative Over Time

With four years of data collection, the TIF evaluation is longer than most evaluations conducted by IES. But the characteristics of the TIF program made that extensive data collection important for a number of reasons.

First, TIF’s approach to educator compensation differs enough from the traditional pay structure that it might take time for educators to fully comprehend how it works. They likely need to experience the new performance measures to see how they might score and how that translates in terms of additional money earned. Second, educators might need to receive a performance bonus or see others receive one in order to fully believe it is possible to earn a bonus. (The chart pictured here is from the second report and compares teachers’ understanding with the actual size of the maximum bonuses that they could receive. This chart will be updated in the third report.)

That also might help them better understand what behaviors are needed to earn a bonus.  Finally, time might be needed for educators to respond to all of the intended policy levers of the program, particularly related to recruitment and hiring.

By Erin Pollard, ERIC Project Officer, NCEE

The ERIC Help Desk frequently receives questions from academic users asking if we have resources to help students and faculty use the Institute of Education Sciences' free online library of education research. In response, ERIC has created several videos that can be embedded in LibGuides (a content management system used by libraries), linked to in syllabi, and posted on course websites. These videos can also be accessed from ERIC’s Multimedia Page.

Our newest videos include:

About ERIC: The About ERIC video is an introduction to our digital library and answers basic questions about who our website is designed for and what types of information you’ll find there. Users can view this video to determine if ERIC is the right resource to use for their research.

Using ERIC to Write a Research Paper: The ERIC Help Desk has received a lot  of requests from students asking how to use ERIC to write a research paper.  This video was specifically created to be a resource that librarians and faculty members could direct students to as a starting point for research projects. For more information on this video, stay tuned for next week’s blog post!

Searching eric.ed.gov: ERIC was redesigned in 2013 using intuitive search technology, as opposed to more traditional searching mechanisms. This video explains how our search engine works and offers tips to help users get the best results.

How ERIC Selects New Sources: Users frequently ask how sources are selected to be indexed in ERIC. This video describes the process laid out in the ERIC Selection Policy and explains how users can recommend new resources for our collection.

IES Public Access Policy: How Grantees and Contractors Meet Requirements by Submitting Work to ERIC: Are you conducting research funded by IES? If so, then you are likely subject to the IES Public Access Policy (learn more about the policy for publications and data).  This video explains how researchers can meet their compliance requirements by submitting their work to ERIC.

Tips for Using the ERIC Online Submission System:  ERIC encourages authors to submit their work through our Online Submission System. Learn what types of materials are eligible to be included and how to prepare publications for submission to ERIC.

Have suggestions for other videos ERIC should make? Email our Help Desk! To learn more about ERIC, sign up for our Newsflash and follow us on Facebook and Twitter.

By Vanessa Anderson, Research Scientist, NCEE

The What Works Clearinghouse (WWC) was founded in 2002 and, in its first decade, focused mainly on reviewing studies of programs, policies, products and practices—or interventions—for improving student outcomes in pre-K, elementary and secondary schools. But in 2012, the WWC broadened its focus and has been using rigorous standards to review studies of interventions designed to increase the success of students in postsecondary education.

This week, the WWC launches a new topic—Supporting Postsecondary Success—and it is a good time to look at the work we’re doing, and will do, in the postsecondary area. 

The WWC postsecondary topic area includes reviews of studies on a wide range of interventions, including learning communities, summer bridge programs, multi-faceted support programs, academic mentoring, and interventions that aim to reduce performance anxiety. As of today, 294 postsecondary studies have been reviewed by the WWC. Those reviews are summarized in six Intervention Reports, 25 Single Study Reviews, and four Quick Reviews. And there’s much more in the works!  For instance, a WWC Educator’s Practice Guide that includes strategies for supporting students in developmental education is planned for publication later this year. (Learn more about Practice Guides)

Identifying Studies for Review

In the postsecondary topic area, there are currently three main ways that studies are identified by the WWC for review.

The first is studies that are reviewed for WWC Intervention Reports. All WWC Intervention Reports use a systematic review process to summarize evidence from all available studies on a given intervention. The WWC conducts a broad search for all publicly available studies of interventions that are related to the topic. This process often identifies hundreds of studies for review. The effectiveness studies are then reviewed against WWC standards. Only the highest quality studies are summarized in an Intervention Report.

We released two new intervention reports this week as part of our new Supporting Postsecondary Success topic. You can view the new Intervention Reports on Summer Bridge programs and first-year experience courses on the WWC website.

The second way that studies are reviewed by the WWC is through Quick Reviews, which are performed on studies that have received a great deal of media attention. In these reports, the WWC provides a brief description of the study, the author-reported results, and a study rating. We like to think of Quick Reviews as a way to help people decide whether to fully believe the results of a study, based on the research design and how the study was conducted. For example, we released a quick review earlier this month that focused on a study of computer usage and student outcomes for a class at the U.S. Military Academy at West Point.

Finally, the WWC reviews postsecondary studies submitted as supporting evidence for discretionary grant competitions funded by the U.S. Department of Education, such as the Strengthening Institutions Program, First in the World and TRIO Student Support Services. These grant competitions require applicants to submit studies as evidence of the effectiveness of the interventions they propose to implement. The WWC reviews these studies and includes the results of those reviews in our database.

If you want to see all the studies on postsecondary interventions that have been reviewed by WWC you can check out—and download—the Reviewed Studies Database. In the “Topic Areas” dropdown menu, just select “Postsecondary,” and then easily customize the search by rating, publication type, and/or reasons for the review (such as a grant competition).  

For more information, visit the WWC postsecondary topic area on the website. To stay up-to-date on WWC news, information, and products, follow us on Facebook, Twitter and sign up for the WWC newsflash!

By Diana McCallum, Education Research Analyst, What Works Clearinghouse

It’s been more than a decade since the first What Works Clearinghouse reports were released and we have a wealth of information and resources that can help educators and leaders make evidence-based decisions about teaching and learning. Since 2005, the WWC has assessed more than 11,500 education studies using rigorous standards and has published hundreds of resources and guides across many content areas. (View the full version of the graphic to the right.) 

The WWC website has already received more than 1.7 million page views this year, but if you haven’t visited whatworks.ed.gov lately, here are five reasons you might want to click over:

1) We are always adding new and updated reviews. Multiple claims about programs that work can be overwhelming and people often lack time to sift through piles of research. That’s where the WWC comes in. We provide an independent, objective assessment of education research. For example, we have intervention reports that provide summaries of all of the existing research on a given program or practice that educators can use to help inform their choices.  In addition, when a new education study grabs headlines, the WWC develops a quick review that provides our take on the evidence presented to let you know whether the study is credible. In 2015, we added 43 publications to WWC and we’re adding more every month this year.

2) We’ve expanded our reach into the Postsecondary area. In late 2012, the WWC expanded its focus to include reviews of studies within the Postsecondary area to capture the emerging research on studies on a range of topics, from the transition to college to those that focus on postsecondary success.  To date, the WWC has reviewed over 200 studies on postsecondary programs and interventions, and this area continues grow rapidly. In fact, several Office of Postsecondary Education grant competitions add competitive priority preference points for applicants that submit studies that meet WWC standards. (Keep an eye out for a blog post on the postsecondary topic coming soon!)

3) You can find what works using our online tool. Wondering how to get started with so many resources at your fingertips? Find What Works lets you do a quick comparison of interventions for different subjects, grades, and student populations. Want to know more about a specific intervention? We’ve produced more than 400 intervention reports to provide you the evidence about a curriculum, program, software product, or other intervention for your classroom before you choose it.  Recently, we’ve added a feature that allows a user to search for interventions that have worked for different populations of students and in different geographic locations. As we mentioned in a recent blog post, the Find What Works tool is undergoing an even bigger transformation this September, so keep visiting!

4) We identify evidence-based practices to use in the classroom. The WWC has produced 19 practice guides that feature practical recommendations and instructional tips to help educators address common challenges. Practice guides (now available for download as ebooks) provide quick, actionable guidance for educators that are supported by evidence and expert knowledge within key areas.  Some of our guides now feature accompanying videos and brief summaries that demonstrate recommended practices and highlight the meaning behind the levels of evidence. The work of practice guides are also actively disseminated during Regional Educational Laboratory (REL) Bridge events. For instance, REL Southwest held a webinar on Teaching Math to Young Children, which was based on a WWC practice guide. For more information, read a previously published blog post on practice guides.

5) We compile information by topic. Our “Special Features” pages focus on common themes in education, such as tips for college readiness, information for heading back to school, and guidance for what works in early childhood education. These Special Features provide a starting point to access a variety of WWC resources related to a topic.

In the coming months, we’ll post other blogs that will explore different parts of the WWC and tell you about ongoing improvements. So keep visiting the What Works website or signup to receive emails when we release new reports or resources. You can also follow us on Facebook and Twitter.

The What Works Clearinghouse is a part of the National Center for Education Evaluation and Regional Assistance in the Institute of Education Sciences (IES), the independent research, evaluation, and statistics arm of the U.S. Department of Education. You can learn more about IES’ other work on its website or follow IES on Twitter and Facebook.