Detailed Information on the
National Writing Project Assessment

Work with Congress to terminate this duplicative program.

Collect baseline performance data and establish targets for new program annual performance measures.

Establish long-term performance measures and finalize program efficiency measures.

Implement an independent program evaluation to obtain reliable program outcome information.

Develop performance metrics that measure, as directly as possible, the impact of program services on teacher effectiveness and/or student learning.

Create a program evaluation strategy, along with a schedule for an independent program evaluation, to obtain reliable program outcome information.

Ensure that valid and reliable performance data are collected on an annual basis.

Measure: The percentage of sites that surpass all NWP quality review criteria in the areas of : 1) overall adherence to the NWP model; and 2) structural support and strategic effectiveness.

Explanation:Every NWP site undergoes an annual peer review. The peer review process utilized is designed, developed, and implemented by the NWP. Reviewers (mostly teachers from other sites) use a structured protocol to rate sites based on established criteria. Sites achieve one of three quality ratings as a result of this review: 2=meets or exceeds all criteria; 1 = requires technical assistance; and 0= not recommended for funding. The peer review process considers three types of NWP professional development programs offered at each site (summer institute, in-service programs, and continuity programs), as well as each site's overall adherence to the NWP professional development model. Specifically, this process is intended to rate structural support and strategic effectiveness for each site using the following criteria: a) adequacy of institutional partnerships (i.e., between universities as host institutions and schools in the service area); b) development and deployment of teacher leadership; and c) knowledge of educational context and challenges in the site geographic area.

Measure: The percentage of NWP summer institute training syllabi deemed to be of high quality by an independent review panel of qualified experts.

Explanation:Data collection methodology are still being developed. The Department will conduct expert panel reviews, using qualified experts and a Department-developed scoring rubric. Data are not yet available. The Department expects to convene this panel for the first time during the Fall of 2008.

Measure: The percentage of NWP summer institute training session participants who improve the quality of the writing assignments given, as demonstrated through an independent review of lesson plans by a panel of qualified experts or individuals.

Explanation:Data collection methodology are still being developed. The Department will conduct expert panel reviews, using qualified experts and a Department-developed scoring rubric. Data are not yet available. The Department expects to convene this panel for the first time during the Fall of 2008.

Measure: The average annual Federal cost per contact hour for educators participating in NWP training and professional development.

Explanation:This measure provides the average annual Federal cost to support a single contact hour of National Writing Project programming to an individual participant. Contact hours are calculated at the site level, on a "per program" basis, by identifying the total number of participants in each NWP program multiplied by the average number of hours per participant for that program. Contact hour calculations are averaged across all 196 current National Writing Project sites to obtain a national average for a single year period. The average national Federal cost per contact hour is calculated for each site using just the Federal award to each site as the base.

Measure: The average annual total cost per contact hour for educators participating in NWP training and professional development.

Explanation:This measure provides the average annual total cost to support a single contact hour of National Writing Project programming to an individual participant. Contact hours are calculated at the site level, on a "per program" basis, by identifying the total number of participants in each NWP program multiplied by the average number of hours per participant for that program. Contact hour calculations are averaged across all 196 current National Writing Project sites to obtain a national average for a single year period. The average national total cost per contact hour is calculated by using NWP sites' total income from all sources, including NWP Federal appropriations, as the base.

Is the program purpose clear?

Explanation: By statute, the purposes of the National Writing Project (NWP) are to: 1) "support and promote the expansion of the NWP network so that teachers in every region of the U.S. have access to an NWP program;" 2) "ensure the consistent high quality of the sites through ongoing review, evaluation, and technical assistance;" and 3) "support and promote the establishment of programs to disseminate effective practices and research findings about the teaching of writing."

Evidence: ESEA, Title II, Part C, Supbart 2

Does the program address a specific and existing problem, interest or need?

Explanation: Available research suggests that writing is an essential learning skill, foundational to learning in other content areas. The National Assessment of Educational Progress (NAEP) 2002 Writing Report Card states, "...the ability to write clearly is a critical skill for advancing knowledge, enhancing competence, posing new ideas, and making those ideas comprehensible to an information-dependent citizenry." Student achievement data in writing are mixed. While average student scores on the NAEP writing assessment increased slightly between 1998 and 2002 in grades 4 and 8, there was no improvement for students in grade 12 over the same period. Also, while most students have mastered "basic" levels of proficiency in writing, far fewer students demonstrate "proficient" and "advanced" skills.

Evidence: "The National Report Card: Writing 2002" (http://nces.ed.gov/pubsearch/ pubsinfo.asp?pubid=2003529) Also see "The Neglected "R" -- The Need for a Writing Revolution: The National Commission on Writing in America's Schools and Colleges." (The National Writing Commission and the College Board, April, 2003).

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: There is some overlap between the purpose and targeted beneficiaries of the NWP and Improving Teacher Quality State grants programs. While NWP's non-profit membership service delivery model (which now has 185 sites) may be different than typical Federally-supported professional development in writing, there is no evidence available that this model is effective and not redundant of activities supported by Teacher Quality State Grants

Evidence: N/A

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The statute names the National Writing Project, a national non-profit educational organization, as the sole eligible grantee for NWP funding. This requirement is at odds with both the Department of Education's (ED) regulations and the President's Management Agenda, which are devoted to the premise that Federal investments are more effectively targeted through competitive processes rather than directed awards. However, it is not clear that NWP's administration of subgrants to participating sites is less thorough than comparable Federal administration. NWP conducts annual performance reviews of all 185 member sites, and includes performance clauses in all site subcontracts. In addition, the program has a 50% matching requirement that serves to, in effect, double the impact of the Federal NWP investment.

Evidence: See program authority, ESEA, Title II, Part C, Subpart 2. See also the Education Department General Administrative Regulations (EDGAR), 34 CFR Part 75. Review of NWP management files suggests that approximately 95 percent of sites are considered "acceptable" in annual performance reviews.

Is the program effectively targeted, so that resources will reach intended beneficiaries and/or otherwise address the program's purpose directly?

Explanation: The NWP effectively targets intended program beneficiaries (e.g., teachers and students), and program funds are also well targeted to meet program purposes. Nearly all local NWP sites receive some support from a variety of non-Federal sources, including host campuses and private corporations -- but, program quality would likely decrease without the Federal subsidy.

Evidence: NWP files of site performance reviews. Review of NWP performance clauses in sub-contracts with sites.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The program does not have long-term measures.

Evidence: N/A

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: N/A - see above.

Evidence: N/A

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: ED and OMB are developing annual measures that will reflect the extent to which the students of teachers that received training at an NWP site develop their writing skills. The major obstacle to finalizing related measures is the wide divergence in approaches used by individual NWP sites in measuring student achievement.

Evidence: N/A

Does the program have baselines and ambitious targets for its annual measures?

Explanation: No data exist yet for this program, thus, there is not yet any meaningful basis for establishing baselines and targets.

Evidence: N/A

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: The program has not yet established short- and long-term performance measures, and as such, cannot demonstrate that all partners work towards such goals. When the program has established measures, ED will closely review the work of the NWP organization in monitoring its funded sites and conducting grant reviews that assess progress against the program's measures.

Evidence: N/A

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: The NWP organization is using evaluation funds to support site-based evaluation activities. At the national level, NWP has employed two approaches to determining the effectiveness of its programs -- both national evaluations. The first approach focused on teacher satisfaction and impact on educational practice, while the second approach attempted to measure effects on student performance. While both studies suggest that NWP may support programs that have positive effects on student outcomes, neither approach is sufficiently rigorous to yield reliable information on the effectiveness of NWP-supported interventions. For example, the latter evaluation found a significant increase in the writing skills of students in the NWP teachers' classrooms, but failed to compare these gains to comparable control groups or carefully matched comparison groups. As such, ED's Institute of Education Sciences concluded that it is not possible to draw any reliable conclusions regarding impact on student learning in NWP classrooms relative to comparable non-NWP classrooms.

Evidence: www.writingproject.org/pressroom/impact.html -- See the Inverness Research Associates evaluation, and the Academy for Educational Development evaluation. It is also noteworthy that NWP recently made modest evaluation resources (up to $20k) available to local NWP sites, through the Local Sites Research Initiative. For details, see: www.writingproject.org/cs/nwpp/print/nwpn/21

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: Without long- and short-term goals that yield reliable and appropriate program outcomes data, it is not possible to link the budget request to accomplishment of such goals. Budgeting is not currently linked to long-term goals and/or a strategic plan.

Evidence: N/A

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: Although ED has very few tools to employ in identifying and correcting strategic planning deficiencies because this is a directed grant, ED program staff are working to identify a limited number of annual performance goals. While most of the necessary strategic planning for NWP is conducted by the grantee, ED program staff are also working closely with NWP to determine the most useful and cost-effective site-based evaluation strategies. ED has a limited oversight role in strategic planning and management of activities supported under this program, but could assume a better defined accountability role in planning and managing program activities. The current staff role is largely limited to providing technical assistance as requested by the grantee.

Evidence: Grantee Annual Performance Reports; discussion with program staff; on-going monitoring of NWP interactions with ED staff.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The NWP regularly collects performance data and uses it to adjust program resources and take other appropriate management actions. For example, the 185 NWP sites must apply (to the NWP national office) to receive continued funding annually. This panel review process yields data on the type, depth, and reach of programs offered by sites, as well as panelist opinions on the quality and rigor of such programmatic offerings. Sites receive written feedback intended to reinforce strengths and offer advice for improvement. Each year, approximately 2 to 4 sites were not recommended for funding and were closed. However, given the lack of program measures for NWP, it cannot be said that ED has collected the baseline performance data necessary to set meaningful, ambitious performance targets.

Evidence: FY 2004 Annual Report, pages 15-17. Among the measures that are being used at the various sites are: State English language arts tests, a six-trait scoring rubric, the Miller and Daly Writing Apprehension Test; a state writing assessment; the Proficiency Sample Essay Revision Task Scoring Rubric (OH); the PAWLP Domain Scoring Guide (PA), the Observation Matrix for Writing Research, and prompts and rubrics modeled after the statewide writing exam. (FY 2004 Annual Report, pages 17-18). ED program staff meet twice a year with NWP leaders and keep in contact with the NWP's Deputy Director at least twice a month.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: Currently, ED cannot demonstrate how federal managers and program partners are held accountable for program goals. However, the Department has initiated several efforts to improve accountability in its programs. First, ED is in the process of ensuring that EDPAS plans -- which link employee performance to relevant Strategic Plan goals and action steps ' hold Department employees accountable for specific actions tied to improving program performance. ED is also revising performance agreements for its SES staff to link performance appraisals to specific actions tied to program performance. Finally, ED is reviewing its grant policies and regulations to see how grantees can be held more accountable for program results.

Evidence: The President's Management Agenda scorecard (Human Capital and Budget & Performance Integration initiatives) notes ED's efforts to improve accountability. The Department's Discretionary Grants Improvement Team (DiGIT) recommendations indicate that ED is reviewing its grant policies and recommendations.

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: The program successfully obligates funds by the end of each fiscal year. Funds are spent for the intended purposes; this is assessed through grant monitoring. No improper uses of funds have been identified.

Evidence: Funds are alloted and obligated on schedule, during the 3rd quarter every fiscal year.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: To date, the Department has not established procedures for this program to measure and achieve efficiencies in program operations. However, ED is in the process of developing its competitive sourcing Green Plan, and is working to improve the efficiency of its grantmaking activities. The Department has also established a strengthened Investment Review Board to review and approve information technology purchases agency-wide.

Evidence: Department Investment Review Board materials. ED's Discretionary Grants Improvement Team (DiGIT) recommendations.

Does the program collaborate and coordinate effectively with related programs?

Explanation: At the national level, NWP recently collaborated with the College Board in developing and implementing a 30 minute writing test that will soon be part of the SAT exam. In 2003, they also collaborated on an advisory panel for the National Commission on Writing, which produced the publication "The Neglected "R" -- The Need for a Writing Revolution." Local NWP site partnerships are also noteworthy. For example, the Philadelphia Writing Project collaborates with 2 middle and high school GEAR UP Writing Centers (another ED program), in conjunction with Temple University. It also partners with the University of Pennsylvania's teacher professional development program to manage 3 Philadelphia public schools. Beyond such national and local partnerships, NWP also sponsors special-focus networks in target areas like English Language Learners, Urban Sites, and Rural Sites. Each of these special focus networks hold national and regional meetings, publish newsletters and articles.

Evidence: The Voice - NWP's national newsletter; NWP annual reports, program files, NWP Program Annual Reports to Task Force, 2003.

Does the program use strong financial management practices?

Explanation: No internal control weaknesses have been reported by auditors. The Department has a system for identifying excessive draw downs, and can put individual grantees on probation where draw downs need to be approved. NWP also is subject to regular audits by their accountants.

Evidence: Review of program files; NWP audit documents.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: ED is currently developing performance measures for the program that will enable better targeted program oversight and technical assistance.

Evidence: N/A

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: By statute, the only entity eligible to receive support under this program is the National Writing Project. By definition, this earmarking does not allow program funding to be subject to compeititon. At the local level, NWP uses a competitive process for making subgrants, and site continuation awards are subject to annual performance reviews.

Evidence: Review of NWP program files; performance clauses in NWP contracts with local sites.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: ED program staff have a strong relationship with the grantee, and demonstrate a high level of understanding of how grantees use Federal resources. NWP national office staff also maintain very close ties to all 180+ local NWP sites, conducting routine site visits, annual performance reviews, and regular on-line discussions.

Evidence: Discussion with program staff; review of NWP reporting structure and oversight techniques used with local NWP sites.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: NWP GPRA data are now reported in several formats, and made available to the public through a public website; however, grantee performance reports are not made available to the public in a transparent and meaningful manner. Education is developing a department-wide approach to improve the way programs provide performance information to the public. In 2004, Education will conduct pilots with selected programs to assess effective and efficient strategies to share meaningful and transparent information.

Evidence: Review of NWP website and publicly available ED data collections relating to NWP.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The program does not yet have any long-term performance goals.

Evidence: N/A

Does the program (including program partners) achieve its annual performance goals?

Explanation: Program is currently working to develop and implement appropriate annual performance goals.

Evidence: N/A

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The Department has not yet developed an efficiency measure for this program.

Evidence: N/A

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: Although programs with shared goals and purposes exist in other areas of education (e.g. - ESEA Title II, Improving Teacher Quality State Grants program), and various private foundation programs also focus on teacher quality, there is no reliable basis for comparing the National Writing Project to such programs. No current studies, analyses, or evaluations have attempted to make such comparisons, and in the absence of reliable comparisons between these programs further analysis would be arbitrary.

Evidence: N/A

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: While the Department has not conducted any evaluations of this program, the NWP has employed two approaches at the national level to determine the effectiveness of its programs. The first approach focused on teacher satisfaction and impact on educational practice. The second approach measured the effects on student performance through writing assessments. Unfortunately, both approaches present significant methodological concerns that seriously limit the overall reliability of findings. For example, under the second approach mentioned above, because the evaluation did not use control groups or carefully matched comparision groups it is not possible to draw any reliable conclusions regarding program impact on student learning in NWP classrooms relative to any other classrooms where writing skills are taught.

Evidence: http://www.writingproject.org/pressroom/impact.html -- See the Inverness Research Associates evaluation, and the Academy for Educational Development evaluation. It is also noteworthy that NWP recently made modest evaluation funding (up to $20k) available to local NWP sites, through the Local Sites Research Initiative. For details, see: http://www.writingproject.org/ cs/nwpp/print/nwpn/21.