A r c h i v e d  I n f o r m a t i o n

Program Evaluations and Other Studies

The Department's Planning and Evaluation Service coordinates evaluation activities to provide valid and reliable information for performance measurement in this plan. In addition, the Department and other federal agencies fund a variety of assessments, statistical studies, and grantee performance data systems that provide information useful for this plan. This section provides an overview of the evaluations and coordination activities critical to effective data use. Detailed information on the performance indicators, including data sources and examples of baseline or related data is provided in Appendix A. Descriptions of the program evaluations, national statistical studies and assessments, and grantee performance data systems supporting this plan are provided in Appendix B.

Evaluation Strategies

Examples of existing or planned evaluation and assessment strategies to provide sound performance measurement through program evaluation include:

• Systematic collection of elementary, secondary, and postsecondary student outcome data using federal and state statistical sources. (National Assessment of Educational Progress; National Postsecondary Student Aid Study; Department of Labor's National Longitudinal Study of Youth)

• Wider use of customer surveys. (National Direct Student Loan Program Evaluation)

• Expert reviews of the quality of Department-funded research, development, and program improvement programs and products. (Evaluation of the Regional Education Labs)

• Impact evaluations using time-series design or experimental design (random assignment/control group methodology) to provide comparative information on program effectiveness and to help identify "what works." (Longitudinal Evaluation of School Change and Performance; Evaluation of the TRIO Upward Bound program; National Evaluation of the School-to-Work Program; Evaluation of Effective Adult Basic Education Programs)

• Program evaluation guidance and assistance to program office staff and grantees to improve the reporting and use of sound program information. (Guide to Program Outcome Measurement for the U.S. Department of Education; Title I Parent Compact Guide)

• Increased use of management evaluations, such as ones planned on the Department's new performance appraisal system, the human resource re-engineering, and the flexiplace system.

Data Coordination and Burden Reduction

Internal coordination. Several initiatives for internal coordination are underway. Two key ones include:

• Cross-office coordination of data. In 1996, the Department established the Data Coordination Committee reporting to the Deputy Secretary with representatives from all offices. Its mission is to review and resolve data quality, burden, and duplication issues. The committee also works with key partners such as state education agencies in this process.

• Staff reviews of performance indicators. Under the leadership of the Deputy Secretary, three Department offices--the Planning and Evaluation Service, Budget Service, and National Center for Educational Statistics--are working collaboratively to review program performance indicators with the responsible program offices for content, methodology, and quality of data sources.

Coordination with other federal agencies. Program offices have worked with evaluation and statistics agencies in other federal departments--for example, using the Department of Health and Human Services' Monitoring the Future survey for data useful for drug prevention, the National Science Foundation's data on minority participation in science and engineering, and Labor's National Longitudinal Study of Youth for school to work and other topics involving youth follow-up.

ED's Inspector General. The Inspector General will be conducting independent reviews of the reliability and validity of selected performance measures.

Examples of Program Evaluations

For more detailed descriptions of the evaluations, statistical studies, assessments, and other data systems that support the goals, objectives, and indicators of this plan, see Appendix B.

Program evaluations that support this plan include:

Goals 1 and 2 (K-12) Outcome Indicators

• Special analyses of data from the National Assessment of Educational Progress on the progress of Title I and high-poverty schools and disadvantaged students relative to the nation.
• Analyses of annual state and local performance reports on student results from state and local assessments.
• Assessment information through the Longitudinal Evaluation of School Change and Performance, supplemented with data from the Longitudinal Survey of School Implementation of Standards-Based Reform and Title I.

• Crosscutting Evaluation of Federal Efforts to Assist in School Reform, which will provide baseline indicators of the planning process and early implementation of standards development and key provisions in federal programs, including awareness, problems encountered, and assistance and support from Goals 2000, Title I and other sources.
• Targeting and Resource Allocation Study, which will examine how federal resources from Goals 2000 and Elementary and Secondary Education Act grants are used at the school and district levels to support legislative objectives.
• National Evaluation of School-to-Work Systems, a five-year study, that is determining whether school-to-work is a viable long-term educational strategy that is accessible to all students.
• Study of Local Educational Agency Activities under the Safe and Drug Free Schools and Communities Act (SDFSCA), which will collect information on local policy development, planning, implementation, and evaluation of SDFSCA activities, including progress in achieving measurable goals and objectives and in using research-based approaches.
• Study on School Violence and Prevention: Evaluation of Promising Programs, which will report on disorder and violence in schools nationally, the effectiveness of promising approaches to preventing violence in schools, and distinguishing characteristics of safe schools in various locales.
• Evaluation of Title II Professional Development, which is examining the quality of professional development supported by the Title II Eisenhower Program, its alignment with broader reform efforts, and its impact on teachers' classroom practice.
• Evaluation of the Charter Schools Program, a new study planned for FY 1998, that will track the effects of federally supported charter schools on increasing school flexibility and performance accountability for meeting state content and performance standards.
• Evaluations of federal support for technology will examine the extent to which federal programs are promoting access to and effective use of modern technology that is supportive of challenging curriculum and instruction and the potential for well-implemented technology to improve learning.

• Longitudinal Evaluation of School Change and Performance, which is assessing the extent to which systemic reform can work for all children when properly implemented. It will analyze how curriculum and instruction are changing in leading reform states and the impact in Title I schools.
• Longitudinal Survey of Schools, a study that includes special samples of schools that serve large concentrations of limited English proficient, migrant, and Native American students.
• National Evaluation of Even Start, which documents how these intergenerational literacy projects operate; progress and outcomes for families--including the readiness of young children for school; and promising practices that may be used for program improvement.

Goal 3. Ensure Access to Postsecondary Education and Lifelong Learning.

• Upward Bound Evaluation, a random assignment evaluation designed to assess effects of Upward Bound on students' preparation for college, high school graduation, college entry and achievement.
• College preparation studies, a set of related research studies, focus groups, and surveys to help formulate the national campaign aimed at encouraging middle-school students and their parents to prepare for college.
• Secondary data analyses of the effects of student aid programs, that analyze NCES and Census data to assess the targeting of aid, maintaining affordability to postsecondary education, and equalizing educational attainment across income groups.
• Student Support Services Evaluation, a longitudinal evaluation assessing the effects of TRIO's Student Support Services program on college retention and graduation, grades, and transfer behavior from 2-year to 4-year institutions.
• Title III Performance Measurement Study, a study to develop a system of performance indicators for judging the success of the Title III Strengthening Developing Institutions programs.
• Direct Loan Evaluation, a comprehensive evaluation of management outcomes (institutional, student, and parent satisfaction; operational efficiency; and Departmental support).
• Customer satisfaction surveys, surveys of postsecondary institutions and students satisfaction with the operation and delivery of the student aid programs.
• Gatekeeping studies, a set of studies examining how various components of the gatekeeping process--e.g., accreditation, state licensure, outcome reporting--are functioning.
• "What Works" Study for Adult English as a Second Language (ESL) Students, a comparative evaluation of which instructional practices are most effective for adult ESL students who have low native-language literacy and a description of adult ESL at the classroom and program levels in the six states that comprise more than 75% of all ESL enrollments.
• Evaluation of Effective Adult Basic Education Programs and Practices, a study that is identifying and validating the key elements of effective programs for low-literate learners.
• Longitudinal Study of the Vocational Rehabilitation Service Program, a study of outcomes achieved by rehabilitation services consumers and the extent of return on the program's investment.

[Key External Factors that Could Affect Achievement of ED's Goals and Objectives]

[Development of the Plan]