Detailed Information on the
Byrd Honors Scholarships Assessment

Implementing a strategy for making program performance data accessible to the public in a transparent way.

Developing independent program evaluations of sufficient scope and quality to support program improvements and evaluate program effectiveness and relevance to the problem, interest, and need.

Proposing no funding for this program in the 2006 budget and directing postsecondary scholarship program resources to need-based programs.

Exploring strategies to address those legal barriers that hinder efforts to improve program performance.

Develop and implement technical assistance and data quality strategies to improve performance reporting.

Measure: Persistence rate of Byrd Scholars remaining in school after three years of study.

Explanation:The percentage of Byrd Scholars remaining in school after 3 years of study. The number of scholars in their first three years of study who are enrolled at the end of the academic year divided by the total number of scholars enrolled at the beginning of their first three years of study.

Measure: Graduation rate of Byrd Scholars.

Explanation:The percentage of Byrd recipients graduating within 4 years.

Measure: Cost per persistence or graduation

Explanation:The cost of a successful outcome, where success is defined as persistence or graduation

Is the program purpose clear?

Explanation: The Byrd Honors Scholarships program promotes academic excellence and achievement by awarding merit-based scholarships to high school seniors through formula grants to State educational agencies, who have demonstrated outstanding academic achievement and who show promise of continued academic excellence.

Evidence: Title IV, Part A, Subpart 6, Sec. 419A of the Higher Education Act (HEA) states that the purpose of the program is, "to promote student excellence and achievement and to recognize exceptionally able students who show promise of continued excellence."

Does the program address a specific and existing problem, interest or need?

Explanation: The overwhelming majority of Federal Student Aid is awarded according to a need-based criteria. As a merit-based alternative, the Byrd program provides States with resources to reward high-performing high school students and create incentives for students to excel and continue their education in a post-secondary setting. By encouraging high school students to work hard and get good grades, these scholarships are designed to increase the likelihood that these students will enter postsecondary education with the preparation necessary to succeed.

Evidence: Research shows that U.S. students are not entering college with the skills they need to to succeed. Studies have shown that students who undertake rigorous coursework in high school are more likely to succeed in postsecondary education (Adelman, Clifford, Academic Intensity, Attendance Patterns, and Bachelor's Degree Attainment, Washington, DC: U.S. Department of Education, 1999). However, according to a recent study by the Manhattan Institute, while 70% of all students in public high schools graduate, only 32% leave high school academically prepared for college.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: This is the only Federal program that awards scholarships to students nationally based solely on merit and across all academic areas. However, at the State, local and institutional level there are numerous programs that provide merit-based resources for post-secondary education.

Evidence: N/A

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: Implementation of the program has revealed certain flaws in the program design that limit the program's effectiveness or efficiency. They include: 1) Allowing States to establish unique eligibility criteria. This eligibility variation limits the effectiveness of the program at the national level. 2) Prohibition on State use of funding for collecting performance data. This restriction unnecessarily harms the quality of data provided to the Department of Education (ED).

Evidence: Title IV, Part A, Subpart 6, Sec. 419 of the Higher Education Act

Is the program effectively targeted, so that resources will reach intended beneficiaries and/or otherwise address the program's purpose directly?

Explanation: The program is focused on rewarding high-performing high school students and encouraging them to go on to postsecondary education. By stipulating that recipients have demonstrated a high level of academic achievement, the statute ensures that the program is effectively targeted. However, there is no evidence to suggest that scholarship recipients would otherwise be unable to attend college. It is therefore reasonable to suggest that this program subsidizes activities that would have occured without the program.

Evidence: Title IV, Part A, Subpart 6, Sec. 419C (a) of the Higher Education Act

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: In order to track the effectiveness of the program in achieving its goals, ED has developed a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program. They are: 1) The persistence rate of Byrd scholarship recipients 2) The graduation rate of Byrd scholarship recipients

Evidence: The measures reflect performance with regards to the ultimate goals of the program, to help support high-achieving high school seniors to participate in and complete postsecondary education.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: Long term targets have been finalized and exceed national averages for graduation and retention of college students.

Evidence: N/A

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: In order to track the effectiveness of the program in achieving its goals, ED has developed a limited number of specific annual performance measures that focus on outcomes and meaningfully reflect the purpose of the program. They are: 1) The persistence rate of Byrd scholarship recipients 2) The graduation rate of Byrd scholarship recipients In addition, the program's efficiency measure will track the cost per successful outcome (college persistence and graduation).

Evidence: The measures reflect performance with regards to the ultimate goals of the program, to help support high-achieving high school seniors to participate in and complete postsecondary education. ED has also developed an efficiency for the program which will focus on the cost per successful outcome, where successful outcome is defined as the number of students who persist and graduate.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: Annual targets have been finalized and exceed national averages for graduation and retention of college students.

Evidence: N/A

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: With the recent development of new annual and long-term performance measures, States have not yet been able to commit to these new measures. The program plans outreach to the States to communicate the new measures and integrate them into each State's work plan. The program has revised the Annual Performance Report to gather the necessary data for the new measures.

Evidence: Annual performance reports

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: No evaluations have been undertaken for this program.

Evidence: N/A

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: ED has not satisfied the first part of the question because program performance changes are not identified with changes in funding levels. The program, at this time, does not have sufficiently valid and reliable performance information to assess (whether directly or indirectly) the impact of the Federal investment. However, ED has satisfied the second part of this question in that ED's budget submissions show the full cost of the program (including S&E). ED's 05 integrated budget and performance plan includes the program's annual and long-term goals.

Evidence: N/A

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The program has identified strategic planning deficiencies and taken meaningful steps to address these deficiencies. ED has revised the efficiency, annual and long-term perfomance measures for the Byrd program. In addition, the Department of Education has revised program materials, including its annual performance reports, to reflect the new Byrd scholarship measures.

Evidence:  

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: Although the collection of performance information has been improved by revisions to the Annual Performance Report (APR), the data is not customarily used for program management.

Evidence: ED plans to use the data from the revised APR to target program management when it becomes available.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: Currently, ED cannot demonstrate how federal managers and program partners are held accountable for program goals. However, the Department has initiated several efforts to improve accountability in its programs. First, ED is in the process of ensuring that EDPAS plans -- which link employee performance to relevant Strategic Plan goals and action steps ' hold Department employees accountable for specific actions tied to improving program performance. ED is also revising performance agreements for its SES staff to link performance appraisals to specific actions tied to program performance. Finally, ED is reviewing its grant policies and regulations to see how grantees can be held more accountable for program results.

Evidence: The President's Management Agenda scorecard (Human Capital and Budget & Performance Integration initiatives) notes ED's efforts to improve accountability. The Department's Discretionary Grants Improvement Team (DiGIT) recommendations indicate that ED is reviewing its grant policies and recommendations.

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: ED obligates all funds obligated according to an annual spending schedule that is established at the beginning of the fiscal year. At the partner level, grantees are obligating funds at an acceptable rate.

Evidence: Annual Spending Plans and review of program financial records.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: To date, the Department has not established procedures for this program to measure and achieve efficiencies in program operations. However, ED is in the process of developing its competitive sourcing Green Plan, and is working to improve the efficiency of its grantmaking activities. The Department has also established a strengthened Investment Review Board to review and approve information technology purchases agency-wide.

Evidence: Department Investment Review Board materials. ED's Discretionary Grants Improvement Team (DiGIT) recommendations.

Does the program collaborate and coordinate effectively with related programs?

Explanation: The program does not routinely collaborate with related programs, although it set program measures in conjunction with the other Office of Postsecondary Education scholarship programs.

Evidence:  

Does the program use strong financial management practices?

Explanation: No internal control weaknesses have been reported by auditors. Plus, the Department has a system for identifying excessive draw downs, and can put individual grantees on probation which requires ED approval of all grantee draw downs. Additionally, all State financial and administrative systems must meet State finance standards.

Evidence: Program financial records

Has the program taken meaningful steps to address its management deficiencies?

Explanation: A review of management practices has led to management improvements that have significantly improved the administration of the program. The development of performance measures and subsequent alterations to the program's Annual Performance Reports are two prime examples of such meaningful steps.

Evidence: N/A

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: ED reviews the State Annual Performance Reports and monitors State drawdown of funds and expenditures to ensure program integrity. As stated previously, while ED is aware of varying State criteria for awarding Byrd scholarships, statutory constraints prohibit ED from attempting to standardize these criteria.

Evidence: NA

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: GPRA and grantee-level Byrd Scholarships data are now reported in several formats, including on the Department's website. However, this publicly available information is not performance related. Education is developing a department-wide approach to improve the way programs provide performance information to the public. In 2004, Education will conduct pilots with selected programs to assess effective and efficient strategies to share meaningful and transparent information.

Evidence: NA

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: Baseline data is not yet finalized for comparison to the long-term performance measures.

Evidence: ED expects to complete this work by Fall 2004.

Does the program (including program partners) achieve its annual performance goals?

Explanation: Baseline data is not yet finalized for comparison to the annual performance measures.

Evidence: ED expects to complete this work by Fall 2004.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The program has developed an efficiency measure. In future years this measure will allow the Department to assess the relative efficiency of the program.

Evidence: N/A

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: There are not yet data available to compare this program to other ED scholarship programs.

Evidence: The Department may be able to make some comparisons between comparable programs in future years as performance measures for these programs are implemented.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: No evaluations of this program have been conducted and there are currently no plans to conduct such a study.

Evidence: N/A