Detailed Information on the
Javits Fellowships Assessment

Completing a study of ED's graduate fellowship programs and utilizing the results to validate program performance measures and improve program performance.

Establishing targets for the newly created efficiency measure.

Developing strategies to use efficiency and performance data for program improvement purposes.

Establishing an efficiency measure for the program.

Developing and implementing a strategy for making program performance data accessible to the public in a transparent way.

Measure: Percentage of Javits fellows who complete a doctorate within seven years.

Explanation:

Measure: Average number of years required for Javits fellows to attain degree completion.

Explanation:Average time to degree completion for Javits fellows, in years.

Measure: Average cost per terminal degree awarded

Explanation:Cost per successful outcome, where success is defined as attainment of a terminal degree is the designated academic areas.

Is the program purpose clear?

Explanation: The Javits Fellowships program provides financial assistance to students who have demonstrated superior academic ability and achievement, financial need, and exceptional promise to undertake graduate study in the arts, humanities, and social sciences leading to a doctoral degree or a master's degree, where the master's degree is the terminal highest degree awarded.

Evidence: Title VII, Part A, Subpart 1 of the Higher Education Act

Does the program address a specific and existing problem, interest or need?

Explanation: Researchers and policymakers agree that there is a shortage of highly qualified individuals in certain critical academic areas and that this shortage has a detrimental impact in a variety of critical professions.

Evidence: Research shows that inadequate funding is a main obstacle for students who do not pursue an advanced degree (For example; Kerlin, Scott; Pursuit of the PhD.)

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: Of all Federal scholarship programs designed to address needs in specific disciplines, Javits is the only activity to solely support graduate studies in the humanities and social sciences based on the merit and financial need of program applicants. Beyond the Federal government, a number of private organizations also provide fellowships for graduate studies in the humanities and social sciences, but these efforts are either limited in their geographical scope, limited to specific disciplines within the humanities or social sciences, or do not include a financial need component.

Evidence: Title VII, Part A, Subpart 1, Sec. 701 (a) of the Higher Education Act

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: Implementation of the program has not revealed any major flaws in the actual program model that limit the program's effectiveness or efficiency. This program provides fellowships directly to individual students. This model provides program managers with a greater amount of direct control than other ED programs that support postsecondary institutions which, in turn, use their own unique criteria for making fellowships to their students. Due to the workload demands of this direct model, it is only appropriate for programs like the Javits fellowships which make a limited number of awards.

Evidence: In 2003, Javits fellowships funding supported 237 new or existing recipients in their graduate-level studies.

Is the program effectively targeted, so that resources will reach intended beneficiaries and/or otherwise address the program's purpose directly?

Explanation: The program is focused on recruiting talented students with high financial need into graduate studies in areas of national need. By stipulating that recipients have a high level of financial need, and requiring that fellowship applicants demonstrate that need by completing the FAFSA process, the statute ensures that the program is effectively targeting students who have demonstrated achievement, financial need, and exceptional promise.

Evidence: Title VII, Part A, Subpart 1, Sec. 701 (a) of the Higher Education Act. In order to demonstrate high financial need, every fellowship applicant must complete the FAFSA process.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: ED has developed a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program. They are: 1) Percentage of Javits fellows that complete a terminal degree 2) Average time to degree completion for Javits fellows

Evidence: These long-term measures will enable ED to monitor the program's effectiveness in providing fellowships to the appropriate students with the potential to make outstanding contributions to the field. Research demonstrates that the longer students take to complete graduate studies the higher their attrition rate.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The program has developed ambitious targets and timeframes for its long-term measures. Targets for these measures are designed to exceed the national average for students in Javits-eligible fields of study.

Evidence: Javits fellows must demonstrate high financial need. This group of students traditionally takes longer to complete terminal graduate degrees and has a significantly higher attrition rate than the student population as a whole. As such, achieving and maintaining this level of performance would demonstrate that the program is effectively achieving its long-term goals.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: ED has developed a limited number of specific annual performance measures that focus on outcomes and meaningfully reflect the purpose of the program. They are: 1) Percentage of Javits fellows that complete a terminal degree 2) Average time to degree completion for Javits fellows In addition, ED is developing an efficiency measure for this program which will likely focus on the cost per terminal graduate program completer.

Evidence: These annual measures will enable ED to monitor the program's effectiveness in providing fellowships to the appropriate students with the potential to make outstanding contributions to the field. Research demonstrates that the longer students take to complete graduate studies the higher their attrition rate.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The program has developed ambitious targets and timeframes for its annual measures. Targets for these measures are designed to exceed the national average for students in Javits-eligible fields of study.

Evidence: Javits fellows must demonstrate high financial need. This group of students traditionally takes longer to complete terminal graduate degrees and has a significantly higher attrition rate than the student population as a whole. As such, achieving and maintaining this level of performance would demonstrate that the program is effectively achieving its annual goals.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: Of the program's two performance goals, only the time-to-completion measure was in place when the latest round of fellowships were awarded. As such, fellowship recipients have only explicitly commited to this measure. However, the Department believes that graduation is an implicit goal of any fellowship program and this would have been understood to be such by all fellowship recipients.

Evidence: With the recent development of new annual and long-term performance goals, fellowship recipients have not yet been able to explicitly commit to both of these goals. The program plans outreach to fellowship recipients and the IHE's that these fellows attend to communicate the new goal and integrate both performance goals into each recipient's fellowship agreement. The Department has developed a mechanism to collect data on these indicators as part of the Graduate Fellowships Outcomes Study that will be looking at the outcomes of all of the Department's graduate fellowship programs.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: The Department is currently implementing a comprehensive study of all of the graduate fellowship programs in the Office of Postsecondary Education. This study will provide specific data to support the annual and long-term performance measures for this program.

Evidence: The Graduate Fellowships Outcomes Study will look at graduation rates and time-to-degree completion of fellowship recipients. The first impact data will be available in FY 2006.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: ED has not satisfied the first part of the question because program performance changes are not identified with changes in funding levels. The program, at this time, does not have sufficiently valid and reliable performance information to assess (whether directly or indirectly) the impact of the Federal investment. However, ED has satisfied the second part of this question in that ED's budget submissions show the full cost of the program (including S&E). ED's FY05 integrated budget and performance plan includes the program's annual and long-term goals.

Evidence: N/A

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The program has identified strategic planning deficiencies and taken meaningful steps to address these deficiencies. Most significantly, ED has revised the performance measures for the Javits Fellowship program and developed the Graduate Fellowships Outcomes Study to provide performance data to support these measures. The program has also initiated a process to revise program materials, such as application packets and annual performance reports, to reflect its new long-term and annual performance measures.

Evidence: N/A

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: Grant recipients are required to submit need analysis certification reports on an annual basis certifying that fellows are maintaining satisfactory progress towards degree completion. ED uses data gathered from these reports to manage the program and improve program performance. In rare cases where students do not maintain satisfactory progress ED can reallocate program resources to other program applicants. After a review of its oversight processes in FY 2002, ED revised its oversight process to ensure resources can be reallocated before the end of a given fiscal year.

Evidence: Needs analysis certification reports.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: Currently, ED cannot demonstrate how federal managers and program partners are held accountable for program goals. However, the Department has initiated several efforts to improve accountability in its programs. First, ED is in the process of ensuring that EDPAS plans -- which link employee performance to relevant Strategic Plan goals and action steps ' hold Department employees accountable for specific actions tied to improving program performance. ED is also revising performance agreements for its SES staff to link performance appraisals to specific actions tied to program performance. Finally, ED is reviewing its grant policies and regulations to see how grantees can be held more accountable for program results.

Evidence: The President's Management Agenda scorecard (Human Capital and Budget & Performance Integration initiatives) notes ED's efforts to improve accountability. The Department's Discretionary Grants Improvement Team (DiGIT) recommendations indicate that ED is reviewing its grant policies and recommendations.

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: At the Federal level, all funds are obligated according to an annual spending schedule that is established at the beginning of the fiscal year. At the partner level, grantees are obligating funds at a reasonable rate. While ED has lapsed a small amount of Javits fellowships funding in recent years, the lapses have only occured when remaining funds are less than the amount needed to fund one fellowship at the mandatory minimum level, per the program's authorizing statute.

Evidence: Annual Spending Plan and program financial records.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: To date, the Department has not established procedures for this program to measure and achieve efficiencies in program operations. However, ED is in the process of developing its competitive sourcing Green Plan, and is working to improve the efficiency of its grantmaking activities. The Department has also established a strengthened Investment Review Board to review and approve information technology purchases agency-wide.

Evidence: Department Investment Review Board materials. ED's Discretionary Grants Improvement Team (DiGIT) recommendations.

Does the program collaborate and coordinate effectively with related programs?

Explanation: Notable examples of successful coordination include ED's consolidation program management of the Graduate Assistance in Areas of National Need (GAANN) and Javits fellowships programs to enhance scholarship program coordination. ED annually coordinates stipend levels with the National Science Foundation (NSF) and has established an Executive Board with members that represent a broad range of disciplines in graduate education in arts, humanities, and social sciences.

Evidence: GAANN and Javits fellowships program managament coordination had resulted in the development of parallel policies, administrative procedures and performance measures.

Does the program use strong financial management practices?

Explanation: No internal control weaknesses have been reported by auditors. Plus, the Department has a system for identifying excessive draw downs, and can put individual grantees on probation which requires Departmental approval of all grantee draw downs.

Evidence: Program financial management records.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: A review of management practices has led to management improvements that have significantly improved the administration of the program. The steps include the development of performance measures and tightening of needs certification requirements to ensure most efficient distribution of program funds.

Evidence: Needs certification for continuing fellows were not being submitted in a timely manner which resulted in the program office not being able to use unobligated NCC funds to offer new fellowships to alternate applicants. ED established new procedures and a June 30 deadline date for the certifications. The result of these modifications is a more efficient allocation of program resources.

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: Independent peer review panels are used to score and rank all applications.

Evidence: Program funds are used to pay for the peer review process. 100 percent of grants are subject to peer review.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: ED is in close communication with grant recipients and provides technical assistance to fellows and grantee institutions throughout the grant period via e-mail exchanges and telephone conversations. Grantee institutions must consult ED for approval of any change in the status of an individual fellow, such as an interruption of study, to work or to travel abroad.

Evidence: Need analysis certification reports (include academic progress reviews).

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: GPRA data are now reported in several formats, including on the Department's website. However, this publicly available information is not performance related. Education is developing a department-wide approach to improve the way programs provide performance information to the public. In 2004, Education will conduct pilots with selected programs to assess effective and efficient strategies to share meaningful and transparent information.

Evidence: N/A

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: Preliminary data shows that actual program performance is on track to achieving the program's long-term performance goals pertaining to time-to-degree completion rates and graduation rates under the Javits Program. Performance data from annual performance reports reveals that Javits Fellows earn doctorates at a faster rate than the national average and that the graduation rate for Javits fellows is higher than the national average. The Department is currently undertaking a comprehensive evaluation of all of the Department's graduate fellowship programs, the results of which will be used to confirm the validity of the performance report data.

Evidence: ED has exceeded its annual goals for this program and appears to be on-track to meet of exceed targets for its long-term performance measures.

Does the program (including program partners) achieve its annual performance goals?

Explanation: Preliminary data shows that actual program performance is achieving the program's annual performance goals pertaining to time-to-degree completion rates and graduation rates under the Javits Program. Performance data from annual performance reports reveals that Javits Fellows earn doctorates at a faster rate than the national average and that the graduation rate for Javits fellows is higher than the national average. The Department is currently undertaking a comprehensive evaluation of all of the Department's graduate fellowship programs, the results of which will be used to confirm the validity of the performance report data.

Evidence: While an average Javits fellow completes his/her degree in 6.3 years, the most recent available data from the National Research Council's (NRC) annual Survey of Earned Doctorates, reveals that the national median time to degree completion rates for comparable programs were 7.5 years in 2002. Furthermore, the most recent data compiled by the NRC Survey of Earned Doctorates indicates that the graduation rate for doctorates awarded nationally during the 2001-02 academic year in the humantities and social sciences was 29%. Meanwhile, the percentage of Javits Fellows obtaining doctorates is approximately 30%. Although this figure is only slightly higher than the national average, this rate is promising given the fact that the Javits Program contains a need based component.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: ED is in the process of developing an efficiency measure.

Evidence:  

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: Although there are some programs, especially in the private sector, that are comparable, outcome data is not available on these programs to provide the basis for meaningful comparison.

Evidence: The Department may be able to make some comparisons between graduate fellowship programs in future years as performance measures for these programs are implemented.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: The Department is currently undertaking a comprehensive study of all of the graduate fellowship programs in the Office of Postsecondary Education. This study will provide specific data to support the performance measures for these programs. The first impact data will be available in FY 2006.

Evidence: N/A