Detailed Information on the
Graduate Assistance in Areas of National Need Assessment

Completing the study of ED's graduate fellowship programs and utilizing the results to validate program performance measures and improve program performance.

Establishing a strategy to track program performance after expiration of grant period.

Collecting annual performance information for all performance measures and using this information to guide program management.

Finalizing an efficiency measure and establishing targets for this metric.

Use grantee level performance measure data (especially data from the efficiency measure) to target technical assistance.

Make performance data (both program-wide and disaggregated to the grantee level) accessible to the general public.

Reviewing and revising program mangers' performance agreements to ensure that managers are held accountable for achieving key program results.

Measure: Graduation rate for program fellows.

Explanation:The percentage of GAANN fellows completing a doctorate in the designated areas of national need

Measure: Time to program completion

Explanation:The medium time from entering graduate school until degree completion.

Measure: Cost per successful GAANN fellow

Explanation:Success is defined as graduation or passing prelim exams during that period.

Measure: Time to program completion

Explanation:The medium time from entering graduate school until degree completion.

Measure: Time to program completion

Explanation:The medium time from entering graduate school until degree completion.

Measure: Time to program completion

Explanation:The medium time from entering graduate school until degree completion.

Measure: Time to program completion

Explanation:The medium time from entering graduate school until degree completion.

Is the program purpose clear?

Explanation: The Graduate Assistance in Areas of National Need (GAANN) program provides fellowships, through 3-year grants to academic programs and departments of postsecondary institutions, to graduate students of superior ability and a high level of financial need studying in areas of national need. These academic areas currently include: biology, chemistry, computer and information sciences, nursing, engineering, geological and related sciences, mathematics, and physics.

Evidence: Title VII, Part A, Subpart 2 of the Higher Education Act, which grants the Secretary of the Department of Education (ED) authority to make grants to Institutions of Higher Education (IHEs) to provide fellowships for graduate students in areas of national need as designated by the Secretary.

Does the program address a specific and existing problem, interest, or need?

Explanation: Researchers and policymakers agree that there is a shortage of highly qualified individuals in certain critical academic areas and that this shortage has a detrimental impact in a variety of critical professions.

Evidence: According to the National Science Board's Science and Engineering Indicators of 2002, the National Science and Technology Council has expressed concerns about the ability of the United States to meet "technical workforce needs" and reinforce America's preeminent international position in these sectors.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: This program is one of a range of programs in the Federal government designed to address the shortage of highly qualified individuals in certain critical academic areas. Within this range of programs, GAANN is the only one that focuses on providing resources to outstanding individuals with a high level of financial need pursuing a terminal graduate degree. Beyond the Federal government, a number of private organizations also provide fellowships for graduate studies in these disciplines, but these private efforts are either limited in their geographical scope or do not have a financial need component.

Evidence: While there are many programs within the Federal government--such as the National Science Foundation's Graduate Research Fellowship Program or the National Aeronautics and Space Administration's Graduate Student Researchers Fellowship Program--that provide fellowships for graduate studies in areas of national need, none of the other programs limit fellowship applicants to individuals with high financial need.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: Implementation of the program has not revealed any major flaws in the actual program model that would limit the program's effectiveness or efficiency.

Evidence:

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: The program is focused on recruiting talented students with high financial need into graduate studies in areas of national need. By stipulating that recipients have a high level of financial need, and requiring that fellowship applicants demonstrate that need by completing the FAFSA process, the statute ensures that the program effectively targets exemplary students who would not otherwise be able to complete graduate studies.

Evidence: Title VII, Subpart 2, Sec. 713 (b)(5)(A) of the Higher Education Act. In order to demonstrate high financial need, every fellowship recipient must complete the FAFSA process and the awarding IHE must certify such individual need in its reporting to ED.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The long-term performance measures reflect the statutory intent for the program. They are: 1) percentage of GAANN fellows that complete a terminal degree, 2) average time to degree completion for GAANN fellows, and 3) percentage of GAANN fellows from traditionally underrepresented backgrounds.

Evidence: These measures allow ED to monitor the program's effectiveness in providing fellowships to students who have financial need and the potential to make outstanding contributions in areas of national need. Research demonstrates that the longer students take to complete graduate studies the higher their attrition rate.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The program has developed ambitious targets and timeframes for its long-term measures. Targets for these measures are designed to exceed the national average for students in GAANN-eligible fields of study.

Evidence: GAANN fellows must demonstrate high financial need, putting them in a group that traditionally takes longer to graduate and has a significantly higher attrition rate than the student population as a whole. As such, achieving and maintaining this level of performance would demonstrate that the program is effectively achieving its long-term goals.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The annual performance measures reflect the statutory intent for the program. They are: 1) percentage of GAANN fellows that complete a terminal degree, 2) average time to degree completion for GAANN fellows, and 3) percentage of GAANN fellows from traditionally underrepresented backgrounds. In addition, ED has developed an efficiency measure for this program that tracks the cost per successful outcome, where success is defined as terminal graduate program completer.

Evidence: These measures allow ED to monitor the program's effectiveness in providing fellowships to the appropriate students with the potential to make outstanding contributions to the field. The annual measures also track progress toward achieving the program's long-term goals. Research demonstrates that the longer students take to complete graduate studies the higher their attrition rate.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The program has developed ambitious targets and timeframes for its annual measures. Targets for these measures are designed to exceed the national average for students in GAANN-eligible fields of study.

Evidence: GAANN fellows must demonstrate high financial need, putting them in a group that traditionally takes longer to graduate and has a significantly higher attrition rate than the student population as a whole. As such, achieving and maintaining this level of performance would demonstrate that the program is effectively achieving its annual goals.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: GAANN grantees all work toward the annual and long-term goals of the program. Annual performance reports (APRs) are required of all grantees and their performance is measured on the basis of how well they meet the program goals.

Evidence: Program documentation, including application packets and closing date notices, clearly articulate the program goals and performance measures and indicate that all grantees are expected to submit data in their APRs that documents their progress in addressing these performance measures. The program performance plan is posted to the program website: http://www.ed.gov/programs/gaann/performance.html

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: The Department is currently implementing a comprehensive study of all of the graduate fellowship programs in the Office of Postsecondary Education. This study will provide specific data to support the performance measures for these programs.

Evidence: The Graduate Fellowships Outcomes Study will look at graduation rates of fellowship recipients. The study is in the data collection phase and the first outcome data will be available in FY 2007.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: ED has not satisfied the first part of the question because program performance changes are not identified with changes in funding levels. However, ED has satisfied the second part of this question in that ED's budget submissions show the full cost of the program (including S&E). ED's 05 integrated budget and performance plan includes the program's annual and long-term goals.

Evidence: N/A

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: As part of a comprehensive strategic review, ED has revised the performance measures for the GAANN program and developed the Graduate Fellowships Outcomes Study to provide performance data for these measures. The program has also initiated a process to revise program materials, such as application packets and annual performance reports, to reflect its new long-term and annual performance measures.

Evidence: N/A

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: GAANN grantees are required to submit Annual Performance Reports (APRs) and a Final Performance Report (FPR). APR data have been used to monitor the recruitment efforts and programmatic expenditures of grantee institutions to determine if continuation funding is warranted and if grantees are complying with program regulations. The GAANN program has also created and implemented a series of evaluation instruments used to track and monitor the performance level of grantee institutions. These instruments analyze performance and implementation data taken from the APR for each grantee to determine the optimum level of technical assistance needed and to determine whether or not an on-site review is warranted. This practice enables ED to work with grantees to improve implementation and program performance. The GAANN program is also working to provide grantees with access to performance data collected from both the APRs and the FPRs (such as the percentage of fellows completing terminal degrees or advancing to candidacy, or time-to-degree completion) at the grantee and program level. Performance data (including the efficiency measures) will be shared with all grantee institutions and will be accessible through the program's website.

Evidence: ED's review of APRs detected high levels of error in certain data elements in the newly established web-based annual performance report. After consultation with stakeholders, ED revised the report, clarifying the key questions, and as a result the data collection has improved. In addition, ED's review of grantee APR data has, in some cases, led to improved data quality. Grantee-level analyses are being conducted during 2006, and results will be available on the program's website in the summer 2006. This process will allow grantees to view their own performance within the context of other similar institutions, which will increase opportunities for peer-to-peer technical assistance and program improvements.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: ED's managers are subject to EDPAS, which links employee performance to relevant Strategic Plan goals and action steps, and is designed to measure the degree to which a manager contributes to improving program performance. GAANN managers are identified and their EDPAS agreements are linked with the performance of the program.

Evidence: The EDPAS standards for GAANN program managers set forth requirements that program managers set forth strategies for implementing GPRA and Strategic Plan initiatives related to GAANN. GAANN program officers are held accountable for measuring and assessing project performance and monitoring the progress of projects in achieving program goals and objectives. GAANN grantees receive continuation funding on the basis of their reported progress in achieving the program goals.

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: At the Federal level, all GAANN funds are obligated according to an annual spending schedule that is established at the beginning of the fiscal year. At the partner level, grantees are obligating funds at a reasonable rate.

Evidence: Annual Spending Plans and program financial records.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: GAANN has developed an efficiency measure, the annual cost per successful outcome, to assess the cost effectiveness of each GAANN project on an annual basis. GAANN plans to develop efficiency targets, work with projects to focus on methods for improving efficiency, and implement procedures for comparing efficiency over time.

Evidence: The annual cost per successful outcome for GAANN was $92,557 in 2002 and $127,514 in 2003.

Does the program collaborate and coordinate effectively with related programs?

Explanation: Notable examples of successful coordination include ED's consolidation of the program management of the GAANN and Javits fellowships programs to enhance scholarship program coordination. ED annually coordinates stipend levels with the National Science Foundation (NSF) and has developed a collaborative process to establish the areas of national need that includes consultation with representatives from NSF, the Department of Energy, the National Endowment of the Arts, the National Endowment of the Humanities, NASA, the Department of Labor and several non-profit organizations such as the Council of Graduate Schools and the Association of American Universities.

Evidence: GAANN and Javits fellowships program management coordination has resulted in the development of parallel policies, administrative procedures, and performance measures.

Does the program use strong financial management practices?

Explanation: No internal control weaknesses have been reported by auditors. Plus, ED has a system for identifying excessive draw downs, and can put individual grantees on probation, which requires ED approval of all grantee draw downs.

Evidence: Program financial management records.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: In an effort to increase the accuracy of data in the APRs, ED routinely conducts conference calls with all newly funded grantees within the first month of the grant award to provide technical assistance on data reporting and grant administration. As a result, grantee data quality and timeliness has improved. In addition, ED developed performance measures for this program.

Evidence: N/A

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: Independent peer review panels are used to score and rank all applications.

Evidence: 100 percent of grants are subject to peer review. Program funds are used to pay for the peer review process.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: The program conducts routine financial monitoring, enabling the program office to determine if a grantee is failing to award fellowships in a timely manner. If a grantee is considered "at risk" ED contacts the Project Director to provide technical assistance designed to ensure that all awards are properly made. Annual Project Directors meetings are also convened to provide technical assistance and to address grantees' concerns.

Evidence: N/A

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: Program performance data are now reported by a variety of means, including on ED's website. In addition, basic award information and grant amounts are also available on the website. The program office collects and compiles data from performance reports into a program profile report. This report is available on the program's website and makes performance data available to the public at both an aggregated and disaggregated level.

Evidence: Because of privacy concerns, the program profile report does not make performance data available at the level of the individual student or institution (due to concerns about student privacy issues). The program profile report disaggregates the data according to individual student characteristics and institutional characteristics.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The program has already exceeded the annual initial targets established for this program and appears to be on-track to meet or exceed targets for its long-term performance measures: 1) time-to-degree completion rates, and 2) graduation rates of program participants. Performance data from 2005 performance reports reveal that GAANN fellows earn doctorates in 5.3 years, which is faster than the national average of 6.8 - 7.0 years and better than the 2005 performance target of 6.45 years. The actual graduation rate for GAANN fellows of 48% in 2005 was also better than the than the 2005 target of 28%, and also better than the more ambitious targets established for subsequent years (note: targets for this measure were initially set based on incomplete data in 2002. Revised data enabled more accurate, ambitious targets to be set). In addition, preliminary data indicate that the GAANN program is having some success at attracting greater numbers of women and minorities into pursuing graduate studies in areas of national need than would otherwise be expected. ED is currently undertaking a comprehensive evaluation of all of the Department's graduate fellowship programs, the results of which will be used to confirm the validity of the performance report data.

Evidence: See Measures section.

Does the program (including program partners) achieve its annual performance goals?

Explanation: The program has already exceeded the annual initial targets established for this program's annual performance measures: 1) time-to-degree completion rates, and 2) graduation rates of program participants. Performance data from 2005 performance reports reveal that GAANN fellows earn doctorates in 5.3 years, which is faster than the national average of 6.8 - 7.0 years and better than the 2005 performance target of 6.45 years. The actual graduation rate for GAANN fellows of 48% in 2005 was also better than the than the 2005 target of 28%, and also better than the more ambitious targets established for subsequent years (note: targets for this measure were initially set based on incomplete data in 2002. Revised data enabled more accurate, ambitious targets to be set). In addition, preliminary data indicate that the GAANN program is having some success at attracting greater numbers of women and minorities into pursuing graduate studies in areas of national need than would otherwise be expected. ED is currently undertaking a comprehensive evaluation of all its graduate fellowship programs, the results of which will be used to confirm the validity of the performance report data.

Evidence: See Measures section.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: GAANN has developed an efficiency measure, the annual cost per successful outcome, to assess the cost effectiveness of each GAANN project on an annual basis. The annual cost per successful outcome for GAANN was $92,557 in 2002 and $127,514 in 2003.

Evidence: GAANN has two years of program-level efficiency measure data. The measure fluctuated considerably (38%) from the first to the second year. The fluctuation is due to wide variation in the number of successful outcomes (terminal degrees obtained, or fellows advancing to candidacy) from one year to the next and are the result of the fact that rates are cohort-based and therefore rely on a relatively small number of candidates. To best determine appropriate efficiency targets and make program changes to boost cost effectiveness, additional years of program data and analyses of project-specific outcomes and efficiency measures are needed. Project-specific outcome and efficiency analyses will be completed during 2006, prior to the conclusion of the 2006 PART process.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: Similar graduate fellowship programs exist, especially in the private sector. However, until comparable outcome data are available for such programs, it is not possible to make meaningful comparisons.

Evidence: N/A

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: ED is currently undertaking a comprehensive study of all graduate fellowship programs in the Office of Postsecondary Education. This study will provide specific data to support the performance measures for these programs. The first outcome data will be available in FY 2007.

Evidence: N/A