Detailed Information on the
Gaining Early Awareness and Readiness for Undergraduate Programs Assessment

Developing and implementing a strategy to use efficiency measures to improve cost effectiveness in achieving the program goals.

Utilizing newly available information on the program's college participation performance measure to assess program progress and target technical support to GEAR UP grantees.

Utilizing the results of the recently completed Early Effects Study to develop strategies to improve program management.

Implementing a plan for responding to Inspector General concerns for monitoring program expenditures.

Collecting baseline data for annual measures and working to ensure that appropriate data collection strategies are in place for the long-term measures.

Developing a meaningful efficiency measure.

Reviewing and revising program managers' performance agreements to ensure that managers are held accountable for achieving key program results.

Measure: Percentage of program participants completing Algebra I by the 9th grade.

Explanation:Targets for 2004-2005 were based on 2003 baseline data for percentage of students enrolled in Algebra I at the end of the 9th grade. Data beginning in 2004 reflect completion levels, so targets for 2006-2007 were revised accordingly.

Measure: Percentage of GEAR UP students who have knowledge of necessary academic preparation for college.

Explanation:

Measure: Percentage of program participants that complete high school.

Explanation:The first complete cohort of GEAR UP students was not scheduled to finish the 12th grade until 2005.

Measure: Percentage of program participants that enroll in college.

Explanation:The first complete cohort of GEAR UP students was not scheduled to finish the 12th grade until 2005.

Measure: Average cost for each successful outcome.

Explanation:A successful outcome is defined as a participant that immediately enrolls in college after high school graduation.

Is the program purpose clear?

Explanation: The program is designed to provide support services and financial aid to low-income middle and high school students so they can attend college.

Evidence: The statutory purpose as stated in Section 404A of the Higher Education Act is to: "provide or maintain a guarantee" of financial aid and a wide range of additional services as preparation "to attend an institution of higher education."

Does the program address a specific and existing problem, interest, or need?

Explanation: Data indicate that low-income students do not attend college at the same rates as students who are less disadvantaged, and they lack adequate middle and high school preparation.

Evidence: Data from the National Education Longitudinal Study of 1988 indicate that the overall college enrollment rate for low-income students is 64% compared to 79% and 93% for middle- and high-income students. The 4-year college enrollment rate for low-income students is 33% compared to 47% and 77% for middle- and high-income students. A wide-range of other data are available in NCES publications.

Is the program designed so that it is not redundant or duplicative of any Federal, state, local or private effort?

Explanation: GEAR UP targets entire grades of students beginning in middle school, leverages significant community resources and commitment through partnerships with State and local entities, and provides a comprehensive set of services, aid, and reform mechanisms.

Evidence: Other Federal programs, including Upward Bound and Talent Search, do not share these characteristics. Local and private efforts that are similar to GEAR UP, including those on which GEAR UP was modeled, are geographically limited in scope. Also, the GEAR UP statute requires that funds supplement and not supplant existing programs.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: Similar approaches upon which this program is modeled are effective. There is no evidence that other approaches are more effective. The ongoing program evaluation should further inform as to the effectiveness of GEAR UP's design.

Evidence: I Have a Dream and Project GRAD are both proven models. According to one study, Project GRAD in Houston increased high school graduation rates by 64% as compared to a district-level decrease of 7% in graduation rates. Another study showed that the I Have a Dream program raised high school graduation rates to 90% among participants in a New York City school as compared to 25% for other students in that school.

Is the program effectively targeted, so program resources reach intended beneficiaries and/or otherwise address the program's purpose directly?

Explanation: GEAR UP funds are used to support entire grades of students at high-poverty schools. Additionally, the evaluation of GEAR UP indicates that services tend to be weighted toward the neediest students in those grades.

Evidence: The statute requires that more than 50% of the students in participating schools be eligible for free or reduced-price lunch. Within this population, the national evaluation of GEAR UP (Westat, 2003) indicates that tutoring and other services are weighted toward the students that are failing and otherwise have the highest levels of need.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: GEAR UP has long-term goals of increasing high school graduation and college participation rates. These goals are derived directly from the purpose of the program. The Department is working with OMB on developing an appropriate efficiency measure for this program.

Evidence: The primary GPRA objective is to increase high school graduation and postsecondary participation rates. Until participating students reach the 12th grade and beyond, data on other measures are being collected.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: Although baseline data for the long-term goals will not be available for several years, targets have been developed on the basis of comparable NCES data.

Evidence: Targets for high school completion (73%) and college enrollment (65%) have been set to reduce the gap between low-income and middle-income students.

Does the program have a limited number of specific annual performance measures that demonstrate progress toward achieving the program's long-term measures?

Explanation: The program has a number of short-term goals that measure progress toward high school graduation and college participation. Once participating students reach the 12th grade, annual measures also will be developed for these long-term goals. The Department is working with OMB on developing an appropriate efficiency measure for this program.

Evidence: The GPRA indicators measure Prealgebra and Algebra I course completion (a key indicator of future college enrollment), attendance and promotion rates, and knowledge of necessary preparation and available financial aid.

Does the program have baselines and ambitious targets and timeframes for its annual measures?

Explanation: Annual targets have been developed through 2007 as a benchmark for short- and long-term success.

Evidence: Selected targets for long-term performance are: 70% Algebra I completion, 98 percent grade promotion, and 75 percent knowledge of necessary preparation for college.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, etc.) commit to and work toward the annual and/or long-term goals of the program?

Explanation: The program design (projects must serve cohorts into and through high school) is structured to achieve the long-term program goals. Although targets for performance have been established only recently, all grantees are required to work toward the program goals. Funding for new and continuing awards is based on a project's plan for and success in meeting these goals.

Evidence: The statute requires all partnerships and States serving cohorts to "ensure that the services are provided through the 12th grade." New grant proposals receive up to 15 points for how well projects propose to use performance measures to assess progress toward achieving their intended outcomes. The statute also requires grantees to "biennially evaluate the activities...in accordance with the standards" prescribed by ED.

Are independent and quality evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: A longitudinal study is currently underway to measure program impacts through the 8th grade. Follow-up studies are expected to measure impacts through the 12th grade and into college.

Evidence: The first impact report is due to be released toward the end of 2003.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: ED has not satisfied the first part of the question because program performance changes are not identified with changes in funding levels. The program, at this time, does not have sufficiently valid and reliable performance information to assess (whether directly or indirectly) the impact of the Federal investment. However, ED has satisfied the second part of this question in that ED's budget submissions show the full cost of the program (including S&E). ED's 05 integrated budget and performance plan includes the program's annual and long-term goals.

Evidence:  

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: GEAR UP is using performance report data to develop targets for annual goals.

Evidence: Targets have been developed for annual goals through 2007. Specific action steps for improved performance and revised performance targets will likely emerge when evaluation data become available.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: GEAR UP utilizes annual performance reports to oversee grantee performance, and staff conduct quarterly conference calls with projects to collect information. While this performance data is limited due to the program's recent inception, ED uses it to shape its technical assistance workshops and to identify grantees that require additional program management assistance.

Evidence: GEAR UP is currently redesigning the performance reports to reduce the reporting burden and increase the quality of data collected. This will increase ED's ability to make informed decisions regarding program management and performance. GEAR UP relies on performance data to increase the focus of national conferences on parental involvement and other issues that have been shown to be most difficult to implement.

Are Federal managers and program partners (grantees, subgrantees, contractors, cost-sharing partners, etc.) held accountable for cost, schedule and performance results?

Explanation: ED's managers are subject to EDPAS which links employee performance to relevant Strategic Plan goals and action steps, and is designed to measure the degree to which a manager contributes to improving program performance. To receive a "Yes", ED needs to: (1) identify for OMB the federal managers for this program; and (2) demonstrate the relationship between these managers' performance standards and the program's long-term and annual measures; and (3) demonstrate the relationship between program partner's performance standards and the program's long-term and annual measures.

Evidence:  

Are all funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: Funds are obligated in a timely matter but an IG report has indicated that monitoring of expenditures needs improvement. New office-wide monitoring plans have been implemented.

Evidence: GEAR UP developed a monitoring plan and actively monitors the draw down of Federal funds. In one case, a grantee was required to return funds. On another occasion, GEAR UP prohibited a grantee from using the interest accrued on scholarship funds for inappropriate purposes.

Does the program have procedures (e.g., competitive sourcing/cost comparisons, IT improvements, approporaite incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: This program has not yet instituted procedures to measure and improve cost efficiency in program execution. However, as part of the President's Management Agenda, the Department is implementing "One-ED" -- an agency-wide initiative to re-evaluate the efficiency of every significant business function, including the development of unit measures and the consideration of competitive sourcing and IT improvements. A "yes" answer is likely once the One-ED process is applied to this program's relevant business functions.

Evidence:  

Does the program collaborate and coordinate effectively with related programs?

Explanation: GEAR UP projects are often linked with Talent Search, Upward Bound, and Student Support Services projects, creating a pipeline of services through college. Projects must coordinate with student aid offices and FSA.

Evidence: The statute requires each grantee to ensure that activities are coordinated with other GEAR UP projects in that school district or State, and with "related services under other Federal or non-Federal programs." Projects providing scholarships must coordinate with student aid offices and FSA on award packaging. Additionally, there are several GEAR UP grantees that also have UB, Talent Search, EOC, SSS, and McNair grants.

Does the program use strong financial management practices?

Explanation: GEAR UP has not been revealed to have internal control weaknesses and follows Departmental guidelines for financial management.

Evidence:  

Has the program taken meaningful steps to address its management deficiencies?

Explanation: GEAR UP has developed a plan for responding to IG concerns such as inadequate grantee budget review and grant monitoring. The program office has trained staff to perform site visits and other monitoring activities.

Evidence: Each audit recommendation relating to GEAR UP's 2000 competition was addressed and implemented prior to the 2001 competition including: eligibility checklists, procedures to review budget materials prior to award of funds, and procedures to minimize the risk of error in the application review process. Procedures also have been developed to ensure that ED's grant oversight office will be notified of any future changes to the status of warrant holders.

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: Independent peer review panels are used to score and rank all applications. The number and distribution of new GEAR UP grantees indicate that new/first-time grantees are able to compete fairly.

Evidence: 100% of grants are subject to review. Funded partnerships include many non-traditional grantees such as faith-based and other community organizations, and many serve a high percentage of Hispanic students.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: New procedures have been developed for improving the monitoring of expenditures based on IG concerns. GEAR UP engages in a number of systematic monitoring activities that look at both compliance with statutory and regulatory requirements and performance of grantees.

Evidence: GEAR UP's monitoring efforts focus on the review of annual performance reports, telephone contacts with each grantee at least quarterly, and selected site visits. Examples of corrective action include: 1) reducing funding for a grantee that was not meeting its matching requirements, 2) requiring a grantee that had not hired sufficient staff to make appropriate staffing changes, and 3) requiring two grantees to discontinue the provision of services at ineligible institutions.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: GEAR UP collects and compiles data from performance reports and makes it available to the public, upon request, via their website. Additionally, aggregated performance data and program indicators are available online. Efforts are underway to increase the transparency of the data for all users, including online availability of program performance indicators. APR data should be aggregated in future years to maintain a YES answer for this question.

Evidence: GEAR UP's website (www.ed.gov/gearup) indicates that performance report data at the grantee level are available in an SPSS format. Data will be updated annually. Steps also are being taken to provide the data in an aggregated and easily understood format directly on the website. The first report from GEAR UP's national evaluation is available at www.ed.gov/offices/OUS/PPSS/gearup.html.

Has the program demonstrated adequate progress in achieving its long-term outcome performance goals?

Explanation: Since GEAR UP is relatively new, there are not sufficient data available to make this determination.

Evidence: The first cohort of students have yet to reach the 12th grade. Once baseline data are available, future performance reports and evaluations will measure progress toward achieving long-term goals.

Does the program (including program partners) achieve its annual performance goals?

Explanation: Since GEAR UP is relatively new and targets have recently been established, it is too early to assess progress toward achieving short-term goals. However, initial results for some of the less critical program goals have been encouraging, with the program exceeding its initial targets for student knowledge of academic preparation, and for grade promotion.

Evidence: Annual performance data will begin to inform about achieving annual goals next year.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program performance goals each year?

Explanation: The Department is working with OMB on developing an appropriate efficiency measure for this program.

Evidence:  

Does the performance of this program compare favorably to other programs, including government, private, etc., that have similar purpose and goals?

Explanation: There are insufficient data regarding GEAR UP and related programs to make this determination.

Evidence: The first impact report for GEAR UP is due to be released toward the end of 2003. Next year, comparisons may be possible with Talent Search and Upward Bound, which also provide services to increase the postsecondary participation rates of low-income students.

Do independent and quality evaluations of this program indicate that the program is effective and achieving results?

Explanation: GEAR UP is relatively new and the national evaluation has produced only a descriptive report thus far. However, other recent studies, including 3 released since February 2003, indicate that programs receiving GEAR UP funding and programs with the same core elements as GEAR UP are effective in improving college enrollment for at-risk students.

Evidence: The Institute for Higher Education Policy's (IHEP) study 'Investing Early: Intervention Programs in Selected U.S. States' examines the 17 leading State early intervention programs, 8 of which have GEAR UP grants. The study concludes that effective programs tend to be comprehensive; include financial assistance; provide access to challenging coursework, supportive enrichment activities, and peer groups; and maintain such services over a long time period. The Center for Higher Education Policy Analysis' (CHEPA) study 'Preparing for College: Building Expectations, Changing Realities' finds that keys to college enrollment include: strong academic preparation beginning no later than middle school, parental involvement, opportunities to enroll in rigorous coursework, tutoring, and coordination between K-12 and college educators. GEAR UP projects address these obstacles with many of the strategies articulated in the CHEPA and IHEP reports.