Detailed Information on the
Substance Abuse Prevention Programs of Regional and National Significance Assessment

Assure all grantees providing direct services collect pre-post data

Developing an efficiency measure and baseline data.

Posting disaggregated program performance data online.

increase grantee reporting of population estimates for environmental and community wide programs and strategies.

Developing a new efficiency measure

Developing baselines and targets for new annual measures

Measure: Strategic Prevention Framework State Incentive Grants (SPF SIG): Percent of SPF SIG states showing a decrease in state level estimates of percent of survey respondents who report 30 day use of alcohol a) age 12-20; b) age 21 and up (New measure, added August 2007)

Explanation:

Measure: Strategic Prevention Framework: Percent of SPF-SIG states showing a decrease in state level estimates of survey respondents who report 30-day use of other illicit drugs a) age 12-17; b) age 18 and up (New measure, added August 2007)

Explanation:

Measure: Strategic Prevention Framework: Percent of SPF-SIG states showing an increase in state level estimates of survey respondents who rate the risk of substance abuse as moderate or great a) age 12-17; b) age 18 and up. (New measure, added August 2007)

Explanation:Perception of harm anticipated from substance use is closely correlated with decrease in use.

Measure: Strategic Prevention Framework: Percent of SPF SIG states showing an increase in state level estimates of survey respondents (age 12-17) who somewhat disapprove or strongly disapprove of substance use. (New measure, added August 2007)

Explanation:

Measure: Total number of evidence-based policies, practices, and strategies implemented

Explanation:

Measure: Substance Abuse Prevention/HIV Prevention: Percent of participants who used alcohol at pre-test who report a decrease in use of alcohol at post- test (user decrease) a) age 12-20; b)age 21 and up (New measure, added August 2007)

Explanation:Measure uses participant level data.

Measure: Substance Abuse Prevention/HIV Prevention: Percent of participants who report no alcohol use at pre-test who remain non-users at post-test (non-user stability) a) age 12-20; b) age 21 and up (New measure, August 2007)

Explanation:Measure uses participant level data.

Measure: Substance Abuse Prevention/HIV Prevention: Percent of participants who used illicit drugs at pre-test who report a decrease in use at post-test (user decrease) a) age 12-17; b) age 18 and up (New measure, August 2007)

Explanation:Measure uses participant level data.

Measure: Substance Abuse Prevention/HIV Prevention: Percent of participants who report no illicit drug use at pre-test who remain non-users at post-test (non-user stability) a) age 12-17; b) age 18 and up (New measure, August 2007)

Explanation:Measure uses participant level data.

Measure: Substance Abuse Prevention/HIV Prevention:Percent of program participants (age 12-17) who somewhat disapprove or strongly disapprove of substance use. (New measure, August 2007)

Explanation:Measure uses participant level data.

Measure: Substance Abuse Prevention/HIV Prevention:Total number of evidence-based policies, practices, and strategies implemented by HIV program grantees. (New measure, added August 2007)

Explanation:Measure uses participant level data.

Measure: Cost per participant improved on one or more measures between pre-test and post-test.

Explanation:This measure is defined as the total cost of the HIV program divided by the number of program participants who improved. CSAP defines "improved" as improved on one or more of the measures between pre and post." Pre and post means the time from program beginning participation to exit. The length varies depending on the program but must equal or exceed 30 days. There is limited long term follow-up in these programs. The HIV measure reflects the ability of the program to measure pre-post improvements in identified participants. This measure will be reassessed Oct, 2009.

Is the program purpose clear?

Explanation: The purpose of the Center for Substance Abuse Prevention Programs of Regional and National Significance (CSAP PRNS) is to decrease substance use and abuse by supporting States and communities to provide effective prevention programs. The two program categories include Targeted Capacity Expansion (TCE) and Best Practices (BP). TCE activities encompass grants for service delivery and include 1) those that address emerging substance abuse trends, Substance Abuse/HIV, Methamphetamine, Ecstasy, Fetal Alcohol Spectrum Disorder, and workplace substance abuse, and 2) the State Incentive Grant (SIG) Program to develop infrastructure in States and communities to provide effective prevention programs in areas of highest need. BP activities include 1) dissemination of information and technical assistance to states and communities on selecting and implementing proven-effective prevention programs and strategies, and 2) support for evaluation of promising practices that have not yet been rigorously evaluated, in order to increase the number of proven-effective strategies from which communities can select. Coordination across these components is strong and is supported through the Strategic Prevention Framework (SPF), a comprehensive community planning and implementation model that will guide all CSAP PRNS programs. Additionally, most CSAP PRNS TCE grants require or encourage grantees to use evidence-based programs identified through BP activities. The program purpose is consistent with 1) authorizing legislation, 2) SAMHSA's strategic plan, and 3) the SPF.

Evidence: The program is authorized in section 516 of the Public Health Services Act (42 USC 290bb-22). The program's purpose is also clearly articulated consistent with the authorizing statute in the CSAP Strategic Plan, SAMHSA Strategic Plan, and CSAP Mission Statement. Additionally, the description of CSAP PRNS activities in FY 2005 Congressional Justification align with the program's purpose.

Does the program address a specific and existing problem, interest or need?

Explanation: The need for substance abuse prevention is clear and current. An estimated 19.5 million Americans--8.3 percent of the population aged 12 or older--are current illicit drug users. The number of individuals who have used illicit drugs in the past 30 days remains well above 1992 lows, according to the National Institute on Drug Abuse's (NIDA's) Monitoring the Future (MTF) survey of youth. The National Institute on Alcohol Abuse and Alcoholism (NIAAA) has found that people who begin drinking before age 15 are four times more likely to develop alcohol dependence than those who have their first drink at age 21 or older. The projected burden attributed to substance abuse in 2000 due to healthcare costs, drug related crime and loss of productivity was over $160 billion.

Evidence: National-level trend data are available in SAMHSA's Results from the 2002 National Survey on Drug Use and Health: National Findings and NIDA's Monitoring the Future National Results on Adolescent Drug Use: Overview of Key Findings. The information on the NIAAA-funded study on the onset of alcoholism can be found at www.niaaa.nih.gov/press/1998/aging.htm

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: The program makes a unique contribution. While other publicly- and privately-funded programs address aspects of substance abuse and prevention, neither focus on regional, emerging problems. Targeted Capacity Expansion grants to address emerging substance abuse trends are designed specifically to fill gaps. The SIG program is unique in its purpose to foster coordination at the State level by all relevant agencies and stakeholders and the planning for all prevention resources in the State by requiring a state-level SPF advisory council, mobilization of key state and community stakeholders, and development of a comprehensive plan for all prevention resources in the State. CSAP coordinates at the Federal level with a broad range of programs with the key goal of avoiding duplication and fostering joint efforts. The agency also supports the Substance Abuse Prevention and Treatment (SAPT) Block Grant, which shares goals with the CSAP PRNS program but is designed for a different purpose. To further reduce the potential for overlap and duplication at the federal level, CSAP's Division of State and Community Systems Development manages all state-level grants, including the Block Grant and State Incentive Grants. Each project officer oversees the Block Grant and the SIG, if applicable, for his/her state.

Evidence: The Safe and Drug Free Schools program in the Department of Education provides block grants to states which provide subgrants to schools for violence and substance use prevention. The Drug-Free Community Support Program at ONDCP provides grants primarily to small community coalitions to promote protective factors and reduce risk factors that prevent substance use and abuse. SAMHSA's SAPT Block Grant is designed to provide resources to every state for substance abuse prevention activities. While other substance abuse prevention programs funded by states, localities, and private sources exist, Prevention PRNS avoids duplication by taking the needs and resources already available to communities and grant applicants into account when selecting grantees and developing funding streams.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The design of CSAP PRNS is effective. The organization of activities into Targeted Capacity Expansion and Best Practices is a structure used throughout the agency, with an effective targeting of funds through 1) competitive grants to States and localities and 2) supporting the development and implementation of effective services through the Centers for the Application of Prevention Technologies (CAPTs) and the National Registry of Effective Programs (NREP). The program accomplishes its goals primarily through competitive grants and contracts, which are awarded based on a peer review process that uses evaluation criteria designed to identify those activities with the best chance to succeed in the areas of greatest need. Strategic Prevention Framework State Incentive Grants were specifically designed to address flaws in program planning and accountability that were identified in previous program efforts.

Evidence: There is no evidence that another approach or mechanism, such as regulatory action, loans, etc., would be more efficient or effective to achieve the intended purpose of supporting communities to reduce illicit substance use. The use of competitive grants enables the program to target areas of need and distinguishes it in design and purpose from the SAPT Block Grant.

Is the program effectively targeted, so that resources will reach intended beneficiaries and/or otherwise address the program's purpose directly?

Explanation: The program design effectively reaches intended beneficiaries by 1) including demonstration by grant applicants that the community they will serve is in relatively greater need of prevention programming, 2) requiring grantees to target their programming to areas of greater need within their jurisdictions, and 3) detecting emerging trends and developing grant programs to address said trends. The SPF and evaluation criteria in the Standard Grant Announcements used in the competitive grants process strengthen this targeting by requiring grantees to assess needs, expend resources in areas of highest need, use proven effective prevention models, and provide data on outcomes. Emerging trends are identified through national survey data, program data, and regular communication with grantees and the prevention field. CSAP produces a regular report of trends and directions using these and other data sources.

Evidence: Evidence includes the grantee selection criteria in the Standard Grant Announcements, the Strategic Prevention Framework, the SPF SIG Request for Applications, and the substance abuse prevalence trends documented at oas.samhsa.gov.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The program is developing two primary long-term outcome measures, which are already being used at the national level in the ONDCP National Drug Control Strategy and in Healthy People 2010 and directly measure the program's purpose to reduce and prevent substance use. Baselines and targets are currently set at the national level; however, the program will require states to collect and report data from subgrantees in order to measure the particular impact of CSAP PRNS funding on performance goals. These estimates will enable SAMHSA to identify the impact of CSAP PRNS targeted capacity expansion and SIG grants on substance use trends at the state and, eventually, the national level.

Evidence: National-level trend data are available in SAMHSA's Results from the 2002 National Survey on Drug Use and Health: National Findings and NIDA's Monitoring the Future National Results on Adolescent Drug Use: Overview of Key Findings. Further information on current and developmental performance measures is available in the 2005 and 2006 (forthcoming) GPRA Plans, both of which are included in SAMHSA's budget justification to Congress, and in ONDCP's 2005 National Drug Control Strategy.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The program's targets are both ambitious and realistic, given that the rate of illicit substance use and youth alcohol use have remained relatively stable over the last decade. Additionally, the program's long-term target for reducing youth illicit drug use is consistent with the ONDCP goal to reduce substance use by youth by 10% in two years and 25% in five years.

Evidence: Trends documented in SAMHSA's National Survey of Drug Use and Health (NSDUH) survey and NIDA's Monitoring the Future survey illustrate that while the President's goal to reduce youth substance use by 10% in two years has been exceeded, substance use and abuse by adults remains high and stable. CSAP PRNS's targets reflect evaluation findings that suggest prevention reduces substance use and decreases risk factors among program participants, while recognizing that several other factors affect the prevalence of substance use in the population at large.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The program has annual outcome and output measures that correspond with accomplishing long-term goals. Perception of both harm and adult/peer disapproval of illicit substance use are correlated closely with decreased use. Evidence-based programs, on which CSAP PRNS requires that a percentage of its funding be used, have been proven to increase perceptions of harm and disapproval of drug use. The program is also currently developing an efficiency measure.

Evidence: National-level trend data are available in SAMHSA's Results from the 2002 National Survey on Drug Use and Health: National Findings and NIDA's Monitoring the Future National Results on Adolescent Drug Use: Overview of Key Findings. Further information on current and developmental performance measures is available in the 2005 and 2006 (forthcoming) GPRA Plans, both of which are included in SAMHSA's budget justification to Congress, and in ONDCP's 2005 National Drug Control Strategy.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The program has baseline data and ambitious targets for current annual measures; baselines and targets will be developed for the efficiency measure by December 2005. Targets are reviewed annually and adjusted as appropriate according to the data received. Targets for perception of harm and disapproval of drug use were developed by examining national trends and the rate of imprevement expected by other measures and goals. Because disapproval rates for most illicit substance have declined steadily since 1990, targets that may seem relatively modest actually represent ambitious accomplishments and a reversal of national trends.

Evidence: National-level trend data are available in SAMHSA's Results from the 2002 National Survey on Drug Use and Health: National Findings and NIDA's Monitoring the Future National Results on Adolescent Drug Use: Overview of Key Findings. Further information on current and developmental performance measures is available in the 2005 and 2006 (forthcoming) GPRA Plans, both of which are included in SAMHSA's budget justification to Congress, and in ONDCP's 2005 National Drug Control Strategy.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: CSAP has been working with the States and the National Prevention Network since 1990 on issues involving prevention outcomes, measurement, and data. The result of those meetings include the seven domains on which SAMHSA will require data from all grantees. CSAP core measures have been adopted by other prevention programs, such as ONDCP, Safe and Drug-Free Schools, and the Drug-Free Communities Program. All grantees are required to collect data and report on program goals as a condition of the grant award.

Evidence: The seven domains are included in all of SAMHSA's funding announcements. Grantee contracts and funding announcements articulate the program's requirement that grantees report performance outcome data.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: All PRNS programs are evaluated to ensure their competitiveness, responsiveness, and performance. External independent evaluations are conducted for each grant funding stream, contract, and activity to test the effectiveness of programs individually and, where applicable, across sites. The program maintains a schedule for current and forthcoming evaluations. Five grant funding streams have undergone recent external evaluations: High-Risk Youth Program (2000): This evaluation, conducted by EMT Associates and ORC Macro, both of which are independent contractors, included a cross-site analysis, comparing a treatment group (which participated in the program) to a control group (which did not) and following up to measure outcomes 6- and 18-months after completion of the program. The evaluation found that 2/3 of funded grantees reduced substance use and/or risk factors associated with substance use. The evaluation also found that programs conducted after school and programs that are more "intense" (longer sessions or program intervention length) are more likely to show positive results. Community-Initiated Prevention Interventions (CIPI) (2001): This evaluation, conducted by EMT Associates and Caliber Associates, Inc., included a cross-site analysis, comparing a treatment to a control group, neither of which were randomly selected. The evaluation suffered from several grantees not reporting outcome data, and found small but not statistically significant effects on reducing substance use among participating youth. In part as a result of these evaluation findings, Prevention PRNS now requires grantees to report outcome data. Community-Oriented Substance Abuse Prevention (2002): This evaluation, conducted by EMT Associates and ORC Macro, included a cross-site analysis, comparing a treatment group to a control group, neither of which was randomly selected. The evaluation included data collected from participants during the intervention and 6 months following exit from the program. The evaluation found increased perception of harm associated with substance use for particpants; the effect disappeared at 6 months for the youngest participants but not for those age 9 and up. Again, this evaluation suffered from missing data and one recommendation included stricter data requirements for grantees. Project Youth Connect (PYC) (2002): This evaluation, conducted by EMT Associates, included a cross-site analysis, comparing a treatment group to a control group, neither of which was randomly selected. The evaluation included data collected from participants during the intervention and two years following exit from the program. The evaluation found that the program was not effective in achieving outcomes, noting the evaluation suffered from missing data. One recommendation included holding grantees accountable for outcomes and data collection, which Prevention PRNS has addressed by requiring grantees to report performance data. Family Strengthening Program (2003): This evaluation, conducted by McFarland and Associates, Inc., collected survey data before and after youth participated in the program. The evaluation found increased family functioning among program participants but acknowledged the need for a more rigorous study that includes a control and treatment group.

Evidence: Evaluations of the programs described in the explanation section, as well as the evaluation schedule provided by program staff, comprise the evidence for this question. In particular, performance data reporting requirements in current CSAP PRNS grant programs reflect the recommendations and lessons learned from these evaluations.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The 2005 Budget does not provide a presentation that clearly ties the impact of funding decisions to expected performance or explains why the requested performance and resource mix is appropriate. The FY05 Budget does provide full cost accounting for the CSAP PRNS program. The SAMHSA and CSAP Strategic Plans and the Strategic Prevention Framework guide the development of the budget. The 2006 Budget will present, for the first time, an integrated performance and budget document.

Evidence: None.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The CSAP PRNS program had previously been comprised primarily of short-term, specific grant programs that did not systematically address prevention needs in states or communities. Consequently, communities rarely were able to sustain CSAP-funded prevention programming after federal funding ended. Additionally, the use of performance and epidemiological data for decision-making regarding prevention programming was limited at the federal, state, and community levels, making it difficult to determine where and in what manner resources should be used to effectively prevent substance use and abuse. CSAP responded to these issues by developing the Strategic Prevention Framework (SPF). The program sought to identify ways in which it could most effectively use its resources to develop prevention infrastructure in states and communities, which led to the development of NREP to disseminate information on evidence-based prevention programs, and State Incentive Grants to expand the use of evidence-based strategies and comprehensive planning among community stakeholders. Additionally, the program developed GPRA measures, required grantees to report on these new measures, and created the Data Coordinating Center to collect and analyze grantee data.

Evidence: Conversations with program staff as well as documentation in funding announcements and grantee contracts and agreements support the explanation for this question.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The program requires grantees to collect and report on performance information on a semi-annual basis (at a minimum) and uses the information to manage the program and improve performance. For example, if a quarterly progress report shows that a grantee has not recruited the expected number of participants, the project officer would contact the grantee to develop corrective actions and determine whether technical assistance is needed. Performance data across CSAP PRNS are reported annually in the Data Coordinating Centers Accountability Report.

Evidence: Sample program performance reports and correspondence between project officers and grantees show examples of the use of performance data as described in the explanation.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: All performance plans for CSAP staff track to management/program objectives in the Administrator's performance contract. CSAP awards only Performance Based Contracts that include schedules, deliverables, and performance standards. After the first year of a contract, all subsequent years are option years, facilitating the ability of CSAP staff to cancel a contract for poor performance. Fees awarded to the contract are also tied directly to the performance of the contractor in meeting its deliverables. All grantees agree to provide performance data and provide regular reports that include both cost and performance information.

Evidence: The Administrator's performance contract, sample contract reviews and award fee determinations, and other sample terms and conditions for contracts established accountability for performance. Discussions with agency staff regarding examples of how deficiencies have been addressed and how performance standards in employee contracts are tied to follow-up actions further solidified this evidence.

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: Funds are obligated efficiently and in accordance with planned schedules. The agency will be releasing all funds, including CSAP PRNS, earlier this fiscal year than previous years. Budget execution staff monitor awards to ensure timely obligation of funds. Project staff monitor programs, through review of progress reports and site visits when funds are available, to ensure that funds are spent for the intended purpose.

Evidence: Evidence includes sample funding plans, grant reports, audit reports, and financial status reports.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The program has procedures in place to improve efficiencies in execution. The establishment of standard grant announcements for the agency has reduced time and labor hours that had previously been expended in developing specific announcements. CSAP has taken the first steps to develop cost bands appropriate to prevention as an efficiency measure to monitor grantee costs and assure that they are reasonable. Competitive sourcing has resulted in an outsourced function that provides review for CSAP PRNS competitive grants and contracts. Redundancies are avoided through cross-program and inter-agency coordination on IT issues. The program operates with a relatively limited number of Federal staff. Program staff review proposed budgets to identify excessive or inappropriate costs.

Evidence: Evidence includes Standard Grant Announcements (available at alt.samhsa.gov/Grants/generalinfo/useful_Info.asp#standard), the contract to pursue a cost-band efficiency measure, the SAMHSA competitive sourcing plan, and the SAMHSA President's Management Agreement (PMA) IT improvement plan.

Does the program collaborate and coordinate effectively with related programs?

Explanation: The CSAP Director convenes the Prevention Partnership, a broad group of 29 national public and private prevention partners. The program collaborates and coordinates with a broad range of Federal partners. One example of the effectiveness of this collaboration is that NIDA will be co-funding the evaluation of the SPF SIG. Additional co-funding examples are: (1) co-funding for NCADI from ONDCP, Justice, and Education; (2) co-funding for NREP from the Centers for Disease Control and Prevention, Justice, Education, and NCI; and, (3) co-funding for CAPTs from Justice, Education, and ONDCP. Within SAMHSA, the CSAP PRNS program collaborates and coordinates with the SAPT Block Grant Program; as a result, a consistent set of 7 performance measurement domains will be used across all programs. The program also coordinates and collaborates with programs of the other two SAMHSA Centers. The SIG program promotes coordination at the State level of all public and private prevention resources.

Evidence: Discussions with agency staff as well as funding announcements for co-funded activities, the use of the 7 domains for performance measurement across SAMHSA, and the development of the SIG program provide evidence of the outcomes of CSAP PRNS's collaboration with other related programs.

Does the program use strong financial management practices?

Explanation: The program receives clean opinions on its audits and is free of material internal control weaknesses. The agency has procedures in place to ensure strong financial management practices, including: 1) audits of grantees and contractors that receive $300,000 or more; 2) OIG audits; 3) pre- and post-award financial management capability reviews; 4) site monitoring reviews; 5) financial audits; 6) information management system audits; 7) internal management control reviews; 8) progress reporting; and 9) financial status reporting. The agency's fiscal monitoring of grant awards is conducted through the SAMHSA Grants Information Management System (SGIMS), which tracks awards and obligations, carry over and submission of quarterly reports, application renewals and final reports.

Evidence: Evidence includes recent audit reports and a description of procedures to identify financial management issues.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: Over the last several years, the expectations and requirements for data collection and reporting have been strengthened and made explicit in grant announcements. When a grantee does not meet these requirements, CSAP provides TA through the Centers for the Application of Prevention Technologies. If noncompliance continues, grantees may be placed on "high-risk" restriction. The project officer, together with a grants management specialist, will determine the course of action, which may include grantee submission of status reports on program and financial activities every 20 days as a condition of continued funding. Grantee monitoring duties are an element of all project officers' performance contracts.

Evidence: Discussions with agency staff and grantee contracts provide evidence that CSAP PRNS is strengthening its management practices to encourage and reward performance for both grantees and employees.

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: A central office within the agency, with the support of an outsourced review function, organizes and conducts independent review of grant and contract applications for CSAP PRNS programs. Applications are peer reviewed based on clear criteria and awards are made based on merit as judged through the peer review process. Hard earmarks specified by Congress in the agency's appropriation undergo an objective review. 97% of funds in 2003 were competitively awarded. Outreach is undertaken to encourage the participation of new grantees. A TA Manual is available at the SAMHSA web site to assist potential applicants with the development of grant applications. The web site also includes a webcast and CDRom for potential applicants, the standard grant announcements, Federal Register Notices, Notices of Funding Availability, and links to other sites to assist applicants. On-site technical assistance outreach efforts have been undertaken regionally to encourage applications from community-based and faith-based organizations.

Evidence: CSAP PRNS conducts outreach and technical assistance to ensure that as many promising programs as possible can prepare to compete for the program's grants. Grant announcements include a clear description of selection criteria. The percent of funds earmarked is low compared to many other federal programs.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: CSAP staff serving as project offices receive data on grantee activity quarterly through the agency's SGIMS system. Project officers review and approve annual budgets. Grantees report quarterly or annually on performance. Project staff oversee grants and contracts through phone contact, site visits, grantee meetings, and meting with program coordinating centers.

Evidence: Sample quarterly and annual reports, descriptions of oversight techniques, correspondence with grantees, and site visit schedules establish that through these activites, project officers are able to identify and manage issues that arise regarding grantee activities.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: Annual performance data aggregated to the national level are currently posted on the SAMHSA web site in the GPRA Performance Report. The CSAP Data Coordinating Center is developing a public web site where disaggregated data from all CSAP programs, analyzed data, and outside data sources will be available to the public.

Evidence: Aggregate data are available at www.samhsa.gov and in the 2005 GPRA performance report. Grantee-level data will be available on the internet by December 2005.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: CSAP PRNS is refining long-term measures already used by ONDCP and Healthy People 2010 so that the data accurately measure the particular impact of this program on state and national substance abuse trends. The program has set baselines and targets, which may be revised based on improved state epidemiological data that will be required by grantees. Evaluations suggest that some CSAP PRNS components are achieving these long-term goals.

Evidence: National-level trend data are available in SAMHSA's Results from the 2002 National Survey on Drug Use and Health: National Findings and NIDA's Monitoring the Future National Results on Adolescent Drug Use: Overview of Key Findings. Further information on current and developmental performance measures is available in the 2005 and 2006 (forthcoming) GPRA Plans, both of which are included in SAMHSA's budget justification to Congress, and in ONDCP's 2005 National Drug Control Strategy.

Does the program (including program partners) achieve its annual performance goals?

Explanation: The program makes progress in achieving annual performance output goals, such as the large increase in state adoption of evidence-based policies, practices, and strategies. The remaining annual goals listed in the PART are developmental; baselines and targets have been set. Evaluations suggest that some CSAP PRNS components are achieving these annual goals.

Evidence: Information on current and developmental performance measures is available in the 2005 and 2006 (forthcoming) GPRA Plans, both of which are included in SAMHSA's budget justification to Congress, and in ONDCP's 2005 National Drug Control Strategy.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The program has initiated steps to improve efficiencies both in SAMHSA's internal management and in its grant programs. Personnel, IT, grants review, contract management, and other functions have been consolidated or outsourced. A number of small CSAP data and evaluation contracts are being consolidated into one larger contract, leading to efficiencies in administration and oversight. SAMHSA has also streamlined the grants application process by introducing four standard grant announcements. The program is moving away from having many small grant programs to having a few larger, longer-term programs. The agency is contracting for a cost bands study; when it is completed, CSAP and its grantees will be able to better monitor and control program costs.

Evidence: Competitive sourcing plans for 2003 and 2004, the agency's IT improvement plan, the Standard Grant Announcements available on the agency's website, and discussions with staff regarding consolidation of evaluation contracts and funding streamt are evidence of progress toward improved efficiency. A contract for development of a cost-band efficiency measure has been issued; data from this measure will help to determine and document efficiency gains.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: This program compares favorably with other federal programs aimed at preventing substance abuse. First, while evaluations of program components found modest results, evaluations of other federal programs either have not been conducted (SAPT BG), have focused on process rather than performance (DFCS), or have found no impact on reducing substance abuse (SDFS and Youth Media Campaign). Second, Prevention PRNS is the only federally-funded substance abuse prevention program that both requires grantees to report on outcome data and has baselines and targets for outcome-based performance measures. Finally, this program is more targeted toward areas most in need of substance abuse prevention resources than the Substance Abuse Block Grant. The comparison of the performance of this program with other programs will become more clear once data on progress towards achieving annual and long-term performance goals are available. There have been no comparisons of this program with other publicly-funded or privately funded (ie foundations, etc.) substance abuse prevention programs. In practice, this would be exceedingly difficult because individual prevention grantees often receive funds from several of these sources.

Evidence: PART assessments of the Substance Abuse Prevention and Treatment Block Grant, Drug-Free Communities Support program, Safe and Drug-Free Schools, and the National Youth Anti-Drug Media Campaign; ONDCP FY 2005 National Drug Control Strategy; and discussions with program staff suggest that this program compares favorably with other similar programs.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: Rigorous evaluations of CSAP program components show modest yet promising results (see details in #4.4). CSAP conducts or contracts for the evaluation of all of its programs, many of which include treatment and control groups to identify program effects. Many of these evaluations determined progress toward achieving program goals among program participants compared with the control group. However, several evaluations suffered from extensive missing data as a result of grantees not reporting on program measures. Consequently, many of the program effects documented in these evaluations were not statistically significant. CSAP PRNS has addressed this by requiring grantees to report performance data and will hold grantees accountable to reporting and performance standards.

Evidence: Evaluations of the High Risk Youth, Youth Mentoring, Family Strengthening, SIG, Community Initiated Prevention Interventions, and Strategic Prevention Framework SIG programs; and cost-evaluation studies support the explanation provided.