Detailed Information on the
Substance Abuse Prevention and Treatment Block Grant Assessment

Developing data for performance measures

Conducting independent and comprehensive evaluation of the national program

Modifying the FY2008 SAPT application to include required National Outcome Measures (NOMs) performance measure form.

Finalize developmental measures through the Technical Consultation Group (TCG) process.

Continue to improve the quality and accuracy of performance and outcome data submitted by the States to SAMHSA

Convene States to review and discuss the interpretation of the submitted data

Create management reports for State Project Officers to share with their states to improve program performance

Presenting performance information disaggregated by State on the website

Begin use of the new Uniform Application

SAMHSA submitted a request to modify its PART measures for the both the treatment and the prevention portions of the SAPT Block Grant set aside to be consistent with SAMHSA NOMs and reflect the State focus of the program. These changes were approved.

Disseminate States' substance abuse prevention and treatment performance data

Measure: Percentage of clients reporting change in drug use abstinence at discharge from treatment

Explanation:The long-term measure of change in abstinence at discharge is retiring and being replaced with two annual measures; one reflects abstinence from drug use at discharge and one reflects abstinence from alcohol at discharge.

Measure: Percentage of states that provide drug treatment services within approved cost per person bands by the type of treatment including outpatient non-methadone; outpatient methadone; and residential treatment services (treatment)

Explanation:The target for percent of grantees in appropriate cost bands was missed for 2006. A substantial number of the States have and are in the process of implementing new or modified data collection systems in response to the mandated National Outcome Measures reporting. These new systems have been focusing on quality of client change data and have not yet refined the cost reporting portions. CSAT expects that once refinements are made to this component of these systems, an increase in this figure will be seen.

Measure: Percentage of clients reporting abstinence from drug use at discharge.

Explanation:

Measure: Percentage of clients reporting anstinence from alcohol at discharge.

Explanation:

Measure: Number of persons served (treatment)

Explanation:Treatment Epidosde Data Set (TEDS) admissions data have been used as proxy data to set targets and track results. However, TEDS data represent admissions to treatment, not the total number of individual clients served. A person who presents for treatment twice during the data collection cycle will be included twice in TEDS. TEDS admissions data do not capture either the total ational demand for substance abuse treatment or the prevalence of substance use in the general population; data only represent admissions to treatment facilities within the scope of TEDS data collection. This measure is one of SAMHSA's National Outcome Measures, which, when fully implemented by the end of FY 2007, will provide more direct and accurate data on number of clients served. The number of admissions measure is one of SAMHSA's National Outcome Measures, which, when fully implemented, will provide more direct and accurate data on number of clients served by reporting an unduplicated count of clients. The unduplicated reporting will be phased in among the States. As States begin to report unduplicated counts, the Treatment Episode Data Set might show that that the number of admissions has gone down, since readmissions of the same individual in the reporting period would be counted as a single client served. Targets may be adjusted to reflect this change. The performance target was set at an approximate target level, and the deviation from that level is slight. There was no effect on overall program or activity performance.

Measure: Percent of program costs spent on Evidence Based Practices (EBPs).

Explanation:The measure is computed as dollars used for evidence based prevention programs or practices/Total prevention program dollars). The SAPT is a State-level program that aims for State-level change. Programs that are evidence based have been shown to produce outcomes beneficial to participants and to be cost-efficient. Therefore, the higher the percentage of funds dedicated to evidence based programs, the more cost efficient would be the program.

Measure: Increase number of States reporting retail tobacco sales violation rates at or below 20 percent (prevention).

Explanation:

Measure: Percent of States showing an increase in State level estimates of survey respondents who rate the risk of substance abuse as moderate or great (age 12-17) (prevention).

Explanation:

Measure: Percent of States showing an increase in state level estimates of survey respondents who rate the risk of substance abuse as moderate or great (age 18 and up) (prevention).

Explanation:

Measure: Number of participants served by prevention programs and strategies (prevention).

Explanation:

Measure: Percent of program costs spent on Evidence-based Practices (prevention).

Explanation:

Measure: Number of Evidence-based policies, practices, and strategies implemented: cumulative (prevention).

Explanation:

Is the program purpose clear?

Explanation: The purpose of the Substance Abuse Prevention and Treatment Block Grant is to distribute by formula funds to states and territories to support substance abuse treatment and prevention services. The block grant provides financial assistance to states to plan, carry out, and evaluate activities to prevent and treat substance abuse and for related public health activities (e.g., HIV and TB). Five percent of the total is used by the agency for technical assistance, data collection and other activities. Up to five percent of state allotments can be used for administrative costs at the state level. States are required to spend no less than 20 percent on prevention. The block grant also addresses special needs such as treatment for pregnant women, women with dependent children, and intravenous drug users. Resources from the block grant can also be used to reduce the rate at which retailers sell tobacco products to minors.

Evidence: The program is authorized by sections 1921-1954 of the Public Health Service Act. The block grant's Synar amendment requires states to enact and enforce legislation to prohibit the youth tobacco sales and meet specific targets for reductions in tobacco sales to youth. The amendment calls for penalties for states that fail to achieve their targeted reductions. Agency and Congressional reports related to the program are consistent with the program purpose as outlined in the authorizing legislation. The program is run by the Substance Abuse and Mental Health Services Administration (SAMHSA).

Does the program address a specific and existing problem, interest, or need?

Explanation: The program is designed to provide resources to states to pay for substance abuse treatment and prevention services. The need for substance abuse treatment and prevention services is clear and current. The agency estimates that of the resources dedicated to treatment, roughly one third support drug treatment, one third alcohol treatment, and one third co-occurring drug and alcohol.

Evidence: The 2001 National Household Survey on Drug Use and Health (NHSDUH) estimates 16 million Americans used an illicit drug in the past month, 6.1 million persons above age 12 need treatment, 5.0 million need treatment but are not getting it, and 4.6 million people who meet the criteria for needing treatment do not even recognize that they need treatment. Youths aged 12 to 17 have the second highest rates of abuse of or dependence on alcohol or an illicit drug (8%), following adults aged 18 to 25 (18%) and higher than adults aged 26 or older (5%). According to the survey, about 10 million youth aged 12 to 20 used alcohol in the past month and nearly 3 million were dependent on or abused alcohol in the past year. Over 3 million persons aged 12 to 17 had smoked cigarettes during the past month.

Is the program designed so that it is not redundant or duplicative of any Federal, state, local or private effort?

Explanation: The program is not overly redundant of other Federal, state, local or private efforts. Numerous federal funding sources are available to support substance abuse treatment and prevention services. SAMHSA also provides competitive grants to state and local entities for treatment and prevention services through the Programs of Regional and National Significance. State and local entities also invest resources in this area. However, the block grant is the only federal activity designed specifically to support state-wide services to all states in this area.

Evidence: According to the agency, the block grant constitutes two of every five public substance abuse treatment dollars expended by the state level agencies funded by the block grant and in some cases states rely entirely on the grant for their substance abuse prevention efforts. Twenty-two of these state agencies reported that greater than half of their total funding for substance abuse prevention and treatment programs came from the federal block grant and 11 states reported over 60 percent and seven states reported over 70 percent. When including all public funding expended through various sources (including Medicaid, TANF, other), the block grant constitutes roughly one of every seven public substance abuse treatment dollars.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The block grant is free from major design flaws that prevent it from meeting its defined objective of supporting state efforts to prevent and treatment substance use. However, improvements are needed and the agency is reviewing approaches to shift the program emphasis from set-asides and other state funding requirements to reporting on the outcomes of grant expenditures. While there are possible flaws to the distribution of funds described below, there is no strong evidence that another approach or mechanism such as competitive grants would be more efficient or effective.

Evidence: Section 1930 of the PHS Act specifies maintenance of effort requirements for states and territories. As reauthorized by the Children's Health Act of 2000, the requirement excludes non-recurring activities. Statute and regulations require states to report how they spent their grant funds and do not require reporting on the impact the funds have on individuals or targeted populations. GAO HEHS 00-50 describes patterns of state expenditures and current limitations on reporting on the outcomes of block grant funded services. Specifically, the statute and regulation requires states to report how they spent funds, not on the impact the funds have on individuals or targeted populations. The transition to a performance partnership grant is intended to increase the emphasis on outcomes, performance, and program improvements. The proposal does not include changes to the formula, eligibility, or basic functions of the block grant.

Is the program effectively targeted, so program resources reach intended beneficiaries and/or otherwise address the program's purpose directly?

Explanation: The guidance requires consideration of how well funds are targeted to meet the purpose and the allocation of funds and the prevalence of drug use by state are not correlated. A 2001 internal report completed by the agency looked specifically at this issue of block grant funding allocations compared with state drug use prevalence rates from an HHS drug use survey (NHSDUH, 2001) and found no correlation. The calculation plots the amount of funding distributed in accordance with the formula in statute against prevalence. A strong correlation with prevalence would improve the chances that individuals will have the same probability of getting care regardless of where they live. It is clear, however, that states provide needs assessments and target funds to appropriate populations and maintenance of effort guards against supplantation. The age profile of the population was the best available proxy for dependence when the formula was created. Finding a data source for prevalence that is sufficiently stable and that also captures substance abuse prevention is difficult.

Evidence: The formula relies on age of population with urban weighting as a proxy for prevalence, total taxable resources, and the cost of services as determined by the cost of health care worker wages and other costs. A 1992 hold harmless provision and subsequent minimum allotment requirements have maintained funding patterns while drug abuse patterns have changed. A 1995 RAND evaluation concluded a focus on a more narrowly defined population, such as the poor and uninsured, rather than the general state population, would have a significant impact on state distributions (RAND, MR-533-HHS/DPRC, 1995). The report also found the emphasis on urban populations is incongruent with higher alcohol dependence rates in rural areas and the emphasis on 18-24 year olds does not align with prevention services. Among persons above age 12, the rate of current illicit drug use in 2001 was 8.3 percent in the West, 7.5 percent in the Northeast, 6.8 percent in the Midwest, and 6.2 percent in the South (HHS, NHSDU, 2001).

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The program has adopted some new long-term outcome measures including: Percentage of clients reporting change in abstinence at discharge; and, Percentage of states that provide drug treatment services within approved cost per person bands by the type of treatment including outpatient non-methadone, outpatient methadone, and residential treatment services. The cost ranges are for outpatient non-methadone $1000-$5000, outpatient methadone $1500-$8000, and residential $3000-$10,000. Outcome measures for the prevention element of the block grant are not yet available. Age of initiation of drug use and also thirty-day use are key indicators of youth drug use. SAMHSA views the two proposed measures, age of initiation of drug use and thirty-day use, inappropriate measures for the program's prevention activities.

Evidence: For the first measure, a discharge record is created for all clients who enter and leave treatment by completion, transfer to other facilities, withdrawal from treatment before completion or death. The discharge record must completed by 30 days post discharge date. For clients who leave treatment before completion, the clinical provider conducts an assessment to provide abstinence data. The cost measure was developed based on the Substance Abuse Treatment Programs of Regional and National Significance measure. SAMHSA has been working with NASADAD, the National Prevention Network and state representatives to develop and refine performance measures for the performance partnership grants since 1995. A notice in the December 24, 2002 Federal Register describes central elements of the proposed transition to performance partnership grants. (Master Summary, NASADAD, 1997-2003; Report on Consensus Building Effort, CSAP, 2001).

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: Baselines and targets for the long-term outcome measures that have been adopted are not yet available. Once a long-term outcome measure for prevention is adopted, baseline and targets will also be developed for the prevention measure.

Evidence: Baseline data for both measures will be available in the FY 2005 uniform block grant application.

Does the program have a limited number of specific annual performance measures that demonstrate progress toward achieving the program's long-term measures?

Explanation: The program has adopted annual outcome and output measures. The measures include: Perceptions of harm of substance use among program participants (prevention); Percentage of clients reporting change in abstinence at discharge (treatment); and Number of persons served (treatment).

Evidence: The prevention measure captures the agency's programmatic focus on reducing risk factors and strengthening protective factors. The number of persons served is calculated using the number of admissions from the Treatment Episodes Data Set divided by 1.67, which SAMHSA believes is a reasonable estimate for the number of persons served. The current uniform application includes voluntary reporting on the number of persons served and SAMHSA intends to negotiate new reporting through the performance partnership grant process.

Does the program have baselines and ambitious targets and timeframes for its annual measures?

Explanation: Baselines and targets for the annual measures are not yet available.

Evidence: Baseline data for the first measure from the program are not yet available. Baseline data for the second measures will be available in the FY 2005 uniform block grant application. An estimated baseline and targets are available for the number of persons served.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, etc.) commit to and work toward the annual and/or long-term goals of the program?

Explanation: Program managers work to ensure states support the overall goals of the block grant and measure and report on performance as it relates to accomplishing goals. States commit to the overall objectives of the block grant to provide treatment and rehabilitation services to those abusing alcohol and drugs and prevention services to prevent use and abuse. States are also asked to voluntarily report on a number of outcome measures, for example, disapproval of substance use or involvement with the criminal justice system. States include descriptions of how they will meet overarching goals of the program in state plans and reports. States are also involved in the setting of goals through planning for the transition to performance partnership grants. Commitment toward the goals of the program should increase further through this transition in coming years.

Evidence: As of 2001, 25 states reported some or all information, up from no states in 1999. States and territories include needs assessment data in their applications, but do not yet report on outcomes related to the annual and long-term goals of the block grant. A notice in the December 24, 2002 Federal Register describes central elements of the proposed transition to performance partnership grants.

Are independent and quality evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: No comprehensive and external evaluations have been conducted on this program. GAO has reviewed some aspects of the substance abuse block grant, including the extent to which impact data are available. The agency also conducts reviews of state activities through on-site reviews, reviews of applications, and reviews of financial audit reports. By design, accountability and evaluations have been focused on compliance with statute, including set-aside requirements, and not on the impact of the block grant. Many states also conduct evaluations, but they are not currently aggregated or reported on at the national level. Less than half of states report the ability to submit client outcome studies and the frequency, methodologies and definitions of studies vary by state (NASADAD). SAMHSA's Treatment Outcomes and Performance Pilot Studies Enhancement is designed to help states measure outcomes of substance abuse treatment from block grant funded programs.

Evidence: SAMHSA reports grantee efforts for evaluation, but no independent, comprehensive evaluations of the program are available. SAMHSA does conduct not less than ten annual state performance assessments to evaluate compliance with the statute and regulation. The assessments focus on legislative set-aside requirements and systems changes. SAMHSA also performs 15 annual state prevention system assessment reviews and provides technical assistance based on the outcomes. Sixteen states currently report follow up data and three states report on outcomes of treatment through the employment and administrative data systems. For prevention, SAMHSA conducts State Prevention Advancement and Support Project performs assessment reviews in 15 states each year and provides technical assistance based on the outcomes. The prevention state level studies are contracted out and done independently. GAO reviewed efforts to increase information on outcomes (HEHS 00-50). RAND conducted an evaluation of the funding formula in 1995 (RAND, MR-533-HHS/DPRC, 1995).

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The program does not provide a budget presentation that clearly ties the impact of funding decisions on expected performance or explains why the requested performance and resource mix is appropriate. Annual budget requests are not clearly derived by estimating what is needed to accomplish long-term outcomes. The program has different output goals and has not identified how much cost is attributed to each goal. The program is able to estimate outputs (number of persons served) per increased increment of dollars. The block grant supports 40 full time equivalent staff. Other agency program management funds are budgeted separately.

Evidence: This assessment is based on the annual budget submission to OMB and the Congress.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: SAMHSA is currently undertaking a comprehensive strategic planning effort to address accountability, capacity, and effectiveness. A deficiency highlighted in this section relates to program budget alignment with program goals. The program is developing new long-term outcome measures, baselines and targets. Having these measures in place will further enable the program to integrate budget planning and strategic planning and determine the level of financial resources needed to obtain long-term outcomes. States were asked to report on a voluntary basis on alcohol and drug use, employment status, criminal justice involvement and living arrangements in the 2000 applications. The agency's efforts to develop a performance partnership grant will also facilitate agency commitment to and reporting on performance measures for the grant. SAMHSA also plans to pilot test an independent evaluation of several performance measures that relate to national and state goals, objectives, and targets.

Evidence: The agency reports developing performance based budgeting to strengthen the links between performance and budget. The agency's restructuring plan consolidated budget formulation, planning and Government Performance and Results Act activities within one unit. As described in a December 24, 2002 Federal Register notice, the performance partnership grant is based on a shift toward greater accountability in exchange for state flexibility to design, implement, and evaluate community-based responses to substance abuse. SAMHSA is currently working with the states to identify core measures for substance abuse treatment and prevention. The planned evaluation is to be independently conducted and focus on multiple factors, including federal programs and funding streams and state and local resources. SAMHSA has developed an evaluation contract directed toward improving program evaluation in the block grant and other SAMHSA programs.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The program collects performance information on an annual basis and uses the information to manage the program and improve performance. The states submit annual uniform applications that describe past, current, and intended use of program funds. States conduct needs assessments and provide a description by state and sub-state planning areas of the incidence and prevalence of alcohol abuse, alcoholism and drug abuse, current prevention and treatment activities and technical assistance requests. The program also collects annual information on state satisfaction with agency technical assistance and the grant review process. Program performance data are also collected during onsite technical reviews. SAMHSA also uses data from national surveys to guide technical assistance efforts.

Evidence: The assessment is based on agency descriptions of actions taken based on performance information and on state annual reporting forms and plans. More than 20 States now require a percentage of their block grant funds to be allocated to implement science-based or model prevention programs. The agency's prevention system assessments provide states with specific recommendations for technical assistance to improve their prevention programs. These findings also guide agency planning efforts (Prevention System Assessment Summary Report, CSAP, 1999-2003).

Are Federal managers and program partners (grantees, subgrantees, contractors, cost-sharing partners, etc.) held accountable for cost, schedule and performance results?

Explanation: Performance plans for managers at the Division Director level and above track to management/program objectives. Managers review state compliance with the legislative requirements and monitor expenditures through compliance reviews and single audit reports, ensure that applicable financial status reports are completed, and reconcile financial status reports to the Payment Management System. Performance Based Contracting has been initiated for all new SAMHSA contractors' who hold services contracts. The transition to performance partnership grants will increase the accountability of program partners for performance results.

Evidence: The assessment is based on discussions with the agency and program manager vacancy announcements. Employee evaluations at the agency are handled by each of the agency's three centers. One planned element of the performance partnership grants is to use corrective action plans as a means of increasing accountability for performance results and making program improvements. The agency reviews state requests for waivers for maintenance of effort requirements based on extraordinary economic circumstances and notes the agency can reduce state awards if the state does not comply.

Are all funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: The agency reports funds are obligated by the government on a quarterly basis, usually within two-three days after an application has been determined compliant with relevant requirements of the Public Health Service Act. States have two years to obligate and expend funds to sub-recipients. The agency's technical reviews have found states are generally in compliance with allowable expenditure requirements, but some states are not (Aggregate Report of Revised Core Elements Technical Reviews, CSAT, 2002).

Evidence: Agency managers review annual grantee applications to determine funds are used for the intended purpose. Agency staff also examine the states' obligations and expenditures of grant funds during state technical reviews. The technical reviews found of the 32 states reviewed, 12 lapsed block grant funds during the review period and three states expended block grant funds in the criminal justice system, which is a prohibited expense.

Does the program have procedures (e.g., competitive sourcing/cost comparisons, IT improvements, approporaite incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The program has some procedures in place to improve efficiencies in execution. SAMHSA has established a block grant re-engineering team to improve the efficiency of staff operations in managing the program at the federal level in time for the 2004 application process. The agency plans to switch to a web-based application system in 2004. The agency relies on an HHS service clearinghouse for many internal services. The agency is providing FAIR Act targets and appears to be making progress toward outsourcing additional services. There are also elements in the block grant that seek to limit administrative costs. For example, there is a five percent limitation on administrative costs at both the federal and grantee levels. Each state and territory uses the fiscal policies that apply to its own funds for administering the block grant. Additional steps, including adoption of measures for efficiency of operations, are needed to maintain progress in this area.

Evidence: Evidence includes the FAIR Act report, services directed to HHS' consolidated Program Support Center, and Restriction of Expenditure of Grant. Outsourced activities include accounting, graphics, human resources, and property management. With the federal set-aside, there are 22 treatment project officers, including state data infrastructure activities, and 15 prevention project officers, including five associated with Synar. There is, however, continual competition for the block grant set-aside for data resources and other federal-level activities. Beginning next year, SAMHSA plans to convert the application system from Windows to an internet system for states to prepare and submit applications on line. SAMHSA projects savings associated with the new system as the independent contractor reduces staff support by 20%.

Does the program collaborate and coordinate effectively with related programs?

Explanation: The program does collaborate with related activities. For example, in the substance abuse prevention area, by design SAMHSA's prevention state incentive grants collaborate with the block grant at the federal and state level. The state incentive grants also promote changes in activities funded by the block grant and in the entire state prevention system. SAMHSA also collaborates with other federal, state, and local governments as well as non-governmental organizations. SAMHSA collaborates with HHS's Center for Medicaid and Medicare Services on the review of state Medicaid waivers and with the Office of National Drug Control Policy.

Evidence: Evidence for this question is included in the Government Performance and Results Act report, meetings, conferences, and other documentation. Examples of specific activities include work with sister offices HRSA and NIAAA on national alcohol screening day, contributions in TANF and SCHIP regional meetings, collaboration with the Administration for Children and Families, work with the Indian Health Service on tribal populations, research planning with NIH, and joint conferences, workshops and planning meetings with HRSA and other agencies.

Does the program use strong financial management practices?

Explanation: The program receives clean opinions on its audits and is free of material internal control weaknesses. SAMHSA is participating in a department-wide initiative to implement a new Unified Financial Management System. SAMHSA will in the meantime replace the current DOS-based Integrated Financial Management System with a customized government-off-the-shelf system for tracking commitment and obligation data. The Integrated Resource Management System provides for tracking of commitments and obligations and for numerous management reports.

Evidence: Discussions and documents from agency managers, audited statements from the Program Support Center; Office of the Inspector General reports.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The program is taking meaningful steps to address management deficiencies in key areas. A conversion to a performance partnership grant will increase the amount of information gathered on grantee performance on select outcome measures. The program is addressing accountability for results at both the federal and grantee level. The agency is taking steps to begin retraining federal project officers on a new skill set needed to successfully transform the block grant into a performance partnership grant. The new grant will require states to report on a common set of performance measures and state-specific goals. The agency seeks to work with states under the new arrangement to better target technical assistance and help states improve program performance.

Evidence: SAMHSA is developing a website for a state profile database that will include state-specific information excerpted from the uniform applications for the block grant and two of the agency's national surveys and will eventually be made available to the public. The agency plans to implement performance plans for all staff, which must include at least one element that tracks back to these objectives by September 30, 2003. The agency also plans to ensure program and management objectives in the SAMHSA Administrator's performance contract are incorporated into the performance plans of senior management and staffs. The use of performance measures in employee evaluations is under examination.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: The program does have sufficient oversight capacity. This capacity will improve with respect to outcomes of the block grant with the transition to performance partnerships. However, the program is able to document grantees' use of funds in compliance with legislatively designated categories, conducts site visits to a substantial number of grantees on a regular basis and confirms expenditures in annual reports. Through national level relationships and the work of the project officers, the program has a fairly high level of understanding of what grantees do with the resources allocated to them. The agency's State Systems Development Program includes technical reviews of state operations. The reviews examine state systems, quality assurance efforts, and compliance with set-asides and other requirements. Select documentation from states indicate the reviews are also useful from the grantee's perspective.

Evidence: Evidence includes agency documentation, applications and the performance plans and reports. The 1999-2002 technical review project provides details on the 32 of the states. Financial findings include 94 percent of states review financial reports and six percent have annual budget reviews. Quality assurance findings include 91 percent use placement criteria, 28 percent use outcome measures and three percent use performance-based contracts. Three states were not spending at or above the 20 percent prevention set-aside and four more had inadequate data to determine compliance. Other factors include lapsing funds, prohibited expenditures, confidentiality procedures, and management tools (Aggregate Report, 2002).

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: Grantee performance data are currently only available to the public at the national level and not disaggregated by state. Annual performance data are aggregated in the performance report and are available to the public through the SAMHSA web site. A conversion to a performance partnership grant will also increase the amount of information gathered on grantee performance on select outcome measures. Each state conducts a public comment forum on the intended use of block grant funds.

Evidence: Assessment based on agency web site (www.samhsa.gov/funding/funding.html). Additional information is available through the National Association of State Alcohol and Drug Abuse Directors (www.nasadad.org/).

Has the program demonstrated adequate progress in achieving its long-term outcome performance goals?

Explanation: As noted in Question 2 of the Strategic Planning section, the agency has not yet adopted specific targets and developed a baseline for new long-term outcome goals. The program's existing annual measures are output and do not demonstrate progress toward achieving long-term performance goals. By design, the emphasis for executing the block grant has been to provide states with a flexible source of funds, technical assistance, and minimal interference and burden. As a result, the program has not to date developed an infrastructure to capture outcomes data from grantees.

Evidence: Assessment based on annual GPRA report, SAMHSA-wide performance measures document and draft measures for the performance partnership grant.

Does the program (including program partners) achieve its annual performance goals?

Explanation: As noted in Question 4 of the Strategic Planning section, the agency has not yet developed a baseline and adopted targets for all the annual goals that support the desired long-term outcomes of the program.

Evidence: Assessment based on annual GPRA report, SAMHSA-wide performance measures document and draft measures for the performance partnership grant.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program performance goals each year?

Explanation: The program has recently initiated steps to improve efficiencies but has realized only limited efficiency improvements to date. The program has relied on electronic means of conducting business, including state applications and reports. The agency has also directed additional services to a consolidated Program Support Center. The program has also increased the efficiency of technical assistance efforts by succeeding in having more efforts result in change in systems, programs or practice. In the future, efforts to transition to a performance partnership grant can also improve efficiency in achieving program goals. The agency states that changes to an internet based application next year will also reduce administrative costs. A reengineering effort recently initiated may also improve efficiencies in the future better coordinate technical assistance across various agency programs.

Evidence: The percentage of technical assistance events resulting in changes in state systems, programs or practices increased from 66% in 1999 to 84% in 2000. The agency's efforts to transition to a performance partnership grant are intended to reduce requirements in the block grant through an increase reliance on reporting on outcomes. The new structure should enable the program to more efficiently achieve outcome goals in substance abuse treatment and prevention. SAMHSA has also developed a template for states to determine costs of prevention services as a first step toward determining cost-effectiveness. The agency has not undergone an A-76 competition.

Does the performance of this program compare favorably to other programs, including government, private, etc., that have similar purpose and goals?

Explanation: Numerous Federal funding sources are available to support substance abuse treatment and prevention services. State and local entities also invest resources in this area. However, the block grant is the only federal activity designed specifically to support state-wide services to all states in this area. No comparisons of the effectiveness of treatment services through Medicaid and treatment services supported by the block grant have been conducted.

Evidence: Evidence includes GAO HEHS 00-50, agency budget reports.

Do independent and quality evaluations of this program indicate that the program is effective and achieving results?

Explanation: The program has not had evaluations meeting the standard for this question that are at the national program level, rather than one or more partners, are comprehensive and focused on the program's impact, effectiveness or other measurement of performance. State technical reviews provide information on the states' obligations and expenditures in accordance with the statute, service delivery by modality, quality improvement and opportunities for technical assistance. The agency reviewed 53 state outcome studies that rely on different time intervals, definitions of use, employment, criminal activity and other factors. While definitions and findings vary, the individual studies indicate treatment is effective. However, states are not reporting on common outcome data. OIG conducted a 1997 evaluation of block grant activities in Minnesota. Prevention studies not specific to the block grant conducted by RAND and other researchers have concluded prevention efforts in schools and the community are cost effective and produce savings resulting from reduced tobacco, alcohol and drug use.

Evidence: Source documents include GAO HEHS 00-50, agency GPRA plans and reports, and other agency documentation. GAO found problems with the quality of state data for the implementation of the Synar amendment (GAO 02-74). Treatment effectiveness studies not focused on the block grant include Drug Abuse Treatment Outcomes Study, Services Research Outcomes Study and other research conducted by external organizations. Data from the 1997 National Treatment Improvement Evaluation Study indicate the agency's substance abuse treatment competitive demonstration grants were effective, but no evaluations have been conducted specific to block grant funded activities. The 1997 OIG report found the state agency administered the grant effectively but did not always require grantees to establish program goals for measurable outcomes and lacked a fully compliant independent peer review process.