Detailed Information on the
Drug-Free Communities Support Program Assessment

To ensure validity and inter-rater reliability in the competitive grant review process, provide in-person training to 50% of peer reviewers.

Implement 5-year training for the initial five years of a DFC grant (starting with new FY09 grantees).

Developing and updating targets for all performance measures in order to improve goal-setting for the program.

Developing a refocused evaluation plan to assess the program's performance and impact.

Implementing performance improvements, including rolling out a new grantee reporting system to collect program performance information.

Measure: Percent of coalitions that report decreased risk factors in community.

Explanation:The earlier output measures have been replaced by these new outcome measures. The measure used to monitor this sub-objective is the percent of coalitions that report a decrease in risk factors??each grantee will select the risk factors it proposes to reduce, based on local needs assessments and strategies.

Measure: Percent of coalitions that report increased protective factors.

Explanation:The earlier output measures have been replaced by these new outcome measures. All grantees are required to collect data and report on two protective factors: 1) youth perception of risk for alcohol, tobacco, and marijuana and 2) youth perception of parental disapproval for alcohol, tobacco, and marijuana.

Measure: Percentage coalitions that report at least 5% improvement in past 30-day youth use of alcohol, tobacco, or marijuana in at least one grade

Explanation:30-day use is generally accepted as a core indicator of youth drug use.

Measure: Percentage coalitions that report positive change in the age of initiation of tobacco, alcohol, or marijuana in at least two grades.

Explanation:Age of initiation is generally accepted as a core indicator of youth drug use.

Measure: Percentage coalitions that report positive change in youth perception of risk from tobacco, alcohol or marijuana in at least two grades

Explanation:Perception of risk is generally accepted as a core indicator of youth drug use.

Measure: Percentage coalitions that report positive change in youth perception of parental disapproval of the use of alcohol, tobacco, or marijuana in at least two grades

Explanation:Youth perception of parental disapproval is generally accepted as a core indicator of youth drug use.

Is the program purpose clear?

Explanation: The Drug-Free Communities Act of 1997 established two strategic goals for the Drug-Free Communities Support Program (DFC): (1) reducing substance abuse among youth, and, over time, among adults, by addressing factors in the community that serve either to increase or minimize the risk of substance abuse; and (2) establishing and strengthening collaboration among communities, Federal, state, local and tribal governments and private nonprofit agencies to support community coalition efforts to prevent and reduce substance abuse among youth.

Evidence: 21 USC1521 et seq., as amended; Report Language from the DFC Reauthorization (Rept. 107-175)

Does the program address a specific and existing problem, interest, or need?

Explanation: Illegal use of controlled substances in the United States remains unacceptably high. According to the Substance Abuse and Mental Health Services Administration (SAMHSA) 2002 National Household Survey on Drug Abuse, 19.5 million Americans ages 12 and older (8.3%) reported using an illicit drug in the month before the survey was conducted.

Evidence: 2001 National Household Survey on Drug Abuse: Volume I, Summary of National Findings; DFC FY03 Grant Funding Announcement (GFA)

Is the program designed so that it is not redundant or duplicative of any Federal, state, local or private effort?

Explanation: The DFC Program provides funds for organizing multiple sectors of a community as a means for reducing and/or preventing substance abuse. There appears to be no other substantial Federal, state, local, or private program that provides grant funds for this purpose. The HHS Substance Abuse Prevention and Treatment Block block grant funding is made to the Single State Authority and then passed on to local providers which generally use these funds to deliver direct services to target populations and/or to address specific drug abuse trends. State Incentive Grants (SIG) can only fund a limited number of "science-based" program models. Only one such coalition model (Communities That Care) has been approved for SIG funding. However, due to the $100K statutory cap on a DFC grant, that model may not be affordable for replication by DFC grantees. Therefore, under a strict interpretation of the funding guidelines of the SIG program, most DFC coalition models are not eligible for funding.

Evidence: Discussions with ONDCP and HHS staff.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The DFC program uses a competitive grant process to award funds to community anti-drug coalitions. There is no strong evidence that another approach or mechanism would be more efficient/effective to achieve the intended purpose.

Evidence: 21 USC1521 et seq., as amended

Is the program effectively targeted, so program resources reach intended beneficiaries and/or otherwise address the program's purpose directly?

Explanation: The intended beneficiaries of the DFC program are established, broad-based, anti-drug coalitions. Applicants to the program are reviewed explicitly against this criteria in the review process. ONDCP requires DFC applicants to submit an assessment of drug use in their community with their grant applications. These requirements ensure that limited DFC funds are provided only to organizations demonstrating local commitment and resolve to address its drug problem.

Evidence: 21 USC1521 et seq., as amended; Report Language from the DFC Reauthorization (Rept. 107-175); DFC FY03 GFA (Grant Funding Announcement)

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The DFC program has identified long-term performance measures that reflect the two statutory purposes of the program -- reducing substance abuse among youth and strengthening community coalition efforts to prevent and reduce substance abuse among youth. The measures related to reducing substance abuse among youth (age of onset; use in the past 30 days; perception of harm; and perception of parental disapproval) are generally accepted by researchers as the best surrogate measures for adolescent drug use. Because of the small size of the DFC grants, the performance of the DFC program will not be measured against national level changes in any of these measures. Rather, the performance will be measured against the extent to which grantees meet the targets identified for their communities.

Evidence: ONDCP Strategic Plan 2002-2008 Fiscal Year 2004 Congressional Budget Submission, pages 86-109 (GPRA) FY03 Data Call Documents

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: As of June 30, ONDCP had established specific quantified targets for establishing and strengthening community coalitions. Those targets had recently been revised to address concerns about whether they were sufficiently ambitious. However, no quantified targets or timeframes had yet been established for the performance measure related to reducing substance abuse among youth. ONDCP expects to have those targets in place prior to submission of the FY 2005 Budget request.

Evidence: ONDCP Strategic Plan 2002-2008 Fiscal Year 2004 Congressional Budget Submission, pages 86-109 (GPRA) Discussions with DFC program staff.

Does the program have a limited number of specific annual performance measures that demonstrate progress toward achieving the program's long-term measures?

Explanation: The DFC program has identified annual performance measures that directly support the program's long-term goals. Those goals are largely incremental increases toward the long term goal.

Evidence: ONDCP Strategic Plan 2002-2008 Fiscal Year 2004 Congressional Budget Submission, pages 86-109 (GPRA) Discussions with DFC program staff.

Does the program have baselines and ambitious targets and timeframes for its annual measures?

Explanation: As of June 30, ONDCP had established specific baselines and targets for its annual measures related to establishing and strengthening community coalitions. However, no quantified targets or timeframes had yet been established for the performance measure related to reducing substance abuse among youth. ONDCP expects to have those targets in place prior to submission of the FY 2005 Budget request.

Evidence: ONDCP Strategic Plan 2002-2008 Fiscal Year 2004 Congressional Budget Submission, pages 86-109 (GPRA) Discussions with DFC program staff.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, etc.) commit to and work toward the annual and/or long-term goals of the program?

Explanation: DFC grantees and ONDCP's Federal partners (Office of Juvenile Justice and Delinquency Prevention (OJJDP), which administers the DFC Program for ONDCP, and HHS' Center for Substance Abuse Prevention (CSAP) which provides technical assistance to grantees) commit to and work toward the annual and long-term measures of the DFC program. The 530 grantees submit semi-annual CAPRs (Categorical Assistance Program Report) to OJJDP with information on how they are meeting their goals and objectives.

Evidence: Fiscal Year 2004 Congressional Budget Submission, pages 86-109 (GPRA) CAPR Part I and Part II DFC FY03 GFA (Grant Funding Announcement) National Anti-Drug Coalition Institute Strategic Plan (2003)

Are independent and quality evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: The DFC program has had an independent evaluation in place since 1998. However, the evaluation did not address program performance adequately. After review of the 2002 report, ONDCP concluded that the evaluation required modification to capture the program's intended outcomes adequately. ONDCP has enhanced the original evaluation plan and has taken steps to ensure that the DFC program has a refocused evaluation in place by the end of 2003.

Evidence: ONDCP Strategic Plan 2002-2008 Fiscal Year 2004 Congressional Budget Submission, pages 86-109 (GPRA) Discussions with DFC program staff.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: ONDCP has not provided budget requests that make clear the impact of funding, policy, or legislative decisions on expected performance and explains why the requested performance/resource mix is appropriate. This is largely due to the absence of adequate program performance measures in past years. A second factor has been the absence in the budget requests of all direct and indirect costs associated with the DFC Program. ONDCP is working to revise its budget presentation for FY 2005 and expect to correct both shortcomings at that time.

Evidence: ONDCP Strategic Plan 2002-2008 Fiscal Year 2004 Congressional Budget Submission, pages 86-109 (GPRA) Discussions with DFC program staff.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: In 2002, ONDCP: requested grantees to track and report on levels of drug use as measured by school-based survey instruments in the target communities; began requiring grantees to submit outcome data on four core measures; notified grantees that continuation funding would be jeopardized if grantees did not provide the outcome data; and began requiring grantees to issue an 'Annual Report to the Community' describing dimensions of local drug use and the coalitions' strategies to address this use. ONDCP plans to move the evaluation contract under its direct control, assign additional staff to that effort, and will re-compete the evaluation contract after a new evaluation design and statement of work is developed with the assistance of national evaluation experts.

Evidence: Discussion with ONDCP staff and review of DFC program documentation.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: Currently, ONDCP relies on information collected through the Categorical Assistance Program Reports (CAPR) every six months from all grantees. However, this information is not closely related to the performance of the program. ONDCP recently collected core measure data from all 531 grantees to determine substance abuse rates in grantee communities. This baseline performance data will enable to ONDCP to set meaningful, ambitious targets and measure grantee performance against those targets.

Evidence: Categorical Assistance Progress Report Forms ' Part 1 and 2; discussions with ONDCP staff

Are Federal managers and program partners (grantees, subgrantees, contractors, cost-sharing partners, etc.) held accountable for cost, schedule and performance results?

Explanation: Grantees are required to establish a system to monitor and report on the performance measures stipulated by ONDCP, including the four measures related to substance abuse among youth. Grantees that fail to make satisfactory progress towards the goals and strategies described in their applications could lose their funding or realize other sanctions. In addition, ONDCP has identified the managers and key staff responsible for achieving key program results and has incorporated the program's performance standards into the rating systems for those managers.

Evidence: DFC FY03 GFA (Grant Funding Announcement); Inter-Agency Agreement (IAA) between ONDCP and OJJDP; IAA between ONDCP and CSAP (for Coalition Institute operations); National Anti-Drug Coalition Institute Strategic Plan (2003); Performance appraisal documents for DFC staff.

Are all funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: DFC appropriated funds have been successfully competed and awarded in a timely manner during the first five years of operation. Only a few serious problems (8 of approximately 540 awards) have arisen with individual grantees regarding the spending of funds and all such problems have been quickly detected and corrected. Administrative cost limits stipulated in the legislation have been met. Funds remaining from a terminated project are returned to the grant pool and are not used for another purpose.

Evidence: SF - 132s, SF -133s, Treasury reports, and OJJDP financial summary reporting forms

Does the program have procedures (e.g., competitive sourcing/cost comparisons, IT improvements, approporaite incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: In FY 2003, OJJDP instituted new grant application submission procedures using emerging web-based technology. This grants management system (GMS) enables applicants to submit applications electronically and not only enables OJJDP to more efficiently review the applications, but provides the basis for consolidating grantee information for analytical purposes. Contract awards (e.g. for peer review support by OJJDP) are also competed as was the Coalition Institute grant.

Evidence: DFC FY03 GFA; discussions with ONDCP and OJJDP staff; ONDCP Report to Congress on Administrative Costs associated with DFC Program. (July 2001)

Does the program collaborate and coordinate effectively with related programs?

Explanation: State alcohol and drug agencies are major collaborators as coalitions often are part of the state strategic planning process. CSAP, CSAT, and NIAAA are frequent collaborators in a wide range of national and local activities. Private sector organizations such as CADCA, Join Together (Boston University), and several Robert Wood Johnson Foundation supported programs (e.g. Governor's Spouses' Initiative to Reduce Underage Drinking) are also key collaborators.

Evidence: Discussions with ONDCP and HHS staff.

Does the program use strong financial management practices?

Explanation: ONDCP's Office of Administration monitors DFC program funds and OJJDP's Office of the Comptroller performs a similar role with individual DFC grant funds. Statutory limits on the expenditure of appropriated funds provide clear guidance on allowable expenditures. OJJDP requires that grantees closely track and report on their spending and matching of grant funds. No material internal control weaknesses have been identified, OJJDP financial management systems meet statutory requirements, and financial information is accurate and timely.

Evidence: Discussions with ONDCP staff; review of program documents.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: In August 2002, ONDCP Director Walters moved the DFC program from the Office of Demand Reduction to the Office of the Deputy Director, who is a recognized expert on community anti-drug coalitions. In recent months, ONDCP has moved to assume the direct supervision of the national evaluation of the program. A new statement of work is in preparation and the new design of the national evaluation will be announced for competitive applications in the fall.

Evidence: Discussions with ONDCP staff.

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: DFC grants are peer reviewed and independently scored by expert panels selected by OJJDP. Senior OJJDP staff and ONDCP staff then further review the highest scoring applications, ensure that funded applicants do not duplicate operations in an area already served by a funded application, and that the additional statutorily priorities relating to serving economically disadvantaged and rural areas are adequately represented in the cohort recommended for funding.

Evidence: OJJDP Peer Review Guidelines (general guidelines for all OJJDP programs); DFC Peer Reviewers Guide (specific guidance to reviewers); discussions with ONDCP staff.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: Program managers at OJJDP are responsible for the oversight and monitoring of DFC grantees. Managers regularly talk with grantees on the phone, engage mail correspondence, review progress reports (CAPRs), and make site visits as appropriate. Furthermore, grantees are in frequent email contact with the administrator at ONDCP and senior staff at OJJDP. Grantees are encouraged to telephone and send email to any senior staff should problems or questions arise. The ONDCP DFC administrator has daily contact with program managers as issues warrant. In addition, the 11 members of the Advisory Commission for Drug-Free Communities, who are appointed by the President, also periodically observe grantee performance and provide feedback and guidance to the Director and Deputy Director of ONDCP

Evidence: Categorical Assistance Progress Report Forms ' Part 1 and 2; OJJDP Desk Audit Form; OJJDP Site Visit Reports; discussions with ONDCP and OJJDP staff

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: While ONDCP has improved the collection of DFC Program performance data, it has not made it available to the public in a transparent and meaningful manner. ONDCP plans to extract appropriate data from the final five-year summative reports from the FY 1998 cohort of grantees and place it on the DFC website when they are submitted early in FY 2004. At the end of FY 2003 (the five year mark on the program), ONDCP will prepare a summative report that will include all performance data collected and analyzed to date. ONDCP plans to distribute this report in both print and electronic form. Additionally, the new DFC grant application (FY03) requires grantees to present a plan for reporting the best available data to their community on a regular basis.

Evidence: Review of ONDCP and DFC web sites and publications, discussions with ONDCP staff.

Has the program demonstrated adequate progress in achieving its long-term outcome performance goals?

Explanation: The program has demonstrated some progress in achieving the long-term performance goals related to strengthening collaboration among communities, federal, state, local and tribal governments and private nonprofit agencies to support community coalition efforts to prevent and reduce substance abuse among youth. Performance goals for reducing substance abuse among youth have only recently been established and no quantified targets or timeframes have yet been established.

Evidence: ONDCP Strategic Plan 2002-2008; Fiscal Year 2004 Congressional Budget Submission, pages 86-109 (GPRA); discussions with DFC program staff.

Does the program (including program partners) achieve its annual performance goals?

Explanation: The DFC program has demonstrated some progress in achieving the annual targets related to strengthening collaboration among communities, federal, state, local and tribal governments and private nonprofit agencies to support community coalition efforts to prevent and reduce substance abuse among youth. Performance goals for reducing substance abuse among youth have only recently been established and no quantified targets or timeframes have yet been established.

Evidence: ONDCP Strategic Plan 2002-2008; Fiscal Year 2004 Congressional Budget Submission, pages 86-109 (GPRA); discussions with DFC program staff.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program performance goals each year?

Explanation: With a basically flat administrative budget, the number of active grants that have been funded has grown from about 90 in FY 1998 to more than 600 currently. ONDCP has instituted a new screening process that eliminates non-competitive applications. This new process means that fewer non-competitive applications will undergo the expensive peer review process and that panels will review higher quality applications. This process reduced by approximately 120 applications, the number of applications undergoing peer review, at an estimated cost of more than $800 per application. ONDCP is also developing an internet-based application system that will permit electronic filing of an application and capturing program baseline and performance data.

Evidence: Review of ONDCP and DFC web sites and publications, discussions with ONDCP staff.

Does the performance of this program compare favorably to other programs, including government, private, etc., that have similar purpose and goals?

Explanation: As described in question 1.3 above, there appears to be no other substantial Federal, state, local, or private programs that provides funds for organizing the community and its multiple sectors as the means for reducing and/or preventing substance abuse. Other Federal programs provide funding directly to service providers for more direct provision of services. While these programs share a common broad goal (reducing substance abuse), the methods they use make them inherently different approaches.

Evidence: Discussions with ONDCP and HHS staff.

Do independent and quality evaluations of this program indicate that the program is effective and achieving results?

Explanation: As described in question, 2.6 above, the DFC program has had an independent evaluation in place since 1998 but that evaluation did not address program performance adequately.

Evidence: ONDCP Strategic Plan 2002-2008 Fiscal Year 2004 Congressional Budget Submission, pages 86-109 (GPRA) Discussions with DFC program staff.