Detailed Information on the
Adolescent Family Life Program Assessment

Reevaluate the Office strategic plan, originally created in spring 2006,and develop new goals and timelines. Share this strategic plan with AFL grantees via email, the OPA Web site, and the Annual Conference.

Develop and implement new revised core data collection instruments for AFL grantees.

Implement and analyze an independent cross-site evaluation to study the efficiency and effectiveness of supported activities of AFL Prevention and Care Demonstration projects.

Formalize the coordination function to enhance resource planning and allocation between the AFL program and the Community Based Abstinence Education program to reduce redundancy by creating an interagency agreement to establish quarterly conference calls and the sharing of applicant and grantee lists.

Disseminate AFL grantee results to the public in a transparent and accessible manner by developing processes to podcast grantee findings presented at AFL conferences, make available final grantee reports on the Office of Population Affairs Web site, and have print materials available

Using experts in the field, assess the appropriateness of current performance measures for the AFL program. Propose potential changes to performance measures and data collection methods that more accurately capture the effectiveness of the AFL program by Spring 2010.

Provide targeted and constructive feedback to AFL grantees regarding their 2008 end of year program and evaluation reports.

Develop and implement new revised core data collection instruments for AFL grantees.

Develop performance baselines, measures, and targets based upon data collected from core instruments.

Provide targeted and constructive feedback to AFL grantees regarding their 2007 end of year program and evaluation reports.

Provide technical assistance to AFL grantees regarding the basics of practical evaluation and data collection.

Measure: Increase communication among parents and adolescents on topics relating to puberty, pregnancy, abstinence, alcohol, and/or drugs.

Explanation:This is measured by the change in the proportion of AFL Prevention demonstration project participants who report communication with their parents about sexuality and other health and social benefits between pre and post intervention efforts. The actual percentage of grantees showing positive change decreased in 2007, in part due to intensive technical assistance offered to all AFL grantees. Although the actual outcome data may not have improved, the overall evaluation integrity and data collection process is improving through increased education and guidance on proper evaluation processes and protocols. In addition, grantees may have experienced a more honest level of disclosure from clients with the increase of confidentiality procedures. The targets for FY 2008 and FY 2009 remain the same because a new cohort of grantees will be reporting in FY 2009. The goal for the transition is to maintain the verteran grantee performance.

Measure: Increase adolescents' understanding of the positive health and emotional benefits of abstaining from premarital sexual activity.

Explanation:This is measured by the change in the proportion of AFL Prevention demonstration project participants who report an increase in understanding of the positive health and emotional benefits of abstaining from premarital sexual activity between pre- and post-intervention. Positive change for each grantee was determined by the ratio of change between grantee administered pre- and post-surveys. It is important to note that the sample size is relatively small, this creates a large percentage change even when the difference in the number of grantees is low.

Measure: Maintain the incidence of clients in AFL Care demonstration projects who do not have a repeat pregnancy.

Explanation:Data presented reflect the percentage of clients who were not pregnant one year after giving birth. A 2006 study conducted by Maureen M. Black, et. al, "Delaying Second Births Among Adolescent Mothers: A Randomized, Controlled Trial of a Home-Based Mentoring Program," found that 89% of parenting teens did not experience a repeat pregnancy at a two year follow up, while AFL achieved a 92% result in its baseline year. The goal for the future of this program is to maintain the already high performance in this area.

Measure: Increase infant immunization among clients in AFL Care demonstration projects.

Explanation:Data presented reflect the percentage of clients whose children were appropriately immunized at a twelve month follow-up.

Measure: Increase the educational attainment of clients in AFL Care demonstration projects.

Explanation:Data presented reflects the percentage of clients who were enrolled in or had completed high school or a GED program at the twelve month follow-up.

Measure: Improve the quality of the Title XX independent evaluations.

Explanation:This measure reflects the proportion of annual evaluation reports that meet a standardized set of criteria (i.e., clearly stated hypothesis; adequate sample size; viable comparison strategy; appropriate statistical analyses), as determined by a team of external evaluation consultants. The external evaluation consultants use an evaluation "scoring sheet" that assess the adequacy of the evaluation using a score from 1-5. The percentages below reflect grantees who receive a score greater than or equal to 3.33 , indicating a "promising" evaluation.

Measure: Sustain the cost to encounter ratio in Title XX prevention and care demonstration projects.

Explanation:This measurement is a ratio of costs per number of encounters. Costs equal the expenditure of Title XX project funds for service provision, and number of encounters equals the number of service hours delivered. Number of encounters is used instead of number of clients served because Title XX funded projects use different approaches that vary in the frequency of services delivered. The measurement is separate for prevention and care demonstration projects.

Is the program purpose clear?

Explanation: The purpose of the Adolescent Family Life (AFL) program is to prevent teen pregnancy and sexually transmitted diseases in youth and mitigate the negative consequences associated with being a pregnant and parenting teen. The AFL program supports two types of demonstration grants: (1) prevention demonstration grants to develop and evaluate interventions that provide abstinence education designed to encourage adolescents to postpone sexual activity and promote meaningful family involvement; and (2) care demonstration grants develop and evaluate interventions targeting pregnant and parenting teens in an effort to ameliorate the effects of too-early childbearing for teen parents, their babies, and their families. The AFL program also supports related research through competitive, discretionary grants.

Evidence: Title XX of the Public Health Service Act (P.L. 97-35), which establishes Adolescent Family Life Demonstration Projects, states the need to: (1) find effective means within the context of the family, of reaching adolescents before they become sexually active to promote self discipline and other prudent approaches to the problem of premarital sexual relations, including adolescent pregnancy, (2) establish innovative, comprehensive, and integrated approaches to the delivery of care services for both pregnant adolescents and adolescent parents, and (3) support research concerning societal causes and consequences of adolescent premarital sexual relations, and services which effectively alleviate, eliminate, or resolve any negative consequences for the parents, child, and their family. The Office of Adolescent Pregnancy Programs (OAPP) Demonstration and Care Grant announcements outline the goals and objectives AFL employs to support achievement of the needs mentioned above (Federal Register Vol. 72, No. 62 - pg. 15772, Federal Register Vol. 71, No. 56 - pg. 14786).

Does the program address a specific and existing problem, interest, or need?

Explanation: The AFL program continues to be relevant to current health risks concerning adolescent pregnancy and childbearing, the prevention of adolescent sexual activity and pregnancy, and the continued need to support relevant research and demonstration projects. For example approximately 750,000 teen girls become pregnant each year, of which 82 percent of these pregnancies are unintended. About 40 percent of mothers who have their children before age 18 will never graduate from high school, compared to a 75 percent graduation rate of mothers who delay childbearing until 20-21 years of age. Children of teen parents can also risk early complications by being born with a low birth weight. In the under 15 age category, 13.6 percent of births were classified as low birth weight. Pregnant teenagers 15-19 years had a 9.9 percent low birth weight average compared to women aged 25-29 who had a 7.3 percent low birth weight average. In addition, teenagers and young adults account for nearly half the cases of sexually transmitted diseases in the United States though they make up just a quarter of the sexually active population.

Evidence: (1) The Guttmacher Institute "U.S. Teenage Pregnancy Statistics National and State Trends and Trends by Race and Ethnicity" (2006). (2) Henshaw, S.K. "Disparities in Rates of Unintended Pregnancy In the United States, 1994 and 2001" (2006). (3) The National Campaign to Prevent Teen Pregnancy "By the Numbers: The Public Costs of Adolescent Childbearing" (2006). (4) Child Trends "Facts at a Glance" (2005).(5) National Center for Health Statistics " Births: Preliminary data for 2006" (2007)and "Births: Final data for 2004" (2006).

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: Though there are no other Federal or state programs similar to the AFL's care demonstration or research programs, the prevention demonstrations are duplicative of ACF's Community Based Abstinence Education-CBAE grants. Since 1997, Congress has set aside approximately $13 million of the Title XX appropriation for abstinence education prevention demonstration grants. Both AFL and CBAE programs provide competitively awarded grants to fund abstinence education for adolescents as defined by Section 510(b)(2) in Title V of the Social Security Act. While secondary components of the programs differ in that AFL requires grantee matching funds and CBAE does not and AFL requires that 20%-25% of the grant amount fund evaluation activities compared to 15% of CBAE programs, these are differences in process, not purpose.

Evidence: (1) Section 510(b)(2) in Title V of the Social Security Act; (2) The FY 2007 Program Announcement for the CBAE Program (HHS-2007-ACF-ACYE-AE-0099).

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The program design is efficient and effective and there is no evidence that another approach or mechanism would be better for the intended purpose. The program accomplishes its goals primarily through competitive grants to public or private nonprofit organizations or agencies to fund demonstration projects. The flexibility of the demonstration program structure enables the AFL program to respond to emerging and changing conditions in the field. The current program structure works in tandem with HHS partner agencies, such as the Office of Grants Management (OGM), the Office of Public Health and Science (OPHS).

Evidence: Legislation and grant announcements in addition to Office of Population Affairs (OPA) grant processes including application, competitive award process, site visit template form, and online grants management through GrantSolutions.gov demonstrate an effective method to efficiently administer these grants. Specific evidence include: (1) Title XX of the Public Health Service Act (P.L. 97-35)(42 U.S. C. 300z) established the Adolescent Family Life program demonstration grants "to encourage and support research projects and demonstration projects concerning the societal causes and consequences of adolescent premarital sexual relations and adolescent childbearing". (2) Federal Register Vol. 72, No. 62 - pg. 15772 is an example of application requirements, goals and objectives and award information for prevention grants. (3) Federal Register Vol. 71, No. 56 - pg. 14786 is an example of application requirements, goals and objectives and award information for care grants.

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: The statutory language for the AFL program requires that certain factors such as the incidence of adolescent pregnancy in the geographic area served, the adequacy of existing prevention and care services in the area, and the special needs of rural areas be considered by the Deputy Assistant Secretary for Population Affairs (DASPA) in making grant awards [PHSA, section 2005(b)(1)(2)]. All applicants must demonstrate how they will serve the appropriate adolescent target population. Once a grantee is funded, (1) program materials are reviewed by the Office of Adolescent Pregnancy Programs (OAPP) to ensure they are appropriate for the respective target population; (2) training/conference workshops are provided for AFL grantees to present how to communicate and effectively reach the target population and involve families, management, staffing, etc; (3) annual reports document the number and type of clients served; and (4) assignment of a single Federal Project Officer is made to ensure resources are utilized appropriately and to assess whether program activities are meeting the intended purpose and targeted population specified in the legislation. Technical assistance is also provided to grantees to ensure the program requirements are met, ensure the efficient operation of grant funded programs, and to assist with evaluation efforts.

Evidence: Grantees applying for both demonstration and care projects must complete a Technical Merits and Review Criteria form which requires applicants to demonstrate the need for the project and a detailed action plan for targeting the appropriate audience. In making final funding decisions, AFL program staff and the DASPA utilize a list of current grantees, lists of other program office grant locations (e.g., ACF) and a map to further ensure the balance of special needs for rural areas and under-served areas. Once grants have been administered, AFL makes sure resources continue to be effectively directed to its desired beneficiaries by providing grantees training in areas such as "engaging youth," "understanding and using outside teen influencers," and "recruiting and retaining schools" (2008 OAPP Technical Assistance and Training Calendar and Agenda). Prevention demonstration projects, approximately 44 % or $13.12 million of funds, provide and evaluate innovative abstinence education activities and services targeting youth aged 18 and younger and their parents. The primary purpose is to reach pre-adolescent and adolescent youth before they become sexually active to encourage abstinence from premarital sexual activity. Care program services, which comprise approximately 36 % or $10.99 million of funds, target pregnant and parenting adolescents under age 19 at program entry, with an emphasis on unmarried adolescents who are 17 years old or younger. Care services also promote involvement of the adolescents' children, extended family members, and fathers of their children.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The program has a limited number of long-term performance measures, which are focused on outcomes and reflective of the purpose of the program. The purpose of the program is to prevent teen pregnancy and sexually transmitted diseases in youth and mitigate the negative consequences associated with being a pregnant and parenting teen. The desired long-term outcomes would be to decrease early sexual activity by increasing abstinence knowledge and attitudes and decrease the negative consequences of too-early-childbearing by improving the individual outcomes of teen parents as dictated in section 2001(b) of the Title XX statute.

Evidence: AFL has six long-term performance measures. Two of the performance measures directly relate to prevention demonstration projects: (1) "Increase communication among parents and adolescents on topics relating to puberty, pregnancy, abstinence, alcohol, and/or drug" and (2) Increase adolescents' understanding of the positive health and emotional benefits of abstaining from premarital sexual activity. AFL has three measures directly relate to care projects: (1) "Maintain the incidence of clients in AFL Care demonstration projects who do not have a repeat pregnancy," (2) "Increase infant immunization among clients in AFL Care demonstration projects," (3)"Increase the educational attainment of clients in AFL Care demonstration projects." Finally, AFL measures the caliber of evaluations for both care and prevention projects: "Improve the quality of the Title XX independent evaluations."

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: AFL has set ambitious, five year timeframes to achieve the ambitious targets for its long term/annual measures. The program considered (1) current literature available in the field (2) current and historic grantee data and (3) the demonstration nature of the program to inform its target setting decisions. For example, AFL used an academic study, "Home Visiting for Adolescent Mothers: Effects on Parenting, Maternal Life Course, and Primary Care Linkage," in determining targets for its measurement to increase client educational attainment. The literature shows that at a two year follow-up, 44% of teens in a control group were in school or graduated, compared to 71% of those who had participated in a proven intervention program. AFL has set targets to exceed the study's result and improve their performance from the FY 2007 baseline measure of 68%. When establishing targets to increase adolescent and parent communication on topics such as pregnancy, abstinence, and alcohol/drugs, AFL reviewed past and current grantee data to set the target of 55%, an increase from the FY 2006 baseline measure 44%. AFL's care measure which seeks to maintain the number of parenting teens who do not have a repeat pregnancy is ambitious when compared with data in the field. A 2006 conducted by Maureen M. Black, et. al, "Delaying Second Births Among Adolescent Mothers: A Randomized, Controlled Trial of a Home-Based Mentoring Program," found that 89% of parenting teens did not experience a repeat pregnancy at a two year follow up, while AFL achieved a 92% result in its baseline year.

Evidence: (1) The long-term target for prevention measure (a) to increase communication among parents and adolescents on topics relating to puberty, pregnancy, abstinence, alcohol, and/or drugs is 55% by FY 2012, an increase from the FY 2006 baseline of 44%. The long-term target for measure (b) to increase adolescent's understanding of the positive health and emotional benefits of abstaining from premarital sexual activity is 92% by FY 2012; an ambitious increase from the FY 2007 actual of 54%. The long-term target for measure (c) to maintain the incidence of clients in AFL care demonstration projects who do not have a repeat pregnancy at the FY 2007 baseline level of 92% through FY 2012. The long-term target for measure (d) to increase infant immunization among clients in AFL Care demonstrations is 86% by FY 2012; an increase from the FY 2007 baseline of 76%. The long-term target for measure (e) to increase the educational attainment of clients in AFL Care demonstration projects is 78% by 2012; an increase from the 68% baseline from FY 2007. (2) "Home Visiting for Adolescent Mothers: Effects on Parenting, Maternal Life Course, and Primary Care Linkage", Barnet, et. al. 2007. (3) "Home Visiting for Adolescent Mothers: Effects on Parenting, Maternal Life Course, and Primary Care Linkage", Black, et.al. 2006.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The long term measures are also the program's annual measures, serving as milestones in achieving the long term goals.

Evidence: AFL has six long-term and annual performance measures. Two of the performance measures directly relate to prevention demonstration projects: (1) "Increase communication among parents and adolescents on topics relating to puberty, pregnancy, abstinence, alcohol, and/or drug" and (2) Increase adolescents' understanding of the positive health and emotional benefits of abstaining from premarital sexual activity. AFL has three measures directly related to care projects: (1) "Maintain the incidence of clients in AFL Care demonstration projects who do not have a repeat pregnancy," (2) "Increase infant immunization among clients in AFL Care demonstration projects," (3)"Increase the educational attainment of clients in AFL Care demonstration projects." Finally, AFL measures the caliber of evaluations for both care and prevention projects: "Improve the quality of the Title XX independent evaluations."

Does the program have baselines and ambitious targets for its annual measures?

Explanation: As 2.2 explained, AFL has set ambitious long-term/annual targets. The long-term goals are measured annually by ambitious milestones. In setting its annual targets AFL considered past and current data in setting ambitious, yet achievable targets. For example, AFL set a long term target of increasing the percent of AFL Care demonstration clients who complete high school or a GED program to 78%. Annual targets that increase by two percent each year were established to reach the long term outcome, which exceeds results from literature in the field and is 10 percentage points above AFL's 2007 baseline of 68%. Other AFL annual measures increase in a similar incremental manner, except for the care measure on repeat pregnancies, which aims to maintain the 92% achieved in the 2007 baseline year that exceeds the national average.

Evidence: Each of the long-term measures has annual milestones to achieve. The FY 2008 targets for the the long-term measures are as follows: (a) to increase communication among parents and adolescents on topics relating to puberty, pregnancy, abstinence, alcohol, and/or drugs to 48.8%; (b) to increase adolescent's understanding of the positive health and emotional benefits of abstaining from premarital sexual activity to 68%; (c) to maintain the incidence of clients in AFL care demonstration projects who do not have a repeat pregnancy at 92%; (d) to increase infant immunization among clients in AFL Care demonstrations to 78%; (e) to increase the educational attainment of clients in AFL Care demonstration projects 70%.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: AFL program partners, their grantees, are committed to working toward the annual/long-term goals. One way in which AFL ensures commitment is through the use of the Core Instrument to collect performance information from grantees. Grantees are required by a grant condition to submit a comprehensive end of year report each year, 90 days after the end of their budget year. Enclosure B of this report is the standard way grantees present the results from the AFL Core Instrument to the program office. This information requires grantee to discuss the progress made toward achieving the performance measures.

Evidence: Evidence of grantee comittment to the performance measures include: (1) Request for Application - FY 2006 Care RFA and FY 2007 Prevention RFA; (2) End of Year Report Template; (3) Example End of Year Feedback letter; (4) Enclosure B Templates - Care and Prevention.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: The contract for the cross-site evaluation is scheduled to be awarded on June 3, 2008. (If the contract is not awarded by this date, the answer will be changed to NO.) AFL is in the early stages of planning a cross-site evaluation (the measurement of a common set of variables using a common assessment instrument administered in a standardized manner at multiple grantee sites) of AFL care and prevention demonstration projects. The purpose of the evaluation will be to both describe its implementation and to evaluate its impact on desired outcomes. AFL expects to compete a contract for the evaluation in the spring of 2008 with an estimated completion date of 2010. In October 2006, the US Government Accountability Office (GAO) conducted an independent evaluation of HHS abstinence education programs, including AFL, to assess the accuracy and effectiveness of these programs. However, the scope of this evaluation was limited to the $13 million of AFL program dollars that are set-aside for prevention grants.

Evidence: (1) AFL Cross-Site Evaluation Statement of Work; (2) "Abstinence Education: Efforts to Assess the Accuracy and Effectiveness of Federally Funded Programs", GAO 07-87.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: AFL does not have program budgeting in place that defines the relationship between long-term/annual measure and resources. The impact of funding on program performance is not included in the FY 2009 Congressional Justification. Finally, the program does not discuss direct and indirect costs associated with achieving its performance goals.

Evidence: FY 2009 Congressional Justification.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: In FY 2004, during the initial PART process, the Office of Adolescent Pregnancy Programs (OAPP) developed six long-term/annual performance measures and one efficiency measure. The development and adoption of these performance measures kick-started a larger strategic planning discussion for the AFL program. In FY 2006, OAPP conducted a a three-day strategic planning session to review internal strengths and weaknesses. As a result of those strategic planning efforts, the OAPP mission, vision, and philosophy of practice statements were created. These have been published as part of various AFL print materials for grantees and other OAPP partners and are accessible to the public on the AFL website.

Evidence: (1) PART Long-term/Annual and Efficiency Performance Measures; (2) April 10, 2006 Strategic Planning Meeting Materials; (3) The OAPP mission, vision, and philosophy of practices can be found on the AFL website: http://www.hhs.gov/opa/familylife/strategicplanning/strategicplanning.html.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: AFL monitors grantees and contracts during the year. For grants, AFL collects information annually on performance as prescribed in the notice of grant award (NGA) agreements through continuation applications for future funding and end of the year reports. Assessment of this performance information is synthesized and used to: provide recommendations to grantees, update the program improvement plan, and develop annual technical assistance workshops and orientation conferences. Grantees and contractors are required to respond to any AFL performance concerns with a written corrective action plan. OAPP staff monitors corrective actions through verbal and written communication on a bi-weekly basis. If deemed necessary by the Deputy Assistant Secretary for Population Affairs (DASPA), a site visit is conducted. Contractors, who assist OAPP with training, technical assistance, and evaluation efforts, are required to provide monthly detailed invoices, progress reports, and meet with OAPP on a quarterly basis. Weekly correspondence with contractors also helps OAPP communicate key information.

Evidence: One OAPP staff member is assigned to each grant/contract to ensure the timely submission of credible reports and invoices. Required end of year reports, which are provided four months prior to non-competed applications, help OAPP assess grantee performance and provide written feedback. Site visits are conducted when programmatic, evaluative and/or grants management issues are identified and result in a time-lined corrective action plan. If a grantee is not compliant with grant terms and conditions as stipulated in the NGA or does not implement a corrective action plan, funding can be terminated. Performance issues stemming from grantee reports also guide OAPP in planning technical assistance workshops and conference planning. For instance, OAPP Project Officers observed similar issues with grantee evaluations (i.e., parental consent, data collection problems, power analysis). In conjunction with the evaluation contractor, a technical assistance workshop addressing evaluation was planned and will be implemented May 2008 for AFL grantees. Contractors are held accountable through the contract award, for example, the contract with John Snowe, Inc requires monthly progress reports be submitted to the program office. In addition, contracts are structured so that the vendor is paid by deliverables and if a deliverable is not satisfactory to the OAPP Project Officer, the contract can be terminated.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: Grantees are held accountable for cost, schedule, and performance results through monitoring visits, conferences, technical assistance workshops, and required annual performance reports. Information obtained is used to determine continued funding, need for additional technical assistance, and any need for performance/delivery changes. Assessment of this information is also used for corrective action and continued grant funding is contingent upon demonstrating satisfactory progress. Contractors are held accountable to Federal Acquisitions Regulations and terms of their contract. All OAPP employees have performance management appraisal plans in place to evaluate job performance. Weekly and bi-weekly meetings are conducted with individual staff to continually assess performance and provide feed-back.

Evidence: Federal managers are held accountable through individual performance plans that tie program performance to employee performance (both a federal manager's and the Acting Deputy Assistant Secretary for Population Affairs' performance plans were submitted as evidence). OPHS has assigned a grants specialist and Project Officer for each grantee to review associated budget requests, justifications, and annual performance reports. If a grantee is not able to handle a fiscal matter appropriately, they will be placed on "special status' and will not be able to draw down funds without prior approval. Grantees are required to file a Financial Status Report, Standard Form 269, within 90 days of budget year end to account for all program expenditures. If a grantee is not able to handle a programmatic matter appropriately, the Project Officer in conjunction with the Office of Grants Management develops an action plan for grantee improvement and technical assistance is provided. If a grantee cannot correct either fiscal or programmatic issues as requested, continued funding is jeopardized. Contracts are awarded through the acquisitions process and per Federal Acquisitions Regulations which are stipulated, along with contract specific conditions, in the contract language. Contractors are held accountable through a delivery schedule, whereby payment is not made until the contractor performs. If corrective action is needed but not accomplished, the contract will be terminated.

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: AFL funds are obligated according to the fiscal year and in accordance with statutory requirements and the standards and guidelines of the Department's accounting system, OMB Circulars, Federal Acquisitions Regulations, and Chief Financial Officers Act.

Evidence: Grantees report on planned and actual expenditures. Grantees provide a cash transaction report indicating the drawdown of funds and balances on a quarterly basis. Grantees are required to produce a Financial Status Report (FSR) and reconcile OMB Circular A-133 audits required under Federal law with the FSR. Financial statements show minimal unobligated balances. Using these tools, OAPP and OGM monitor grantee expenditures to ensure compliance with legislation, regulation and policies. All AFL Federal funds are obligated by September 30th of each year, at the latest. The AFL spending plan is reviewed and approved by the Assistant Secretary for Health (ASH) to ensure that funds are spent for the intended purposes and in a timely manner. The OPHS executive office runs monthly accounting reports and discusses them with the Director.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: During the competitive grant and contract award process, AFL reviews closely the relationship between grant/contract amount, services to be provided, and timeline to ensure an efficient and cost effective grant or contract. The grantee's end of the year report, application for continued funding, and periodic site visits are used to ensure the services proposed in the grant award process are being provided. In FY 2006, AFL adopted an efficiency measure that relates directly to the cost of care and prevention demonstration projects. The aim was to maintain the cost per encounter ratio for Title XX demonstration projects at the FY 2006 baseline level of $37 for prevention projects and $125 for care projects. By maintaining the cost in subsequent years, AFL grantees would be achieving efficiencies at least equal to the inflationary costs of delivering services to their clients.

Evidence: During the application process, grantees and contractors must provide a thorough budget request and justification for the coming year which is reviewed by OAPP and OGM staff or Contracts Specialists. All grantees are required to submit a Financial Status Report (FSR) and an end of year report annually. These are used to review progress from the past year and determine if the money spent was commensurate with the activities provided. All contractors are required to submit thorough budget narratives and justifications annually when requesting option years of a contract. OAPP moved to a new on-line system for grants management in FY2007: www.grantsolutions.gov, that automates the full grants business-cycle including: electronic submission of grant applications; web-based forms for online financial data entry; direct file upload of budget narratives and justifications; and direct electronic interface to HHS payment and grants reporting systems. Receipt of contract invoices are processed through the Unified Financial Management System and OAPP staff utilize this process to ensure prompt payment. The AFL program developed an efficiency measure in FY2004 to sustain the cost to encounter ratio for all prevention and care demonstration projects. Since FY2006, all grantees are required to report on the AFL efficiency measure using Enclosure C from the end of year report. The AFL program has demonstrated that it has achieved improved efficiency in the implementation of programs through its efficiency measure to sustain the cost to encounter ratio in its demonstration projects. In FY 2007, AFL exceeded the goal of maintaining costs for both care and demonstration projects and decreased expenditures to $29/$110 for prevention and care projects respectively. The out-year target is set to maintain the better than expected prevention and care costs.

Does the program collaborate and coordinate effectively with related programs?

Explanation: While the AFL program meets regularly with the Community Based Abstinence Education (CBAE) programs, the PART guidance is clear that meetings, discussion groups and task forces by themselves are not sufficient to constitute a "YES" for this question. Based on the documents provided, there is no clear indication that the two engage in meaningful collaboration on management and resource allocation. Furthermore, AFL was unable to provide evidence that they reach out to CBAE when developing grant announcements and/or determining grantees.

Evidence: AFL provided no evidence of meaningful collaboration with CBAE in developing joint grant announcements, performance goals, or referral systems.

Does the program use strong financial management practices?

Explanation: Both internal and external controls are in place to measure and achieve efficiencies and cost effectiveness in program execution. As of September 2007, a review of HHS balance sheets from independent auditors found no material weaknesses related to AFL.

Evidence: The AFL program staff and the OGM staff work in conjunction with one another on all grants management activities. Pre-award assessment of applications reviews an applicant's financial capacity to conduct a Federal project (past experience with grants, organizational charts, and letters of support). Applicants are required to thoroughly discuss proposed budget items, justify each item, and describe all financial match requirements, and provide monthly contract invoices, which are reviewed annually and periodically (i.e. carry-over funds requests). Close monitoring of programs by OAPP and OGM staff minimize the risk of improper payments by grantees. Utilizing grantsolutions.gov, OAPP staff is aware of grantee fiscal activities and must approve all supplements, requests for change in grantee staff, and budget revisions. When fiscal issues arise, site visits are conducted in conjunction with Grants Management staff. Both the OAPP Project Officer assigned to the contract as well as the assigned Contracting Officer review the invoices and ensure deliverables have been received. Receipt of contract invoices are processed through the Unified Financial Management System and OAPP staff utilize this process to ensure prompt payment. The Department of Health and Human Services' fiscal year (FY) 2007 Agency Financial Report did not declare AFL an element of the "Top Management and Performance Challenges Identified by the Inspector General" section or a component of one of the audit's weaknesses.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: AFL identified the need to improve the independent evaluations of grantees. Prior to FY 2007 there was no method to assess the quality of each grant evaluation and to provide detailed feedback for improvement purposes. To address this shortcoming, AFL contracted the Research Triangle Institute (RTI) to develop a scoring tool that assesses each evaluation on an annual basis. This tool allows the contractor to conduct a uniform assessment for all grantee evaluations. The results of the uniform assessment enables AFL to identify areas of strengths and weaknesses for each grantee and provide individualized feedback and technical assistance as needed. In FY 2008, AFL plans to expand the use of the scoring tool to grant applicants, to ensure that when grants are awarded the grantee has a "promising" evaluation design. By incorporating a standard assessment of evaluations that will be used from initial application through the full grant period, AFL can work towards eliminating poor evaluation design among grantees.

Evidence: In 2007, AFL implemented the use of a standardized assessment tool developed by AFL's contractor, RTI. There are nine criteria on this tool that include: process evaluation, research hypotheses, comparison strategy, sampling strategy, outcome measures, data collection, data analysis, attrition, and missing data. Each of these items is given a score to determine the quality of the evaluation. Evaluations that receive a score of 3.3 (on a scale of 5.0) are considered "promising."

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: All grants are awarded competitively. The process includes posting a Request for Applications (RFA) and an application kit on grants.gov for the public; providing technical assistance via the internet for interested applicants; and implementing an application review process that includes both external and internal reviewers. Demonstration grants are awarded for a maximum term not to exceed 5 years.

Evidence: Section 2006(a) of the Title XX legislation mandates the AFL program award grants after carefully and strategically reviewing all applications by a set of specific criteria. All applicants can access guidance through a comprehensive application kit, an internet conference training, and access to OAPP staff. Applications are reviewed and scored by an independent panel of external experts. The grant review criteria used by the external panel are clearly stated in all Requests for Application (RFAs) announcing national competitions for new grants. Final funding decisions are made by the Deputy Assistant Secretary for Population Affairs (DASPA) taking into account several criteria, including geographic area, incidence of adolescent pregnancy, recommendations by objective review panel, and recommendations by AFL program staff. Demonstration grants are awarded for a finite period of time, at the end of which an organization must re-compete for any subsequent funds. Funded grantees must reapply through a non-competing application process each year in order to receive funds. AFL staff review every continuation application to ensure that the program is making progress and is using the funds appropriately.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: An AFL Project Officer is assigned to each grantee. The Project Officer monitors assigned grantees through periodic site visits, technical assistance trainings and conferences, and frequent communication both verbally and in writing. In addition, each grantee is assigned a Grants Management Specialist who reviews all budgetary activities and participates in selected site visits. Grant activities are monitored as part of the annual continuation application process.

Evidence: Oversight is documented in feed-back letters to grantees in response to annual end of the year and continuation application reports. AFL staff review these reports and develop an internal report which is carefully reviewed and discussed with the OAPP Director prior to feed-back being provided. OGM staff carefully review all budgetary items that are described and justified in these reports and provides feed-back by letter and/or email. Grantees are also required to participate in an annual conference of AFL grantees. During these conferences, AFL Project Officers utilize the time to meet with grantees and provide feed-back on program progress. Annual technical assistance meetings are not required but most grantees participate and report on grantee activities. For example, during a recent parental involvement workshop, a grantee described the fact they were not able to fully obtain required parental consent for clients in their program and asked for advice from colleagues. OAPP staff was alerted and the OAPP Project Officer was able to intervene and provide direct assistance on methods to comply with this requirement. An AFL listserv for grantees disseminates information from the AFL office and allows for communication between grantees.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: AFL does not make grantee performance data accessible to the public. The only forum that presents grantee progress is the annual conference that is open only to care and prevention grantees, speakers and invited guests.

Evidence: Grantee performance information beyond what is available on ExpectMore.gov is not accessible to the public.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: There is minimal progress being made towards achieving long-term/annual performance measures. Of AFL's six performance measures, only three have comparative data available in FY 2007, the others were baselined in FY 2007. AFL has shown improvement in the prevention portion of one of the three long-term measures with comparative data. Between FY 2006 and FY 2007 AFL improved the quality of Title XX independent evaluations in prevention programs, from 11% of grantees having evaluations that are characterized as "promising" to 22.2% exceeding the FY 2007 target of 19.25%. In FY 2007, child-parent communication decreased, resulting in falling short of the annual target. Data on adolescent attitudes about the benefits of abstinence have failed to show a positive trend.

Evidence: (1) The long-term/annual measure "improve the quality of Title XX independent evaluations" has both a prevention and care component. In FY 2006 the baseline for prevention evaluations was 11% and 42% for the care evaluations. The prevention portion exceeded the target for FY 2007 achieving 22%. The care portion of this measure experienced a slight decrease to 37% in FY 2007. To improve future performance, AFL has incorporated the scoring criteria for grantee evaluations into the FY 2008 announcement and will not fund an organization that does not propose a promising evaluation. (2)"Increase communication among parents and adolescents on topics relating to puberty, pregnancy, abstinence, alcohol and/or drugs" was baselined in FY 2006 at 44.4%. Since then AFL has focused efforts on providing grantees technical assistance related to evaluation. While the FY 2007 data point, 42%, is lower than the FY 2007 target of 46.6%, AFL believes the integrity of the data has improved. (3) "Increase adolescents' understanding of the positive health and emotional benefits of abstaining from premarital sexual activity" was baselined in FY 2006 at 80% or 8 of 10 grantees. In FY 2007 only 54% or 7 of 13 grantees showed positive results related to this measure. AFL is revisiting the core question and data collection methods used to inform this measure.

Does the program (including program partners) achieve its annual performance goals?

Explanation: There is minimal progress being made towards achieving long-term/annual performance measures. Of AFL's six performance measures, only three have comparative data available in FY 2007, the others were baselined in FY 2007. AFL has shown improvement in the prevention portion of one of the three long-term measures with comparative data. Between FY 2006 and FY 2007 AFL improved the quality of Title XX independent evaluations in prevention programs, from 11% of grantees having evaluations that are characterized as "promising" to 22.2% exceeding the FY 2007 target of 19.25%. In FY 2007, child-parent communication decreased, resulting in fell short of the annual target. Data on adolescent attitudes about the benefits of abstinence have failed to show a positive trend.

Evidence: (1) The long-term/annual measure "improve the quality of Title XX independent evaluations" has both a prevention and care component. In FY 2006 the baseline for prevention evaluations was 11% and 42% for the care evaluations. The prevention portion exceeded the target for FY 2007 achieving 22%. The care portion of this measure experienced a slight decrease to 37% in FY 2007. To improve future performance, AFL has incorporated the scoring criteria for grantee evaluations into the FY 2008 announcement and will not fund an organization that does not propose a promising evaluation. (2)"Increase communication among parents and adolescents on topics relating to puberty, pregnancy, abstinence, alcohol and/or drugs" was baselined in FY 2006 at 44.4%. Since then AFL has focused efforts on providing grantees technical assistance related to evaluation. While the FY 2007 data point, 42%, is lower than the FY 2007 target of 46.6%, AFL believes the integrity of the data has improved. (3) "Increase adolescents' understanding of the positive health and emotional benefits of abstaining from premarital sexual activity" was baselined in FY 2006 at 80% or 8 of 10 grantees. In FY 2007 only 54% or 7 of 13 grantees showed positive results related to this measure. AFL is revisiting the core question used to inform this measure for future data collection.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The AFL program has achieved improved efficiency in achieving program goals. AFL's efficiency measure to sustain the cost to encounter ratio in its demonstration projects exceeded the FY 2007 target by 22% for prevention projects and 12% for care projects. In addition, OAPP has achieved efficiencies in other areas such as streamlined grants management, with the adoption of GrantSolutions.gov.

Evidence: (1) AFL's efficiency measure is "To sustain the cost to encounter ratio in Title XX prevention and care demonstration projects". In FY 2006, the baseline cost for these projects were $37 for prevention projects and $125 for care projects. The target for FY 2007 was to maintain the cost from the previous year, however AFL cut costs even further, with prevention projects costing $29 per encounter and care projects costing $110 per encounter. (2) The AFL program utilizes GrantSolutions.gov, a comprehensive grants management system provided by the ACF Grants Center of Excellence (COE), for receipt and processing of AFL grant applications. Grantsolutions.gov automates the full grants business-cycle including: electronic submission of grant applications; web-based forms for online data entry; direct file upload of project plans, budget narratives, and work plans; eliminates errors by re-using information entered on-line; current status of grant applications available online; direct electronic interface to HHS payment and grants reporting systems; full range of post-award reporting and activities are handled online; and on-line Financial Status Reports (FSR).

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: The program has not provided any performance data from other federal, private, or non-profit programs with similar purposes and goals that would allow for a comparison.

Evidence: No comparative performance data, evaluations, or documentation comparing similar programs was provided.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: While AFL is in the early stages of planning a cross-site evaluation of demonstration projects, no independent evaluations of sufficient scope and quality have been completed.

Evidence: No independent evaluations of sufficient scope and quality have been completed.