Detailed Information on the
Mentoring Children of Prisoners Assessment

Evaluate quality of relationships to create knowledge to drive better results. Milestone: Conduct third annual survey of MCP children to measure the quality of mentor relationships and report performance measure (percentage of positive relationships) through FY 2010 Performance Budget.

Evaluate long term program outcomes to create knowledge to drive better results. Milestone: Collect follow-up child data and survey agency service characteristics.

Evaluate long term program outcomes of the Caregiver's Choice program to create knowledge to drive better results. Milestone: Finalize research design and develop data collection tools for evaluation of outcomes in Caregiver's Choice.

Recalibrate targets and timeframes to reflect feasible levels of mentoring matches that can be achieved. Milestone: Improve performance goals for all new FY 2006 grantees.

Assess and improve grantees' results through performance analysis and targeted technical assistance. Milestone: Hold conference for all new grantees (awarded in FY06) as well as those demonstrated through data analysis to be underperforming.

Evaluate quality of relationships and long term program outcomes to create knowledge to drive better results. Milestone: Conduct first annual survey of Mentoring Children of Prisoners (MCP) children to measure the quality of mentor relationships.

Assess and improve grantees' results through performance analysis and targeted technical assistance. Milestone: Hold six regional conferences to further the T/TA efforts of the contractor.

Evaluate quality of relationships and long term program outcomes to create knowledge to drive better results. Milestone: Conduct second annual survey of MCP children to measure the quality of mentor relationships.

Evaluate long term program outcomes to create knowledge to drive better results. Milestone: Conduct first site visits for operational observations and analysis.

Develop data collection and finalize evaluation design for statutory Service Delivery Demonstration Project to reach families not previously served through an MCP program. Milestone: Submit evaluation and data collection instruments to OMB for approval.

Evaluate long term program outcomes to create knowledge to drive better results. Milestone: Collect baseline child data.

Develop data collection and finalize evaluation design for statutory Service Delivery Demonstration Project to reach families not previously served through an MCP program. Milestone: Obtain OMB approval for data collection, survey instruments and research design.

Measure: Relative stabilization or improvement of intermediate outcomes among a sample of children of prisoners as measured after at least 12 months in one-to-one mentoring relationships in the MCP program.

Explanation:Tim Baker - 110805 - Measure and Data updated "Positive life changes In the impact study of Big Brothers/Big Sisters by Public/Private Ventures, after 18 months, mentored youth were 46% less likely to begin using illegal drugs and 27% less likely to begin using alcohol as compared with youth in a control group. Little Brothers and Little Sisters also skipped half as many days of school as did control youth. (It should be noted that mentees in MCP encompass a broader age range and are a higher risk population than the youth participants in the BB/BS impact study.) Like the BB/BS impact study, FYSB intends to use a control group so that youth outcomes can more reliably be attributed to program impacts. Because all youth are exposed to more risk factors as they age, a control group is needed to determine whether the program is affecting those risk behaviors. For example, the BB/BS study showed that if mentees experienced mentoring for enough time (6- 12 months at minimum), they tended to pause or slow down in their accumulation of risks (rather than make behavioral gains), compared with controls. FYSB will also use a control group so that changes in youth risk behaviors are correctly interpreted in comparison to a comparable peer group. "

Measure: By FY 2014, increase by two percent over the previous year's result the percent of children of prisoners receiving mentoring through Mentoring Children of Prisoners (MCP) who will be or will have been in relationships lasting at least one year.

Explanation:Tim Baker - 110805 - Measure and Data updated "Duration of relationships so as to meet or exceed the 12-month standard Rationale: This is a broader measure than the long term, "sustainability" measure, because it includes the group that has reached the twelve month commitment, as well as those going beyond it. It can be reported annually, since the majority of its cases will probably last only twelve months. Under MCP, twelve month relationships may properly come to an end but are not required to do so. Research shows that it takes at least 6-12 months in a mentoring relationship before youth demonstrate positive outcomes. The greatest benefits are associated with mentoring relationships that last more than 12 months (Grossman & Rhodes, 2001). Accordingly, this key indicator of relationship health will be reported in for (a) all relationships meeting the duration standard and (b) relationships that have truly withstood the test of time, enduring thirteen months and beyond. The mutual commitment by both parties in the (b) group suggests a strong bond has formed. "

Measure: Increase the proportion of youth that consider their mentoring relationship to be of "high quality."

Explanation:Tim Baker - 110805 - Measure and Data updated "Quality of Relationships A positive assessment of a mentoring experience is strongly associated with the duration of the relationship and positive youth outcomes. We feel that a clear majority of cases (55 percent) should meet this standard already and have set the FY 2006 baseline accordingly, in effect making this a first-year target as well as a baseline. 60 percent is the long term (FY 2007) target for the measures of "Duration". Because of the correlation, this "Quality of Relationship" measure also targets 60 percent. "

Measure: Increase the number of children of prisoners in one-to-one matches with caring adults who have been trained and screened by the MCP program.

Explanation:Tim Baker - 110805 - Measure and Data No longer in use "Companionship with caring adults NOTE that annual targets are incremental (and linear to conform with budgetary methodology) and the President's long-term target (FY 2007) is cumulative and based upon FYSB's efforts and those of its partners, where data is available. (See long term measure above.)"

Measure: Increase the proportion of MCP matches that endure beyond three months.

Explanation:Tim Baker - 110805 - Measure and Data updated "Minimize matches of very short duration Matches which end prematurely represent a significant investment loss, because costs are associated to a large degree with outreach, recruiting, screening, training and preparing mentors before the initiation of matches. Premature cessations are also a programmatic liability, since a child's self esteem can be impacted negatively if he or she loses trust or feels abandoned. In the MCP program, mentors are expected to commit to at least twelve month relationships. "

Measure: Achieve the proportion of mentees in active mentoring relationships that have already lasted more than twelve months as a percentage of the entire caseload at 40 percent by FY 2014.

Explanation:Tim Baker - 110805 - Measure and Data updated "Sustainability of relationships (beyond the minimum commitment) Relationships that endure beyond the MCP's 12-month minimum are evidence of a lasting bond and possibly a life-long relationship, which is not uncommon among successful mentoring relationships in general. Relationships that last 12 months or more are associated with the most positive youth benefits."

Is the program purpose clear?

Explanation: The mission of the Mentoring Children of Prisoners program (MCP) is to establish and support sustained and high quality mentoring relationships between carefully screened and trained adult volunteers and children who are in high risk situations because one or both of their parents is in prison. These mentoring relationships are created with the permission of one or both parents or legal guardians. By providing such relationships, the program strives to improve academic and behavioral outcomes among children in the program compared to a control group. The Federal resources and oversight of the Family and Youth Services Bureau (FYSB) in the Administration for Children and Families (ACF) support a network of partners committed to these goals in areas that have significant numbers of children of prisoners, including State and local governments, tribal governments or tribal consortia, faith-based and community-based organizations. These organizations, many of them established specifically for this mission, identify the children, recruit, screen and prepare mentors and develop, manage, and nurture the relationships.

Evidence: "Promoting Safe and Stable Families Amendments of 2001. Subpart 2 of part B of title IV (42 U.S.C. 629-629e) SEC. 439. GRANTS FOR PROGRAMS FOR MENTORING CHILDREN OF PRISONERS. (a) FINDINGS AND PURPOSE.?? (2) PURPOSE.?? (Evidence binder Section 1.1) Funding Opportunity Number HHS-2004-ACF-ACYF-CU-0001 at www.acf.hhs.gov/grants/open/HHS-2004-ACF-ACYF-CU-0001.html and NOTICE OF CORRECTION FOR THE MENTORING CHILDREN OF PRISONERS PROGRAM ANNOUNCEMENT at www.acf.hhs.gov/grants/open/Corrections_to_Mentoring.html "

Does the program address a specific and existing problem, interest, or need?

Explanation: It is estimated that two million children in the United States have an incarcerated parent (Federal Resource Center for Children of Prisoners). Seventy percent of these children will become involved in the criminal justice system, which is approximately six times the average rate among other American children (U.S. Senate Report 106-404). Parental arrest and confinement often lead to stress, trauma, stigmatization, and separation problems for the children. These effects may be compounded by other conditions such as poverty, substance abuse, and high-crime environments. Lacking the support of families, schools, and other community institutions, such children often do not develop values and social skills leading to the formation of successful relationships. Research has shown that mentoring is a promising intervention for reducing risk behaviors that can lead to future involvement in the criminal justice system. A landmark experimental study conducted by Public/Private Ventures compared mentored youth with a control group. The results demonstrated that mentored youth were less likely to begin using drugs and alcohol; less likely to hit someone; more likely to have improved their school attendance, academic and performance, and attitudes toward completing schoolwork; and more likely to have improved peer and family relationships (Tierney and Grossman, 1995). The White House Task Force for Disadvantaged Youth cites mentoring programs as valuable interventions and children of prisoners as an example of youth most in need of targeted national attention.

Evidence: ?? Promoting Safe and Stable Families Amendments of 2001. Subpart 2 of part B of title IV (42 U.S.C. 629-629e) (a) FINDINGS AND PURPOSE.?? (1) FINDINGS. ?? Federal Resource Center for Children of Prisoners. What Happens to Children? Retrieved June 21, 2005, from Child Welfare League of America Web site: www.cwla.org//programs/incarcerated/cop_faq.htm ?? Senate Report 106-404: Departments of Commerce, Justice, and State, the Judiciary, and Related Agencies Appropriation Bill, 2001. September 8, 2000, p. 56. ?? U.S. Department of Justice. (2000). Incarcerated Parents and Their Children. (Bureau of Justice Statistics Special Report, August, 2000). Washington, DC: Christopher J. Mumola. Retrieved on June 21, 2005 at: www.ojp.usdoj.gov/bjs/pub/pdf/iptc.pdf. ?? Beier, S., Rosenfeld, W., Spitalny, K., Zansky, S., and Bontempo, A. (2000). The Potential Role of an Adult Mentor in Influencing High-Risk Behaviors in Adolescents. Archives of Pediatrics and Adolescent Medicine, 154, 327-331. Retrieved June 21, 2005 at archpedi.ama-assn.org/cgi/reprint/154/4/327.pdf ?? Tierney J. and Grossman, J (1995). Making a Difference: An Impact Study of Big Brothers Big Sisters, Philadelphia: Public/Private Ventures. Retrieved on June 21, 2005 at: www.ppv.org/ppv/publications/assets/111_publication.pdf ?? Rhodes, J. (2002). Stand by Me, The Risks and Rewards of Mentoring Today's Youth. Cambridge: Harvard University Press. ?? White House Task Force for Disadvantaged Youth. The Report to the President, (October, 2003,). Retrieved on June 21, 2005 at www.ncfy.com/whreport.htm. ?? Mentoring Children of Prisoners Bibliography (prepared by FYSB's National Clearinghouse on Families and Youth)

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: There is no other federal mentoring programs that specifically targets children of prisoners. This group has special needs and risk factors that are often more severe than other types of youth considered to be disadvantaged and thus served in youth programs. Children of prisoners and their families are often not reached by traditional mentoring programs because of their isolation, transient residency, severe challenges, and sense of stigma. MCP is designed to strengthen and multiply mentoring networks already in place according to documented need, as well as establish new ones where there are significant numbers of children of prisoners. Nationwide standards, oversight, technical assistance, and evaluation can effectively focus the impact of federal funding and reach thousands of families heretofore neglected. The Government Accountability Office (GAO) looked at Federal youth mentoring programs in 1998 and did not identify any mentoring programs for children of prisoners/incarcerated parents. Many programs serving disadvantaged youth with federal funding report that mentoring is, or may be, among the activities provided to participants, with a wide variety of target groups and service models. Non-Federal mentoring programs may focus on various categories of at-risk youth, but don't seek out children of prisoners, who may be very difficult to identify and serve. The exception is the Amachi program, which MCP is partly modeled upon and with which FYSB is collaborating. Amachi is a partnership of secular and faith-based organizations working together to provide mentoring to children of incarcerated parents. Faith institutions work with human service providers and public agencies (particularly justice institutions) to identify children of prisoners and match them with caring adults.

Evidence: The Government Accountability Office (GAO) looked at Federal youth mentoring programs in 1998. "Youth Mentoring Programs (GAO/HEHS-99-129R)" FY 2003 White House survey found over 120 federal youth programs that cited mentoring as an activity (among many) provided to participants in some part of their caseloads. However, less than 30 of these programs specifically included mentoring as an objective (for example, with a statutory or regulatory mandate). Most of these programs in fact provided very little mentoring as a specific or essential program activity. The White House survey covered a spectrum of programs broader than the GAO study. 339 had youth or children included in their target groups; about 120 of these alluded to mentoring as an activity. The survey's focus was more expansive than GAO's (which focused on mentoring only) and thus found a greater number of programs. The White House Task Force for Disadvantaged Youth concluded that youth programs have increased dramatically in number, but without sufficient targeting of resources and services. MCP "gives priority to the neediest youth" (a Task Force recommendation) by targeting new resources and services upon this population of exceptionally disadvantaged young people. Currently, the only other federal program that exclusively focuses on mentoring services is the Department of Education's Office of Safe and Drug Free Schools, which awarded 122 grants to school-based mentoring programs in FY2002, for a total of $50 million over three years. An additional $300 million became available in FY2004 to match 1 million disadvantaged students with adult mentors over a period of three years. There are a number of major non-federal programs that focus on mentoring youth in general, particularly those believed to be "at risk," including children of risoners. Big Brothers Big Sisters (BB/BS) is the oldest and largest youth mentoring organization in the United States, matching caring adults with isadvantaged children, ages five-to-18 years old, in one-to-one relationships. Approximately 500 Big Brothers Big Sisters agencies are located in 5,000 communities across the country. Over one third of the 218 current MCP grantees are affiliated with BB/BS. Additionally, many of these are utilizing the "Amachi" model, which was developed by Public/Private Ventures. Amachi began in 2000 as a demonstration project in Philadelphia and is now the first and only national non-federal mentoring program that specifically targets children of prisoners. As of March 31, 2005, there are 118 Amachi-affiliated programs nationwide; they have partnered with over 1000 churches and served more than 3900 children. Many of the children currently served by MCP are in Amachi-modeled programs.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: MCP utilizes practices that research has shown to be supportive of quality, one-on-one mentoring relationships that lead to positive youth outcomes. Research into the BB/BS program established the efficacy of practices such as screening, orientation, training, ongoing support and supervision of mentors and matches, all of which are mandatory under MCP. The importance of the twelve-month duration factor was confirmed by a reanalysis of the BB/BS evaluation which showed impact differences for youth mentored under 3 months; between 3-6 months; 6-12 months; and over 12 months. The greatest progress occurred as duration approached and passed 12 months. FYSB's standards for MCP are drawn from this research and require grantees to incorporate these programmatic elements and to require a 12-month commitment from mentors. The mentoring relationship, or the child-adult "match," is the key to this model; other youth development programs, such as after-school programs, while also good interventions for at-risk youth, do not always attract high-risk youth (particularly truants, dropouts, or youth lacking after-school transportation). Research indicates that after-school programs do not always result in close adult-youth relationships. (Herrera and Arbreton, 2003) Many mentoring programs are scholastic, athletic, or career-focused and site-based, while MCP does not focus exclusively upon a single dimension of youth development and stresses one-on-one activities based in the community, not primarily on school grounds or in the workplace. This holistic, less structured approach to relationship-building can facilitate close mentoring relationships. Alternative ways of delivering mentoring services are also under consideration. FYSB is planning a demonstration project to test an alternative model of administration and service delivery. Funding will continue for current grantees, while a pilot program would test vouchers and outsourcing methods in selected areas. A national organization under competitive contract would accredit participating providers and oversee the distribution of vouchers in several communities. The demonstration project will introduce elements of competition and parental choice and establish accreditation standards (which will also be of value to the original model for MCP and potentially to other mentoring programs). Evaluating the implementation and outcomes of these projects during the evaluation should reveal the relative effectiveness and efficiency of indirect funding and parental choice for MCP.

Evidence: ?? Research has shown that mentoring is a promising intervention for reducing risk behaviors. A landmark experimental study conducted by Public/Private Ventures (P/PV) showed that, compared with a control group, mentored youth were less likely to begin using drugs and alcohol; less likely to hit someone; more likely to have improved their school attendance and performance, and attitudes toward completing schoolwork; and more likely to have improved peer and family relationships (Tierney and Grossman, 1995). ?? Sipe, C. (1996). What program structures and supports are needed to maximize "best practices" among mentors? In Mentoring: A Synthesis of P/PV's Research: 1988-1995, (pp.9-11). Philadelphia: Public/Private Ventures. Retrieved on June 22, 2005 at: www.ppv.org/ppv/publications/assets/40_publication.pdf. ?? Grossman, J. B. and Johnson, A. Assessing the Effectiveness of Mentoring Programs, in Contemporary Issues in Mentoring. Philadelphia: Public/Private Ventures. Retrieved on June 22, 2005 at: www.ppv.org/ppv/publications/assets/37_publication.pdf. ?? U.S. Department of Health and Human Services/Administration for Children and Families/Administration on Children, Youth and Families/Family and Youth Services Bureau. (March, 2005). The Exchange, Pittsburgh: Mid Atlantic Network for Youth. ?? Herrera, C. and Arbreton A. (2003). Increasing Opportunities for Older Youth in After-School Programs. Philadelphia: Public/Private Ventures. Retrieved on June 29, 2005 at www.ppv.org/ppv/publications/assets/146_publication.pdf.

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: Only children between the ages of 4 and 15 with an incarcerated parental figure are eligible for MCP services, and these services are limited to mentoring and support of mentoring matches. The terms and conditions of the grant are explicit in these areas. Only children whose parents are in the custody of a state or federal prison are eligible to participate in MCP. Children whose parents are on probation or in a local jail are not eligible for the FYSB-funded portion of grantee services, since the parent is able to be present in the child's life. Children whose parents are in halfway houses or under supervision or house arrest are not eligible unless the detention is preparatory to a state or federal prison sentence. Parents and children must meet these criteria at the time the match is made; however, children may continue to receive services if their parent is released from prison during the mentoring relationship. FYSB staff further clarify these issues at grantee conferences. FYSB funds and matching funds cannot be used for "wrap-around" or other services to the child or family such as general case management. However, FYSB requires grantees to partner with organizations that can provide such services.

Evidence: To successfully compete for funding, grantees first had to establish that there was a significant concentration of children of incarcerated parents in their communities (who were not being mentored already). In documenting these facts and in implementing service delivery, grantees used several methods. Some grantees contacted surrounding school systems for student/parent information and/or the Bureau of Prisons to locate participants, while others collaborated with child social service programs such as the foster care system and/or their state prisons. Many of the faith-based community outreach programs already had well-established relationships from their prison ministry program(s) and recruited participants as part of their ministry work. After applications were reviewed, awards were made based on scoring, but adjusted to avoid duplication and redundancy because of the over-concentration of programs in the same general area. Demographic evidence was a factor in the review of applications in order to distribute services widely. High-scoring applicants who proposed to serve an area already served by an existing grantee were passed over in favor of slightly lower-scoring applicants who proposed to target new service areas. The amount of each grant was based partly on the number of children each grantee committed to serving. Funds were also set aside to assure the inclusion of five Tribal grantees. At the service delivery point, eligibility factors are simple to determine and verify. Correctional institutions and social services agencies refer or confirm eligible families. Parents and guardians provide documents (in some areas, notarized letters) confirming parental incarceration. ?? Definitions and clarifications on these issues were spelled out in Funding Opportunity Number: HHS-2004-ACF-ACYF-CU-0001 at www.acf.hhs.gov/grants/open/HHS-2004-ACF-ACYF-CU-0001.html and NOTICE OF CORRECTION FOR THE MENTORING CHILDREN OF PRISONERS PROGRAM ANNOUNCEMENT at www.acf.hhs.gov/grants/open/Corrections_to_Mentoring.html FYSB Documents: ?? Application Assurances ?? Terms and Conditions ?? FAQs on FY 2004 Program Announcement ?? Managing your Grant, Facilitator's Notes for Grants Management Workshop

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: Sustainability of relationships (beyond the minimum commitment) Mentees in active mentoring relationships that have lasted more than twelve months as a percent of the entire caseload Source: Online Data Collection Quality of relationships The percent of mentees surveyed that respond with an overall average score of 3 or above on 15 key questions in a mentoring relationship quality instrument Source: relationship measuring tool/survey instrument Companionship with caring adults Number of children of prisoners in one-to-one matches with caring adults who have been trained and screened by the MCP program and its local and national partners. Source: Online Data Collection and partner survey information Positive life changes Improved risk reduction and academic commitment in the MCP evaluation's experimental group as compared with a control group. 1) Behavioral - After one year in a mentoring relationship, mentored youth will be less likely to report initiating drug or alcohol use than control group youth. 2) Academic - Mentored youth will report skipping fewer days of school than control group youth. Source: FYSB national evaluation (FY 2006 - FY 2009)

Evidence: Sustainability of relationships (beyond the minimum commitment): Rationale: Relationships that endure beyond the MCP's 12-month minimum are evidence of a lasting bond and possibly a life-long relationship, which is not uncommon among successful mentoring relationships in general. Relationships that last 12 months or more are associated with the most positive youth benefits. Quality of relationships: Rationale: The core purpose of mentoring is healthy companionship between at-risk youth and caring adults. These relationships are mediators of successful outcomes (see Rhodes, Journal of Primary Prevention, in the Evidence Binder). A validated and reliable relationship measuring tool, developed by Rhodes, Reddy, Roffman, and Grossman, assesses the dynamics of the mentor/mentee relationships, including: mentee satisfaction with the relationships; the extent to which mentors have helped mentees cope with problems; how happy mentees feel (or don't feel) when they are with their mentors; and whether there is evidence of trust in the mentoring relationships. Companionship with caring adults: Rationale: This measure is based on the number of children of prisoners growing up with caring adult companions in relationships that conform to the evidence-based (one-to-one relationship) standard of the MCP. Forming and supporting these matches is the primary task of MCP grantees. Positive life changes: Rationale: In the impact study of Big Brothers/Big Sisters by Public/Private Ventures, after 18 months, mentored youth were 46% less likely to begin using illegal drugs and 27% less likely to begin using alcohol as compared with youth in a control group. Little Brothers and Little Sisters also skipped half as many days of school as did control youth. (It should be noted that mentees in MCP encompass a broader age range and are a higher risk population than the youth participants in the BB/BS impact study.) ?? MCP excerpt from FY 2006 Performance Plan (as revised) (FYSB document) ?? 1st and 2nd Quarter results for all data elements (FYSB document) ?? List of 15 questions from Stand by Me (Rhodes, 2002). ?? Rhodes J., Reddy, R., Roffman, J., and Grossman J.B. (March, 2005). Promoting Successful Youth Mentoring Relationships: A Preliminary Screening Questionnaire. The Journal of Primary Prevention, 26:2, 147-167. (This study discusses a larger set of questions including some not strongly correlated with quality, and which will not be used by FYSB.) ?? List of grantees and 2nd quarter caseloads (FYSB document) ?? The national evaluation (FY 2006 - FY 2008) (see Section 2.6) will utilize instruments, interviews, on-site observations and statistical sampling to determine long term academic and behavioral outcomes. This will be a full-scale evaluation and not repeated annually, but will be conducted as program development and resources warrant, very likely with modifications focusing upon more evolved programmatic factors. (Evidence Binder, Section 2.6)

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: For the two measures below, baselines are from different periods based on when reliable data was first available. Sustainability of relationships (beyond the minimum commitment) Mentees in active mentoring relationships that have lasted more than twelve months as a percent of the entire caseload. Levelling of this proportion would be expected if (1) new cases continue to be matched and (2) mentors for the most part conclude matches after the 12 month requirement even if they take on a new match with a different child. Companionship with caring adults Number of children of prisoners in one-to-one matches with caring adults who have been trained and screened by the MCP program and its local and national partners. A linear progression of 33,000 cases new each year was established for budgetary purposes based on a median cost per match of $1500. The increment itself is systematically calculated based on a survey of median costs to mentor a child. Actual performance is likely to be exponential during the earlier years. With time, the growth of the program should be more uniform from year to year as it reaches a steady state. The long term projections assume continued reauthorization and level appropriations. Quality of relationships The percent of mentees surveyed that respond with an overall average score of 3 or above on 15 key questions in a mentoring relationship quality instrument. Baseline: to be measured during FY 2006 upon information collection approval and annually thereafter if program appropriations continue.

Evidence: who have met their commitments (and may or may not seek a new mentee) and some mentees who are aging out of the program or moving away. The denominator will also be increased by the influx of new cases as the program expands, adding to the number of matches that are still on their way to twelve months. If the program continues to grow rapidly, this target is not likely to apply to a majority of all cases. Companionship with caring adults Rationale: This measure is based on the number of children of prisoners growing up with caring adult companions in relationships that conform to the evidence-based (one-to-one relationship) standard of the MCP. Forming and supporting these matches is the primary task of MCP grantees. Partnership building at the local level is mandated by the Statute. (See quote in NOTES tab.) FYSB is building national partnerships as well, including with the programs of the Corporation for National and Community Service (CNCS) and with non Federal partners such as Big Brothers/Big Sisters and the National Mentoring Partnership. (See Section 3.5) A coordinated national effort to reach children of prisoners will have far greater impact than isolated activities. Surveys methodically conducted by CNCS will enable us to provide an unduplicated count of children of prisoners being mentored in CNCS-related programs outside of MCP. This will be done for the FY 2007 long-term goal, but it may also be feasible to survey major, non-FYSB programs annually. The FY 2007 target was established by the President. The baseline uses the caseload achieved during FY 2004 by the first group of grantees. See FY 2003 -2004 breakouts in the Evidence Binder. Quality of relationships A positive assessment of a mentoring experience is strongly associated with the duration of the relationship and positive youth outcomes. We feel that a clear majority of cases (55 percent) should meet this standard already and have set the FY 2006 baseline accordingly, in effect making this a first-year target as well as a baseline. 60 percent is the long term (FY 2007) target for the measures of "Duration" and "Efficiency" (percentage of MCP matches which continue to completion as planned, i.e., do not cease or terminate prematurely, Section 3.4). Because of the correlations, this "Quality of Relationship" measure also targets 60 percent over the long term. This is a long term measure insofar as the establishment of an enduring relationship takes at least twelve months, and the most positive youth outcomes only begin to happen at that point (Grossman & Rhodes, 2001). (Citations for evidence in this section are in the NOTES tab and the Evidence Binder Table of Contents. The quote from the Statute is in the NOTES tab and also found in the original Statute in the Evidence Binder, Section 1.1.)

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: Duration of relationships so as to meet or exceed the 12-month standard Mentees in mentoring relationships lasting more than eleven months and concluding as planned in the twelfth month or shortly thereafter plus the percent of mentees in active mentoring relationships that have lasted twelve months or more as a percent of the entire caseload Source: Online Data Collection Companionship with caring adults Number of children of prisoners in one-to-one matches with caring adults who have been trained and screened by the MCP program and its local and national partners Source: Online Data Collection and partner information

Evidence: Duration of relationships so as to meet or exceed the 12-month standard Rationale: This is a broader measure than the long term, "sustainability" measure, because it includes the group that has reached the twelve month commitment, as well as those going beyond it. It can be reported annually, since the majority of its cases will probably last only twelve months. Under MCP, twelve month relationships may properly come to an end but are not required to do so. Research shows that it takes at least 6-12 months in a mentoring relationship before youth demonstrate positive outcomes. The greatest benefits are associated with mentoring relationships that last more than 12 months (Grossman & Rhodes, 2001). Accordingly, this key indicator of relationship health will be reported in for (a) all relationships meeting the duration standard and (b) relationships that have truly withstood the test of time, enduring thirteen months and beyond (sustainability measure). The mutual commitment by both parties in the (b) group suggests a strong bond has formed. Companionship with caring adults Rationale: This measure is based on the number of children of prisoners growing up with caring adult companions in relationships that conform to the evidence-based (one-to-one relationship) standard of the MCP. Forming and supporting these matches is the primary task of MCP grantees. The Statute emphasizes the need for local and regional partnerships. FYSB is building national partnerships as well, including with the programs of the Corporation for National and Community Service (CNCS) and with non Federal partners such as Big Brothers/Big Sisters and the National Mentoring Partnership. (See Section 3.5) A coordinated national effort to reach children of prisoners will have far greater impact than isolated activities. Surveys methodically conducted by CNCS will enable us to provide an unduplicated count of children of prisoners being mentored in CNCS-related programs outside of MCP. This will be done for the FY 2007 long term goal, but it may also be feasible to survey programs annually to measure progress. ?? MCP excerpt from FY 2006 Performance Plan (as revised) (FYSB document) (Evidence Binder, Section 2.1)

Does the program have baselines and ambitious targets for its annual measures?

Explanation: For the two measures, baselines are from different periods based on when reliable data was first available. Duration of relationships so as to meet or exceed the 12-month standard Mentees in mentoring relationships lasting more than eleven months and concluding as planned in the twelfth month or shortly thereafter plus the percent of mentees in active mentoring relationships that have lasted twelve months or more as a percent of the entire caseload. The targets assume 10 to 30% increases annually, which are ambitious for the program. Companionship with caring adults In the companionship measure, a linear progression of 33,000 new matches each year was established for budgetary purposes since the program has been level-funded beginning with FY 2004. Actual performance could be exponential. For example, a small grantee with a goal of 99 cases over their three-year project period probably would not match 33 children each year. The trend of results might be "non-linear," e.g., 1st year: 20 matches; 2nd year: 34 matches, 3rd year: 45 matches. Total matches: 99. There are several reasons for this non-linear growth: the learning curve of organizational development, the increasing visibility of the program leading to more volunteers (who also spread the word), the infusion of additional resources in the third year when the grantee's matching funds jump from 25% to 50%, and the impact of technical assistance/training/technology transfers/best practice identification. The annual targets are ambitious, and FYSB and its partners will work hard to meet them, since they lead to the long term goal set by the President. The ability to count partner cases on an unduplicated basis each year will be dependent upon partners' survey methodology and survey frequency. However, in calculating the long term goal of 100,000 children, FYSB will coordinate with partners (e.g., CNCS, Amachi, BB/BS) and conduct its own analysis to avoid double counting and verify criteria for including non-MCP cases.

Evidence: The BB/BS evaluation showed impact differences for youth mentored under 3 months; between 3-6 months; 6-12 months; and over 12 months. The greatest progress occurred as duration approached and passed 12 months. 62% (246) of Amachi matches (that had begun more than 13 months prior to the study) lasted at least 12 months. FYSB has set a goal of approaching that level over the next period of years, which is ambitious because our measure is a percentage of the entire caseload rather than of a subset of mature matches. Additionally, many MCP programs are starting out with less experience than the Amachi site in the study. Amachi and BB/BS programs are more experienced than many of the new grantees in MCP, but their contribution to meeting this goal should eventually be reciprocated as the rest of the MCP grantees gain experience and, in particular, learn from these pioneers. FYSB will provide facilitation, technology transfers, training, technical assistance, conferences, and dissemination of evidence-based practices (beyond those in the standard) as identified and validated in the evaluation. The influx of new case as the program grows (particularly during FY 2005 as the FY 2004 grantees implement their services) will offset the percentage of long-lasting cases to a somewhat unpredictable degree. Because of this, maintaining the baseline percentage during FY 2005 will be a challenge as the denominator grows. ?? MCP excerpt from FY 2006 Performance Plan (as revised) (FYSB document) (Evidence Binder, Section 2.1) ?? Jucovy, L. (2003). Amachi: Mentoring Children of Prisoners In Philadelphia. Philadelphia: Public/Private Ventures and the Center for Research on Religion and Urban Civil Society. Retrieved on June 22, 2005 at www.ppv.org/ppv/publications/assets/167_publication.pdf.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: Grantees' applications for funding represent commitments to serve given numbers of children over the three years of their project grant. Applications were required to project caseload targets and describe plans for meeting them. FYSB program specialists negotiate with each grantee to finalize annual targets and make adjustments for unforeseen circumstances (for example, an award of less funding than was originally applied for). Progress in reaching caseload targets is monitored closely on a quarterly basis for each grantee through the online data system and narrative reports. FYSB contractors and numerous grantees, researchers, and advocates demonstrated their commitment to meeting the goals of the program by providing input into the design and development of the online caseload data collection instrument, with which FYSB and grantees track and monitor their progress and performance. Most of this input preceded the public comment period of the Paperwork Reduction Act approval process, which resulted in a well-developed, consensus-based instrument for OMB and public review. The online data system itself was built by a contractor for the ACF Office of Information Systems, which is creating online data collection for many ACF programs. MCP was the "pilot" discretionary program in this effort. In the 1st Quarter of FY 2005, the beginning of official data collection under OMB approval, 95% of grantees reported their data, despite the difficulties inherent in utilizing any new management information system for the first time. The 95% response rate was also met in the 2nd Quarter. In the 4th Quarter of FY 2005, FYSB is acquiring a technical support resource to assist grantees, many of whom are small, volunteer-staffed programs without sophisticated IT expertise, having technical or other difficulties with reporting and using the system. Data for all long term and annual performance measures except "Quality of Relationship" and "Positive Life Changes" are collected every quarter. Data for the efficiency measure is collected every quarter. Annual data will be reported in the Performance Budget. Quality of Relationships will be collected annually using a statistical sample, after an instrument is approved by OMB. FY 2006 data on this measure should be available by the end of FY 2006. Data for Positive Life Changes will be collected on a one time basis over several years through a national evaluation using experimental methods. Final results should be available in FY 2009 or sooner. FYSB's partners have also shown a commitment to the program goals through their interest in the evaluation. FYSB has been developing its evaluation plans in the course of an ongoing dialogue with experts and grantees, particularly many who have keen interests in evaluation and research. FYSB also arranged for well-attended user-input sessions at national conferences to discuss the broader issues of evaluation, outcome data collection, and applying such knowledge to program progress.

Evidence: ?? Synopsis of research design with research questions for each component from evaluation design study, Abt Associates, Cambridge, MA ?? List of grantees expressing interest in being involved in evaluation activities (not a list of national evaluation sites, which will be selected statistically, but program sites where augmentations of the national evaluation may be feasible. ?? HHS Focus Group notes: outline of user input discussion with grantees and the National Mentoring Partnership during the February, 2005, national MCP conference. (Evidence Binder, Section 2.5)

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: FYSB has contracted for a detailed design for a national evaluation of MCP, which was delivered in April, 2005. FYSB is analyzing the recommendations. The contract under which the design was produced has been modified with new funds and extended to begin design and development of case-specific outcome data collection instruments and systems. These will enhance or eventually supersede the current, aggregate caseload data collection system. In addition to supporting the independent, sample-based evaluation, an online, case-by-case reporting system will help grantees track all cases, measure and analyze factors such as "quality of relationship" using validated instruments, and record behavioral change and academic attendance (i.e., elements of the MCP's long term measure) on an ongoing basis. Existing systems (such as are used by BB/BS) and available alternatives are being analyzed to avoid redundancy, duplication and strain on capacity. It is important that requirements in this area represent no capital costs and minimal information collection burden on MCP grantees. Upon passage of an FY 2006 budget, FYSB will publish an RFP to implement the evaluation. The design calls for three components which will be implemented largely in parallel, beginning approximately mid-FY 2006. One component will review the capacity, organizational, and service approaches being used by grantees and identify promising approaches in mentoring, service delivery, and other critical aspects of the program (e.g., vouchering, as discussed in Section 1.4). This will build upon existing descriptive research provided by Abt Associates in the Design Study (see Section 4.5). A second component (associated with the case-specific outcome data collection system), will collect and analyze outcome data for all children in the program. A third component will conduct two branches of random assignment research. One branch will compare outcomes among eligible children who are randomly assigned to mentors or waiting lists, an approach similar to the BB/BS evaluation. A second branch will randomly assign eligible children to different mentoring approaches within the MCP program to identify and verify with evidence which practices contribute most effectively to long term outcomes. The evaluation will run for several years to measure changes resulting from mentoring relationships. A report of findings is expected to be available in FY 2009. Prior to that time, useful information will be released if it is well-validated and has a bearing on practice. The final report will be a user-friendly tool kit for practitioners as well as a source of interest to funding sources, policy makers or scholars. Meanwhile, FYSB will look for individual and/or program-wide strengths and weaknesses through on-site monitoring visits (discussed under Section 3.CO2) and ongoing analyses of currently available narrative and caseload data and endeavor to improve results through technical assistance and proactive grants management.

Evidence: ?? Synopsis of research design with research questions for each component from evaluation design study, Abt Associates, Cambridge, MA ?? List of grantees expressing interest in being involved in evaluation activities (not a list of national evaluation sites, which will be selected statistically, but program sites where augmentations of the national evaluation may be feasible. ?? HHS Focus Group notes: outline of user input discussion with grantees and the National Mentoring Partnership during the February, 2005, national MCP conference. (Evidence Binder, Section 2.5)

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: ACF submitted a fully integrated Congressional Justification Performance Budget for FY 2006 that includes performance information in the core program sections of the budget request. The FY 2005 budget request was to sustain 33,000 mentoring relationships. The FY 2006 budget request is the same as the FY 2005 enacted level. These funds will be used to add 33,000 new cases and sustain approximately 72,000 mentoring relationships which will support one of the program's key performance indicators. The linear progression of 33,000 cases new each year was established for budgetary purposes. When the President introduced the Mentoring Children of Prisoners program, the Administration set a goal of mentoring 100,000 children over a four year period. This projected number of children served in the MCPrisoner program was based on a national study in the general mentoring field that had roughly estimated that the median cost of mentoring a youth was about $1533 per case. While the pre-existing programs were not focused on children of prisoners and varied widely in quality, the estimate provided an overall cost estimation range, though not applicable to individual grantees affected by varying circumstances (such as geographic isolation of children and/or volunteers). This does not mean that every additional $1500 cost unit appropriated automatically adds one more mentored child to the caseload in a linear fashion. Start-up costs (such as organizational development, outreach, recruitment, and mentor training) are incurred initially much faster than the intake of fully matched cases; moreover the relative rate between costs and matches may shift unevenly as caseloads grow at varying paces. Altogether, the relationship between inputs and outputs in this program is nonlinear, under the effects of several variables and uncertainties. These include variable inflation and geographic variation of costs, the approximate nature of the $1500 median "unit cost," and the increased levels (from 25% to 50%) at which third year grantees match federal funds. During the summer of 2005, grantee match creation data are being compared to goals set forth in the original applications and renegotiated by FYSB with each grantee. Discontinuation of non-performing grantees is currently under consideration by the Associate Commissioner of FYSB and the Commissioner of the Administration on Children, Youth, and Families. Funds may be reprogrammed from non-performing grantees into locations where there appear to be unserved families of prisoners and where grantees are demonstrating an ability to reach or exceed their targets.

Evidence: ?? Emails with National Mentoring Partnership on the origin of the $1533 estimate (FYBS document) ?? FY 2003-FY 2006 Congressional Justifications for MCP (FYSB document extracted from ACF document) ?? Chart illustrating the approximate flow of Federal resources and their utilization to achieve strategic goals and targets (FYSB document)

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: Performance Outcome Measurement: The PART process has helped FYSB refine and augment its performance measures. Because it will take time to obtain long term outcomes through a controlled experiment, FYSB also decided to add measures for relationship sustainability and quality, key indicators of long-term outcomes. Additionally, following a review of the 1995 Big Brothers/Big Sisters impact study, MCP's previous long term outcome of "measurable behavior and other gains" (positive life changes) has been restructured from a more diffuse measurement into specific behavioral and academic goals. MCP youth will be "??less likely to report initiating drug or alcohol use than control group youth" and "will report skipping fewer days of school than control group youth." After consulting with researchers and mentoring experts, FYSB decided to measure these outcomes in such a way that would most accurately reflect the impact of the MCP program. Accordingly, MCP has chosen to fully implement the original evaluation design (synopsis in Section 2.6), which was originally considered to be limited strictly to the validation of promising practices. Rather, MCP concluded that, because all youth typically increase risk-taking behaviors and change so much during adolescence, the only way to reliably attribute impacts to the program is through a randomized experiment using identical, equally-disadvantaged youth as controls. MCP will continue to use experimental methods to compare approaches and methods among MCP programs and verify efficacy. This will enable the program to disseminate evidence-based practices through technical assistance and will be based on outcomes among MCP children randomly assigned to varying approaches. Strategic Risk Reduction: Strategic planning also consists of anticipating critical risk factors that could radically affect the program. The most serious risk that an MCP program might face would be harmfully inappropriate behavior by an adult mentor with a child, if confirmed. This would have a very damaging effect on the program's reputation, certainly in the local community, and could severely impact performance, possibly nationally. Careful criminal background checks and screening and intensive training of volunteers mediate against that risk, as does making sure that youth know

Evidence: Sustainability: Relationships that endure beyond the twelve month minimum, possible becoming lifelong, clearly indicate bonding and opportunity for the adult to have continuing influence as the child continues to grow. These relationships are associated with the greatest improvements in outcomes, based on research discussed in Section 1.4). Relationship Quality: A child's perception of mutual respect, supportiveness, and other interpersonal assets is a strong influence upon positive outcomes. Researchers have recently tested a 15-item questionnaire that can quickly and reliably measure the quality of the mentoring relationships. Duration and sustainability of matches and an annual survey of mentees will provide good benchmarks of the overall success of the program. (PART Section 2.2). Positive Life Changes: The long-term evaluation will show the extent to which outcomes for children of prisoners resemble those for youth in BB/BS who generally had fewer risk factors. (PART Section 2.1) ?? No "strategic deficiencies" have been evident or suspected at this early stage. No issues have been raised about MCP by the HHS Inspector General, OMB, GAO, or Congress nor are we aware of investigations by journalists. ?? Email with HHS contracting officer regarding extension of research design contract (FYSB document) (FYSB recently funded a 12-month extension of this task order for the development of instruments that will move beyond aggregate, caseload data and enable analysis of the individual mentoring relationships. ) ?? According to a statement of June 13, 2005, from Dr. Jean Rhodes of the University of Massachusetts, an associate or principal investigator on a number of major mentoring evaluations: "There have been very few problems over the years involving mentor abuse of mentees. The intensive screening process employed by many organizations, combined with effective trainings, has led to a minimization of this risk.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: In general, non-financial reports in ACF and HHS grants programs (e.g., in FYSB's RHY programs) are required semiannually. Because MCP was a new program as of FY 2003, with many small and/or "start up" service-providers, FYSB established the requirement for MCP grantees to submit progress and caseload reports more frequently on a quarterly basis so that potential problems could be anticipated or detected in a timely fashion and technical assistance and other interventions could be implemented. Every grantee must submit quarterly progress reports, quarterly caseload data reports, and semi annual financial reports. The response rates for both progress and financial reports are 100%, and the rate for two quarters of caseload data reports is 95%. (The 5% non reporting grantees were among a group that had technical difficulties with the online reporting system. Additionally, during the 2nd quarter deadline period, the ACF network platform failed intermittently. FYSB is establishing a technical support resource to address these issues. (Section 3.7). (1) Quarterly progress reports provide narrative information on accomplishments and challenges. (2) Semi-annual financial reports document fiscal practices and allocation of funds to appropriate activities. A FYSB program specialist is assigned to each grantee and follows up on any information (or missing information) in the progress reports that indicates problems are being encountered, whether programmatic or financial. The financial reports are discussed in more detail under 3.6. (3) Quarterly caseload data reports present the quantitative facts of program operations. In the first year of the program, while the data collection instrument was undergoing OMB and public review, useful caseload information was extracted from the anecdotal narrative reports and used cautiously by limiting the study to reported caseload counts. Official, systematic collection of extensive caseload data began in the first quarter of FY 2005 using an online data collection system prepared by an ACF-wide contractor pursuant to P.L. 106-107.

Evidence: duration, regularity, and frequency of contacts. Data for all long term and annual performance measures except "Quality of Relationship" and "Positive Life Changes" are collected every quarter. Data for the efficiency measure is collected every quarter. Annual data will be reported in the Performance Budget. Quality of Relationships will be collected annually using a statistical sample, after an instrument is approved by OMB. FY 2006 data on this measure should be available by the end of FY 2006. Data for Positive Life Changes will be collected on a one time basis over several years through a national evaluation using experimental methods. Final results should be available in FY 2009 or sooner. Additional information is available on demographics, mentor training and supervision, participation in group activities, time spent on waiting lists, numbers of match terminations and which party sought the termination. These data are analyzed after being received online in the month after the end of each quarter. Now that two quarters of official caseload data are available, FYSB is able to use it as a management tool. This caseload analysis tool provides performance information on more than a dozen indicators, showing the grantee's results and comparing them with the national average. They can also be compared with similar grantees or any other subset that provides a meaningful comparison. FYSB staff members are currently piloting this instrument and will begin using it to guide their daily interactions with grantees and regular monitoring and technical assistance. When the indicators show lagging performance, grantees will be contacted to provide additional information about the issues they are experiencing and, depending on the level of technical assistance required, site visits will be scheduled. ?? Standard Terms and Conditions Funding Agreement between FYSB and MCP grantees (FYSB document) ?? Financial Status Report Form 269 (OMB document) ?? Performance/Progress Report form (FYSB document) ?? Caseload data instrument (FYSB document) ?? MCP Quarterly caseload performance analysis tool (FYSB document) ?? List of grantees and 2nd Quarter caseloads (FYSB document) (Evidence Binder, Section 2.1) ?? 1st and 2nd Quarter results for all data elements (FYSB document) (Evidence Binder, Section 2.1)

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: "The Associate Commissioner of FYSB is directly responsible for the MCP program and the executive performance plan for this position reflects program performance requirements. The Associate Commissioner reports to the Assistant Secretary for Children and Families through the Commissioner of the ACYF, both of whom are Presidential appointments confirmed by the Senate. The Assistant Secretary provides a high level of support for this program, evidenced by the article he authored on the program. FYSB program specialists are rated for their efforts to achieve program compliance, quality, and results. Performance plans for FYSB's Director of Research and Evaluation and the MCP team leader contain performance and management elements relevant to the program (see specific excerpts in evidence section). Contractors for data collection technical support, evaluation design, research to improve cultural competency in mentoring programs, information dissemination, and others are monitored and held accountable by FYSB officers. Contractors are required to submit monthly invoices of services provided under the contract. Trained and certified federal project officers with extensive program knowledge guide projects, review expenditures for contracts on a monthly basis and approve or disapprove reimbursement items. FYSB Project Officers have to be certified by the Contracting Office to manage contracts and are held accountable for cost, schedule, and performance. All FYSB project officers have received training and certification within the last three years. Signatory authority resides outside of FYSB. Every quarter, in both progress narratives and data reports, grantees document their progress toward establishing mentoring matches and achieving long term relationships. The annual performance measures, and a larger set of diagnostic indicators, are applicable for each grantee. Performance for each grantee is being scrutinized individually, not only as a national aggregate. (See discussion and Evidence Binder under Section 3.1 of individual grantee performance measurement and analysis using the quarterly caseload performance analysis tool.) Each project is funded one year at a time. Non performing grantees are subject to discontinuation or may be required to spend any carry over funds before receiving all or part of their grant for a subsequent year. Grantees lagging seriously in their establishment of mentoring relationships may be discontinued. Quarterly meetings are being held with the Commissioner of ACYF and FYSB program specialists to discuss individual grantee issues and what actions to take, in addition to site visits, if serious weaknesses persist.

Evidence:  FY 2004 Performance Contract for Associate Commissioner, FYSB  Horn, W. Helping the Hidden Victims: Mentoring the Children of Prisoners. Retrieved June 22, 2005, from the Administration for Children and Families Web site: www.acf.hhs.gov/programs/cb/hotissues/hiddenvictims.htm. (This article is an example of support and accountability at the Senior Executive level.)  MCP Quarterly caseload performance analysis tool (Evidence Binder, Section 3.1) The first performance element (critical) for the FYSB Director of Research is:  Build the research and evaluation capacity of FYSB to identify proven practices and improve program results and begin integrating at risk family and youth services research and evaluation strategies across RHY, mentoring, family violence, and abstinence programs Direct the design of a major national research effort and online evaluation instrument for mentoring program.  Finalize OMB approval and establish and refine data collection and continuing evaluation activities in the mentoring program.  Develop systems for recording and analyzing mentoring grantees' quarterly caseload reports. -- FYBS Director of Research performance plan The performance plan for the MCP Team Leader includes the following "Major duties":  The lead Program Specialist for the Mentoring Children of Prisoners Program is responsible for working with the Bureau's research staff as it relates to the Mentoring Children of Prisoners Program as well as developing and recommending program policy in order to enhance program effectiveness. This includes: Serving as lead staff person in the development of the Mentoring Children of Prisoners Program Announcement. Participating in the FYSB grant review process serving as the Subject Area Manager (SAM) for the Mentoring Children of Prisoners Program grant reviews. Assisting in the development of funding recommendations for the Mentoring Children of Prisoners Program grant review. Participating in the awarding of new start and non-competing MCP Grants. Conveying accurate information to grantees and the general public as it relates to the Mentoring Children of Prisoners Program. -- FYBS MCP team leader performance plan

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: Administration on Children, Youth and Families under the ACF grants-making process, have been awarded and transferred by FYSB on or ahead of schedule for both years of program operations, and zero funds have remained unobligated at the Federal level at year end. MCP has not incurred a "lapse" of funding. Grantee funds are obligated through the ACF Grants Administration Tracking and Evaluation System (GATES) system and monitored through the SF-269 financial status reports for each project period. Funds are allocated only for one year at a time. Grantees are expected to spend annualized awards during the single fiscal year. Carryovers are not uncommon in start-up programs like MCP, but they are regulated by FYSB. If it is necessary to carry over funds, a carry-over request has to be approved, and funds can only be used for the purpose of completing unfinished, previously-approved activities. Any grantee showing signs of inaction with regard to their funding allotments (such as a significantly slow "burn rate") or serious setbacks in establishing mentoring matches, becomes the immediate focus of concern and technical assistance by FYSB program specialists and by the Office of Grants Management. In cases of sufficient concern, these grantees may not receive new funds until carryovers are spent, and their Federal funding may be at risk altogether. The Commissioner of the Administration on Children, Youth and Families is holding quarterly meetings on these and related issues with FYSB program specialists (and will meet more frequently as needed.) Spending plans must be detailed in grant applications prior to competitive selection. Grantees must provide totals of direct charges, indirect charges, and project costs in their application. Costs must be allowable and include MCP staff salaries, fringe benefits, equipment and supplies used for project purposes only, indirect costs documented with a copy of the current rate agreement, and travel (with detailed documentation) by grantee staff (not consultants) to FYSB conferences and training events. Organizations requesting financial assistance must file Standard Form 424B "Assurances: Non-Construction Programs." If grantees expect to generate income from their program, they must estimate the amount, describe the nature and source of the income, and detail the anticipated use of such income in their budget. Non-Federal resources must be documented and submitted with the application or in the course of their accrual in order to be given credit as matching funds or in-kind services. Grantee financial reports are scrutinized by ACF grants management staff, separate from FYSB, for compliance with spend-down and financial matching requirements. MCP grantees are not disbursing cash, making complex eligibility determinations, administering financial benefits or acquiring tools or materials for construction or production, which lowers the financial risk level. The services they provide to children of prisoners are documented in progress and caseload data reports, found in the Evidence binder under Section 3.1.

Evidence: In general, non-financial reports in ACF and HHS grants programs (e.g., in FYSB's RHY programs) are required semiannually. Because MCP was a new program as of FY 2003, with many small and/or "start up" service-providers, FYSB established the requirement for MCP grantees to submit progress and caseload reports on a more frequent quarterly basis so that potential problems could be anticipated or detected in a timely manner and technical assistance and other interventions could be implemented. In addition to FYSB's oversight systems, there are additional lines of defense for program integrity. ACF receives and acts upon audit reports on all grantees receiving funding of $500,000 or more, pursuant to the Single Audit Act (31 U.S.C § 7502 (a)(1)(A)) and OMB Circular A-133. This represents 23 MCP grantees, or about 10% of the total. Many of the remaining grantees automatically submit to independent audits as a condition of their organizational charters, because of membership in organizations such as BB/BS, requirements of their boards of directors, and as required by non- Federal funding sources, or as requirements of non-profit status. For example, each Big Brothers/Big Sisters agency is an independent nonprofit organization with its own audited financial statement. There are 43 BB/BS programs that receive MCP funding of less than $500,000. (Ten BB/BS are at the $500,000 threshold and subject to the Act.) Another 35 MCP grantees are public agencies (County or State), or belong to reputable national networks with their own organizational integrity requirements such as YMCA, Campfire USA, Catholic Social Services, Covenant House, Girl Scouts, and Volunteers of America. Five more are Tribal governments. Others are community service agencies with requirements generated by their other Federal funding. Thus, over 50% of the entire grantee community conforms to accounting and financial management standards prescribed by their national organizations. All first year grantees subject to the Single Audit Act have filed their findings as required. The ACF end of this process is underway based on the ACF External Audit Resolution Manual. Additionally, status reports are reviewed on a quarterly basis (performance) or semi-annually (finance), monitoring site visits are underway and a monitoring tool, including fiscal inspections, is in development. (See Section 3.CO2) ?? Detailed Common Accounting Number (CAN) Activity Report for 2004 (ACF document) ?? ACF External Audit Resolution Manual (ACF document) ?? Monitoring Protocol/Outline for Pilot Site Visits (FYSB document) (Evidence Binder, Section 3.CO2) ?? Recent list of pilot monitoring sites (FYSB document) (Evidence Binder, Section 3.CO2)

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: All grants were awarded through rigorous competition under standards published in the program announcement. The one exception is where competition was limited in order to select five tribal grantees. (Tribes had not competed successfully during the first year of grant awards.) The Statute provides for an escalating match structure, which maximizes the impact of every federal dollar spent. Grantees provide 25% of the project costs in the first two project years; in the third year the level becomes 50%. In-kind matching is permitted, but must follow the criteria spelled out in the program announcement and "terms and conditions" of funding. FYSB awards contracts in full compliance with Federal Acquisition Regulations Part 92 (procedures for awarding government contracts through full and open competition). Contracts financed by the grantee must be for allowable costs (if Federally-funded) and provide full and open competition under Part 92. Contracts are reviewed by FYSB for such compliance. ACF created an on-line data reporting system to measure program performance and efficiency at no cost to the MCP program itself. An ACF-wide contract provided the programming and web development resources. Grantees were trained on the system by FYSB and contractor personnel. FYSB is investing in a support task under the ACF contract to assist grantees experiencing technical problems in reporting. This task will also provide for data analysis and follow-up to assure reporting compliance. (See discussion under Section 3.7.) A considerable amount of MCP grantee activity, in addition to actual mentoring, is undertaken by unpaid volunteer staff, which saves on salary costs. Grantees expend tremendous effort to retain their volunteer mentors by providing adequate support. For example, some provide support groups where mentors can discuss common challenges or host events to recognize the work of their mentors. Many grantees organize, or provide free tickets to, events that mentors and mentees can enjoy together, such as a baseball game. FYSB will measure efficiency through a measure which focuses on return-on-investment in non-financial terms that embody significant programmatic and qualitative implications, as well as productivity issues: "the percentage of mentoring cases which last no more than three months as a percent of the total number of matches which ceased during the year (prematurely or on schedule)". This measure is indirectly dollar-based, because it addresses effective conservation and management of mentoring relationships, which are primary cost centers with mostly front-loaded expenses. Along with "Duration of Relationship" and "Quality of Relationship" measures, this measure is based on the principle that long term, quality relationships that persist throughout their intended lifespan of at least twelve months provide the best return on investment and are associated with positive behavioral and emotional outcomes.

Evidence: matches which ceased during the year (prematurely or on schedule). Goal: minimize matches of very short duration Rationale: Matches which end prematurely represent a significant investment loss, because costs are associated to a large degree with outreach, recruiting, screening, training and preparing mentors before the initiation of matches. Premature cessations are also a programmatic liability, since a child's self esteem can be impacted negatively if he or she loses trust or feels abandoned. In the MCP program, mentors are expected to commit to at least twelve month relationships. As discussed in Section 1.4, this is because the positive effects of mentoring only begin to appear after six to twelve months, as the relationship matures. While a case which lasts no more than six months is also undesirable, the investment loss for cases lasting less than three months is almost total. Some terminated matches end ahead of time by mutual agreement for neutral reasons and are not due to mentor desertion or failure, for example, if the mentor's job takes him or her out of the area. In addition, often children of prisoners are in itinerant or disorganized families with impermanent living circumstances. Grantees must strive not only to hold mentors to their pledges, but to keep the children connected to the program and its positive benefits by gaining commitment from the family. This measure addresses both sides of the match. By effectively matching adults and children and providing supportive activities, grantees strengthen the odds of continuation by families and by mentors. Baseline: FY 2005: 37% (the actual rate in the 1st Quarter, FY 2005; FY 2004 data are not available) FY 2005 Target: 30% FY 2006 Target: 25% FY 2007 Target: 20% The current targets, which embody the reduction of a negative, may be highly ambitious. "One half of all volunteer [mentoring] relationships dissolve within a few months." (Rhodes, 2002) However, these targets are attainable due to vigilance by FYSB to ensure that grantees incorporate best practices, such as screening, training, and ongoing support of the mentor and the mentoring match. These help prevent relationships from dissolving unnecessarily. ?? MCP program announcement, Evidence Binder, Section 1.1 (FYSB document) ?? MCP excerpt from FY 2006 Performance Plan (as revised), (FYSB document) Evidence Binder, Section 2.1 ?? MCP program announcement, Evidence Binder, Section 1.1 (FYSB document) ?? MCP excerpt from FY 2006 Performance Plan (as revised), (FYSB document) Evidence Binder, Section 2.1 ?? Freedman, M. (1993). The Kindness of Strangers: Adult Mentors, Urban Youth and the New Volunteerism. In Rhodes, J. (2002). Stand by Me, The Risks and Rewards of Mentoring Today's Youth. P.57 Cambridge: Harvard University Press.

Does the program collaborate and coordinate effectively with related programs?

Explanation: FYSB has initiated or joined collaborative efforts with many of the Federal and non-Federal leaders in the mentoring field to strengthen the MCP program. MCP was developed in consultation with several partners, including the Departments of Justice and Education, the White House Office of Faith-Based and Community Initiatives (OFBCI), USA Freedom Corps, and the HHS Center for Faith-Based and Community Initiatives. FYSB applied lessons learned from the Office of Juvenile Justice and Delinquency Prevention (OJJDP) demonstration grants for services to children of prisoners, administered through the National Institute of Corrections and the Child Welfare League of America. The initial program announcement for MCP was drafted in consultation with the White House Office of Faith-Based and Community Initiatives, the HHS Center for Faith-Based and Community Initiatives, USA Freedom Corps, and the Department of Education. MCP is a Presidential priority and one of three pilot programs of the Faith-Based and Community Initiative. Since the program's inception, the HHS CFBCI and the White House OFBCI have reached out to constituents via conferences, listservs and websites to provide information to potential program applicants. In addition, the HHS CFBCI partnered with FYSB to provide targeted outreach and funding for pre-bidder's training conferences (held "live" and by conference call) across the country, with the end result that 450 applications were received for the program in FY04. The HHS CFBCI also disseminated the written training materials provided at these events on their website. Weekly, and now bi-weekly phone conferences are held between the three partners to ensure that efforts are coordinated. The Mid Atlantic Network for Youth provided T&TA to grantees during FY 2004 under a cooperative agreement. FYSB has also participated in intergovernmental and private coordinating councils in an effort to ensure that MCP is implemented in a manner that is consistent with the best practices in the field. FYSB participates in the Juvenile Justice Coordinating Council that brings together Federal agencies that administer mentoring programs. FYSB also attends "Mentors Matter", convened by the Pew Charitable Trusts, which brings together the leading Federal and non-Federal mentoring programs and partners. These roundtables have included Justice (OJJDP and Bureau of Prisons), Labor, other agencies in HHS (e.g. Office of the Assistant Secretary for Planning and Evaluation), and non Federal partners such as Amachi, National BB/BS, U.S. Dream Academy, Angel Tree, and Prisoner Ministries. These meetings may discuss regional issues not always reported by local grantees (e.g., a State correctional system insisting on cooperating with mentoring efforts only through incarcerated parents of their own selection) and have helped develop consensus on performance measures and evaluation plans. The most significant collaborations continue to evolve. The Corporation for National and Community Service (CNCS) has recruited thousands of seniors and foster grandparents to serve children of prisoners in a number of ways, including mentoring. To verify the achievement of the goal of mentoring 100,000 children of prisoners, FYSB, CNCS and other partners will share data on their caseloads.

Evidence: relationships with screened, trained mentors and (2) who among these are served by programs (or segments of programs) not funded through MCP. CNCS is endeavoring to take these measurements on a quarterly basis. However, even if CNCS data are only available annually, FYSB will be able to report an unduplicated count of children served by MCP grantees and partner organizations. ?? CNCS encouraged Retired Senior Volunteer Program (RSVP) volunteers to mentor children of prisoners; many are now serving with MCP grantees. There will also be considerable efforts from FYSB, CNCS, and other national organizations to engage communities, the private sector, and volunteer organizations in recruiting mentors and increasing the visibility of MCP. (CNCS Report on "Serving Children of Prisoners" and "Senior Corps Examples" ?? U.S. Department of Health and Human Services/Administration for Children and Families/Administration on Children, Youth and Families/Family and Youth Services Bureau. (March, 2005). The Exchange, Pittsburgh: Mid Atlantic Network for Youth. (FYSB document) ?? Resources to Support MCP (FYSB document) ?? FYSB staff participate in the Juvenile Justice Coordinating Council - facilitating a unified approach to at-risk youth, the evaluation of mentoring services across government, and other collaborative efforts. (OJJDP Administrator Robert Flores letter to FYSB MCP team leader, with summary of meeting and FYSB presentations) ?? FYSB staff (Stan Chappell and Daniel Corle) played active parts in a series of planning meetings for the Prisoner Reentry Initiative and held workshops to provide technical assistance to stakeholders of the Serious and Violent Offenders Reentry Initiative. ?? The HHS Center for Faith-Based and Community Initiatives (CFBCI) helped with applicant recruitment and technical assistance through several regional pre-bidders conferences and disseminated information during and after conferences. ?? The HHS CFBCI also helped with grant review training to assure that appropriate consideration was given to faith-based and community applicants. (HHS CFBCI power point document) ?? Staff from ACF and other agencies in HHS supplemented FYSB staff in oversight activities by re-reviewing every successful grant application for activities that would be potentially unallowable under the Separation of Church and State clause. These issues have been followed up on by program specialists under the direct guidance of the Commissioner of the Administration on Children, Youth, and Families. ?? May 5, 2005 "Mentors Matter" agenda ?? FYSB arranged a brainstorming session in February, 2005 with the National Mentoring Partnership, ACF Office of Information Systems, FYSB research contractors and MCP grantees to provide input to FYSB in the development of online evaluation instruments and tools. National BB/BS is sharing information and expertise as FYSB undertakes further developments of its information collection system, particularly extending the data set into outcome areas beyond those already incorporated. (Focus group notes from February, 2005, Evidence Binder, Section 2.6).

Does the program use strong financial management practices?

Explanation: A budget narrative is required as part of the funding requirements for all grantees. These are closely analyzed by Federal staff prior to the issuance of awards. After receiving grants, organizations must submit quarterly program reports and semi-annual financial status reports (SF-269). Financial match compliance is documented in financial status reports and subject to specific criteria which FYSB program specialists review and convey to grantees through project monitoring. For example, wrap-around services do not count towards the matching requirement. Grantee funds are obligated through the Grants Administration Tracking and Evaluation System (GATES) and monitored through the SF-269 reports for each project period. Award recipients typically spend annualized awards during the single fiscal year. If it is necessary to carry over funds into subsequent fiscal years, a carry-over request has to be approved and funds can only be used for the purpose of completing unfinished, previously-approved activities. MCP does not require grantees to report or break down allowable costs in percentages per category. Site visits and review by FYSB staff of performance reports alongside the financial status reports helps ensure that funds are used for allowable costs and to support the beneficiaries of the program. Prior approval of budget revisions is required, in accordance with HHS and ACF grants management policy, and are arranged through FYSB program specialists working with the Office of Grants Management. Allowable costs for expenditures of FYSB funds are provided for in the Project Summary portion of the Program Announcement, and programs are evaluated on meeting the allowable costs requirements. These costs include personnel, fringe benefits, travel, equipment, supplies, contractual, indirect charges, program income, non-Federal resources, and assurances of non-construction. Each area must be justified. In the "personnel" category, the employee's position and time commitment to the program (in months) must be outlined. Personnel costs do not include hiring of consultants, the personnel costs of delegate agencies, or other projects or businesses financed by the applicant. Contractors are selected through aggressive competition. For example, more than a dozen research firms competed for the contract to develop an MCP evaluation design. Selection of the winning firm took place after an extensive review by a panel of FYSB and non-FYSB federal officials with significant relevant experience to make accurate judgments about competing proposals. Selection of a national technical assistance/resource center has been similarly rigorous, as will be the selection of an evaluation contractor. Contractors are required to submit monthly, itemized invoices of services provided under the contract. Trained and certified federal project officers with extensive program knowledge guide projects, review expenditures for contracts on a monthly basis and approve or disapprove reimbursement items. FYSB Project Officers have to be certified by the Contracting Office or Govworks to manage contracts and are held accountable for cost, schedule, and performance. All FYSB project officers have received training and certification within the last three years. Signatory authority resides outside of FYSB.

Evidence: FYSB receives and acts upon audit reports on all grantees receiving funding of $500,000 or more, pursuant to the Single Audit Act 31 U.S.C § 7502 (a)(1)(A); OMB Circular A-133. This represents 23 MCP grantees, or about 10% of the total. Many of the remaining grantees automatically submit to independent audits as a condition of their organizational charters, because of membership in organizations such as BB/BS. (See also Section 3.3) ?? Standard Terms and Conditions Funding Agreement between FYSB and MCP grantees, (FYSB document) (Evidence Binder, 3.1.) ?? Program Performance/Progress Report and SF 269 Financial Status Report, (ACF documents) (Evidence Binder, Section 3.1.) ?? Primary Sources of ACF Grants Management Guidance (ACF document) ?? ACF External Audit Resolution Manual (ACF document) (Evidence Binder, Section 3.3.) ?? Examples of agenda from national MCP conferences including grants management training (FYSB documents) ?? Managing your Grant, Facilitator's Notes for Grants Management Workshop (FYSB document)

Has the program taken meaningful steps to address its management deficiencies?

Explanation: "Most difficulties have been logistical in nature and are typical in the development of a new program. 1. Technical Assistance: FYSB made an attempt to procure a national training and technical assistance (T&TA) contract that would replace an expired cooperative agreement under which technical assistance had been provided during the first year. Deliverables included conferences, a list serve, and materials included in the Evidence Binder under 3.7. During the first half of FY 2005, the acquisition process for a permanent T&TA provider, which is always lengthy, was delayed for several additional months because all offerors in the competitive range came in with bids that were far more expensive than FYSB's independent estimate and budget. The scope of work was been narrowed, a new RFP was released in July, 2005, and a contract will be awarded before the end of FY 2005. 2. Data Collection: The contractor who constructed and who is operating the Online Data Collection System is accountable to ACF's Office of Information Systems (OIS), rather than directly to FYSB. By deferring to the new ACF requirement that OIS fund and implement all online systems development under a comprehensive contract, considerable FYSB funds were saved (and reinvested in additional evaluation and data collection design activities). However, FYSB does not direct the work of the OIS contractor and is one of dozens of ACF programs, many much larger, who are part of the overall ACF online data collection system (OLDC) operated by OIS and its contractor. Many of the new and smaller MCP grantees are inexperienced in using computers or websites and have had difficulties using the quarterly caseload data reporting system. FYSB research staff provided considerable direct technical support during the first two quarterly reporting timeframes and achieved 95% response rates in each period. However, the effort was labor intensive and not an optimum investment of FYSB FTEs. It took staff time away from analytical tasks with potentially more value than "help desk" functions. As a solution, FYSB is funding a technical support resource as a subtask to the OIS contract to assist grantees with the system. It should be operational by the 4th quarter of FY 2005. While the contractor will report directly to OIS, as was the case before, this will be a dedicated component and will provide assistance to grantees having difficulties, address possible issues of completeness, timeliness, and accuracy, and assist FYSB in generating and analyzing diagnostic and performance reports.

Evidence: "?? RFP for T&TA contract, released June, 2005 (FYSB document) ?? Email with ACF Office of Information Systems (FYSB document) "

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: The initial MCP program grant announcement was released in April of 2003. It set forth requirements for potential grantees to ensure a competitive application process and lay the groundwork for developing high quality mentoring programs. By the end of FY 2003, FYSB processed over 200 applications and approved 52. These grantees were funded by September of 2003 and began to launch programs. When the FY 2004 program announcement was released, FYSB worked with a number of Federal partners and its own contractor to conduct outreach and share information widely about the MCP funding opportunity. FYSB reached out to all potential applicants and organized two conferences and several tele-conferences that helped clarify the application process and allowed current grantees and potential future grantees to network and exchange ideas about mentoring programs. Subsequently over 450 applications were received and 169 approved. FY 2004 grantees have expanded the geographic base of the program, adding programs from 17 new States and Puerto Rico and five tribal organizations. Applications are scored and funds are awarded after a program announcement has been published according to standard HHS practices. The announcement itself goes through several levels of review outside within ACF and HHS. The criteria for applications are based on an OMB-approved "Uniform Program Description." MCP awards were made after extensive peer review by non-federal panelists selected for their knowledge of mentoring and human services, child/youth development, or program management. Final award decisions are made by the Commissioner after a review by federal staff. To date, there have been no earmark requests to consider. In FY 2004, FYSB set aside limited funds to assure that tribal organizations would apply and be funded. Grant review panelists are independent of FYSB and must provide explicit written assurance of no conflicts of interest or other disqualifying factors. The reviews take place over a period of weeks (depending on the number of applications) in Washington D.C., and are supervised on site by FYSB staff. Panels are led by chairs who ensure consistency and defensibility of scoring award recommendations. The scoring approach is systematic, explicit, and applied consistently to all applications. Training of panelists in scoring methods is provided in depth. After the first year of awards, (FY 2003) FYSB and HHS hired contractors to provide training and technical assistance and conduct conferences around the country. Though not exclusive, FYSB specifically reached out to previously unsuccessful applicants who scored better than 50 in the FY 2003 competition.

Evidence: The ACF Unified Program Description includes detailed instructions and disclosure forms to entities applying for funds. Applicants must provide documentation and detailed narratives about their program objectives, service delivery approach, the local need for assistance in the program area, organizational experience, staff qualifications, and proposed budget. Awards are transmitted annually during three year project periods, subject to certain qualifications required for continuation of funding, such as adherence to terms and conditions, compliance with financial and grants management regulations, and reasonable demonstration of results in establishing mentoring matches, measured against individual grantee caseload targets. After three years, if appropriations continue, grantees must re-apply. In both funding years, a variety of organizations with different prior experience and program goals were selected. Some were new organizations or had not previously provided mentoring services, yet presented program proposals whose merit stood up to intensive review. Fifty-two year one grantees have received a total of $17.8 million (for years one and two of their programs) to establish and run mentoring programs. In FY 2004, the additional 169 grantees received $36.7 million. ?? Grants Review Panelist training materials (FYSB documents) ?? Program Announcement (FYSB document) (Evidence Binder, Section 1.1) ?? FAQs on FY 2004 Program Announcement (FYSB document)

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: "The financial status report, the quarterly progress narratives, and the quarterly caseload data reports provide detailed information about grantee finances, activities, and performance. These reports are scrutinized by program staff in order to identify weaknesses and provide targeted technical assistance. Now that all grantees have programs in place, a monitoring tool and site visit protocols are being tested. A number of pilot site visits have been conducted in Region VI, and these have contributed to the evolution of a systematic approach. An additional series of site visits began in July, 2005 when FYSB staff went into the field to review program sites in Florida where there had been indicators of potential problems. Visits to Pennsylvania, Minnesota, and California have also taken place, generally on a targeted basis upon receipt of information suggesting there may be performance or compliance issues. Technical assistance services will be provided on a national basis by a competitively-selected organization based on an RFP issued in summer, 2005 (See Section 3.7.). Much direct knowledge of grantees will be available through communications, assessments, and other exchanges implemented under the technical assistance program. Activities under consideration include conferences, site visits, and a peer exchange program to pair successful grantees with struggling ones. See also discussion under Sections 3.1 and 3.3.

Evidence: ?? Progress Report, Financial Status Report, Online Data Collection Instrument, and Quarterly caseload analysis tool (Evidence Binder for Section 3.1.) ?? Monitoring Protocol/Outline for Pilot Site Visits (FYSB document) ?? List of pilot monitoring sites (other sites in Florida, California, and other States are being added on an ongoing basis) (FYSB document) ?? RFP for MCP Technical Assistance Resource (FYSB document) (Evidence Binder, Section 3.7)

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: All performance data is collected quarterly and will appear in the ACF Performance Plan on an annual basis. All ACF annual performance plans are found on the ACF website. At this time, the first full year of performance data collected under OMB approval is not yet at its end. At the end of the fiscal year, the data will be published in the Performance Plan. Other information about MCP is already disseminated by FYSB's National Clearinghouse for Families and Youth and in materials for conferences and other events. The Clearinghouse website is used to share information about all FYSB programs and about positive youth development with youth, parents, communities, researchers, service providers, advocates, policy makers and the general public. The technical assistance provider will use performance information in strengthening the program. Individual grantees will be able to cite their performance, if they wish, in applications to other funding sources or in local publicity. They will already have access to their own data through their ACF Online Data Collection accounts and from internal recordkeeping; however, FYSB's comprehensive data will allow them to compare themselves with other sites or the national average.

Evidence:  Online Data Collection Instrument and Quarterly caseload analysis tool, (FYSB document (Evidence Binder, Section 3.1)  MCP excerpt from FY 2006 Performance Plan (as revised) (FYSB document) (Evidence Binder, Section 2.1) Websites:  National Clearinghouse on Families and Youth, www.ncfy.com  ACF Performance Plan website:  www.acf.hhs.gov/programs/opre/acf_perfplan/ann_per/index.html

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: As a very new program, MCP has no possibility of demonstrating progress. The information below and under evidence is provided to show the quality of efforts to date. Companionship with caring adults Goal: By the end of FY 2007, 100,000 children of prisoners will have been in mentoring relationships MCP has been in existence at the service delivery stage for less than two years (and only has performance data for the first two quarters of FY 2005 collected systematically under OMB approval). During the first year of the program (52 grantees), when data collection approval was under OMB review, narrative progress reports were reviewed to gather caseload information. The data that are available indicate that progress is being made in critical dimensions of the mentoring relationship. The average caseload of children in matches per grantee has grown at a very aggressive quarterly rate from the first year to the second quarter of the second year. While this rate will certainly level off, with the collaborations that have been established with the Corporation for National and Community Service (CNCS) and others (See Section 3.5), it is possible to reach the President's goal of 100,000 children by the end of FY 2007. The annual targets include the participation of FYSB's partners in serving children of prisoners with mentors. The current level represents the caseload at a time when more than 75% of grantees have been in operation for only two quarters. Performance has improved exponentially as grantees gain experience, become known in their communities, and receive technical assistance from FYSB. There will also be an added boost in the third year of each group when matching resources double from 25% to 50%. Sustainability of relationships (beyond the minimum commitment) Goal: By the end of FY 2007, 20% of the MCP caseload will have been in sustained mentoring relationships Success in this measure is based on establishing a healthy proportion of sustained relationships (currently 20%) and maintaining the current percentage during a period of potentially rapid growth (in the denominator).

Evidence: Baseline: 2,099 matches, 1st Quarter, FY 2004 FY 2005: 39,000 (cumulative) FY 2006: 72,000 (cumulative) FY 2007: 105,000 (cumulative) In the companionship measure, a linear progression of 33,000 cases new each year was established for budgetary purposes. Actual performance is likely to be exponential. (See discussion under section 2.4)  1st Quarter, FY 2004 actual (FY 2003 grantees): 669  4th Quarter, FY 2004 actual (FY 2003 grantees): 2099  1st Quarter, FY 2005 actual (all grantees): 4407  2nd Quarter, FY 2005 actual (all grantees): 6257 In the 1st Quarter of FY 2004 (the first quarter of performance in the history of the program, prior to information collection but based on narrative information in quarterly progress reports), the FY 2003 grantees carried a caseload of 669 mentoring relationships. 109 of these were eligible children being served in an established program prior to MCP. 51 out of 52 of the FY 2003 group of grantees reported in the 1st and 2nd Quarters of FY 2004. (The 3rd and 4th quarters were transitional, as the new data instrument was circulated, refined, and implemented upon OMB approval.) This same group was managing 2,099 cases as the 1st Quarter of FY 2005 began. Sustainability of relationships (beyond the minimum commitment): Mentees in active mentoring relationships that have already lasted more than twelve months as a percent of the entire caseload Baseline: 16% as of the 2nd Quarter, FY 2005 (FY 2004 annual data not available) FY 2005 Target: 16% FY 2006 Target: 18% FY 2007 Target: 20%  1st Quarter, FY 2005 actual: 18%  2nd Quarter, FY 2005 actual: 16% The growth of this group during FY 2005, although limited percentage-wise by the influx of new cases as the program expands, should nevertheless be strengthened by the passage of time as some of the existing 12 month cases continue into 13 months and more. ?? Quarterly Data, 1st and 2nd Quarters (FYSB document) (Evidence Binder, Section 3.1.) FYSB will release final data on an annual basis at the end of the fiscal year.

Does the program (including program partners) achieve its annual performance goals?

Explanation: As a very new program, MCP has no possibility of demonstrating progress. The information below and under evidence is provided to show the quality of efforts to date. Companionship with caring adults: Number of children of prisoners in one-to-one matches with caring adults who have been trained and screened by the MCP program and its partners Goal: By the end of FY 2005, 20,000 children of prisoners will be in mentoring relationships See data on actual performance for this measure to the right and discussion under Section 4.1 Duration of relationships so as to meet or exceed the 12-month standard: Mentees in mentoring relationships lasting more than eleven months and concluding as planned in the twelfth month or shortly thereafter plus the percent of mentees in active mentoring relationships that have lasted twelve months or more as a percent of the entire caseload Goal: By the end of 2005, 20% of MCP children of prisoners will have been in relationships meeting or exceeding the 12-month minimum Systematically collected data has only been available for this measure for the first two quarters of FY 2005. This measure decreased during the quarter because the denominator was increased by 2,218 altogether new cases that would have lasted at most only three months by the end of the 2nd Quarter. Our annual goal is to reach a level of 20% during FY 2005 despite influx. As noted in the discussion of efficiency under Section 3.4, "half of all volunteer [mentoring] relationships dissolve within a few months." For this reason, as well as being in a rapid growth stage when most mentoring relationships will be brand new, attaining a 20% target level for FY 2005 is ambitious.

Evidence: Companionship with caring adults: FY 2004 2,099 (Baseline: FY 2003 grantees) FY 2005 Target: 39,000 (cumulative) FY 2006 Target: 72,000 (cumulative) FY 2007 Target: 105,000 (cumulative, including partnerships) In the companionship measure, the linear progression of 33,000 cases new each year was established for budgetary purposes. Actual performance is likely to be exponential.  1st Quarter, FY 2004 (FY 2003 group) actual: 669  4th Quarter, FY 2004 (FY 2003 group)actual: 2,099  1st Quarter, FY 2005 (FY 2003 group)actual: 2,770  2nd Quarter, FY 2005 (FY 2003 group)actual: 3,149  1st Quarter, FY 2005 (FY 2004 group)actual: 1,458  2nd Quarter, FY 2005 (FY 2004 group)actual: 3,108  1st Quarter, FY 2005 (all grantees)actual: 4,407  2nd Quarter, FY 2005 (all grantees)actual: 6,257 Duration of relationships so as to meet or exceed the 12-month standard: Baseline: 19% (2nd Quarter, FY 2005,actual) FY 2005 Target: 20% FY 2006 Target: 30% FY 2007 Target: 60% Actual 1st Quarter, FY 2005: 22% (this includes 68 cases that were still active in their eleventh month during the quarter). 2nd Quarter, FY 2005: 19% (this includes 100 cases still active in their eleventh month during the quarter). ?? Freedman, M. (1993) in Rhodes, J. (2002). (op. cit. under Section 3.4) ?? Quarterly Data, 1st and 2nd Quarters (FYSB document) (Evidence Binder, Section 3.1) FYSB will release final data on an annual basis at the end of the fiscal year.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: As a very new program, MCP has no possibility of demonstrating progress. The information below and under evidence is provided to show the quality of efforts to date. Based on data from the "Companionship" measure and using a logical unit cost factor (average FYSB funding per completed mentor/mentee match), the program is becoming more efficient. This is based on the performance of the FY 2003 grantees, who are more experienced at this time than the FY 2004 group. $8.9 million was issued in FY 2003 grants to 52 grantees. In the first quarter, there were 669 matches ($13,303/match), some of which pre-existed the program. By the end of FY 2004, there were 2,099 matches ($4,240/match). In the first quarter of FY 2005, the FY 2003 group had 2,770 matches. In the second quarter the group had 3,149 matches. Although funding is issued for the entire year, the quarterly funding amount would be $2,225,000 (dividing the annual grants by four for the sake of illustration). On this hypothetical basis, the quarterly unit cost in the more experienced group went from $803/match to $707/match from the 1st to the 2nd Quarter. However, because a steady state has not been reached for the entire program, we have selected the measure discussed under Section 3.4. Efficiency Measure: the proportion of mentoring cases which last no more than three months as a percent of the total number of matches which ceased during the year (prematurely or on schedule) Goal: minimize matches of very short duration Efficiency is embodied in conserving the up-front investment in mentor recruitment and training by limiting the number of cases of very short duration. This measure is also based on the programmatic principle that long-term, quality relationships that persist throughout their intended lifespan of at least twelve months provide the best return on investment and are associated with positive behavioral and emotional outcomes.

Evidence: Efficiency Measure Baseline: 37% (the actual rate in the 1st Quarter, FY 2005) Target FY2005 Target: 30% FY 2006 Target: 25% FY 2007 Target: 20% Actuals 1st Quarter, FY 2005: matches lasting no more than three months, 156 divided by matches ending during the quarter, either prematurely or concluding as planned in the twelfth month 425  = very short duration cases as a percent of all ending cases 37% 2nd Quarter, FY 2005: matches lasting no more than three months, 104 divided by matches ending during the quarter, either prematurely or concluding as planned in the twelfth month: 348 = very short duration cases as a percent of all ending cases: 26% In the 2nd Quarter, this important rate, which embodies the reduction of a negative, was reduced significantly. Despite this promising performance, the current targets may be highly ambitious. "One half of all volunteer [mentoring] relationships dissolve within a few months." (Rhodes, 2002) The baseline rate may be low only because three quarters of the grantees had been in operation only during the first quarter, and any case still active at the end of the three-month quarter would be excluded from the rate. As more time accumulates, it may be apparent that target rates should be adjusted. ?? Rhodes (2002), page 57, citing M. Freedman. The Kindness of Strangers: Adult Mentors, Urban Youth and the New Volunteerism (San Francisco: Jossey-Bass, 1993.) ?? Quarterly Data, 1st and 2nd Quarters (FYSB document) (Evidence Binder, Section 3.1) FYSB will release final data on an annual basis at the end of the fiscal year.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: MCP compares favorably to the major recent Federal mentoring efforts (OJJDP's JUMP program and the Department of Education's Mentoring Program) and also to Amachi, the primary private sector program that provides mentoring for children of prisoners. Because these programs differ from MCP in their phase of development, scope and focus, data cannot be used to directly benchmark the progress of MCP; however, data from these programs provide insight into what is typical progress for a new mentoring program serving high-risk youth. Based on a review of the available data, MCP appears to be meeting, or surpassing comparisons for the number of new and lasting mentoring matches that have been made this early into its development. Within the first year-and-a-half of program start-up, 6,257 youth had been placed in mentoring relationships at MCP programs with a caring, screened, and trained adult. Over 92% of these relationships lasted at least 6 months. Although there are no directly comparable figures, data from the Amachi and JUMP programs provide a context that suggests that these figures can be counted as successes (see evidence section for Amachi and JUMP data). MCP has also taken aggressive steps toward research and evaluation that appear to surpass the early research efforts of the other major Federal mentoring programs. FYSB has designed and installed a fully operational online data collection system that, only four quarters after grants were awarded, was already providing quarterly data on promising performance. This was put in place no more than two years after the appropriation, with considerable input from grantees and researchers. FYSB continues to develop systems to collect outcome data and, with its contractor, has developed a research design for outcomes and impacts (Evidence Binder, Section 2.6). (See evidence section at right for specifics on DOE and DOJ research and evaluation efforts).

Evidence: recruiting and serving high-risk youth. OJJDP originally required its grantees to match from 50 to 60 youth each year over a period of three years; that goal was later reduced to 25 youth per site each year over three years. In just its first year of operation, FYSB was able to meet OJJDP's initial, ambitious goals for the JUMP sites by awarding 52 grants, with each site serving an average of 40 youth by the 4th quarter of fiscal year 2004. With the addition of 169 new sites (FY 2004 grantees), the average caseload remains at 40, although most new grantees started with zero cases. MCP's data on mentoring relationship duration compares favorably to available data from the Amachi demonstration project, which did exclusively serve children of prisoners. This data was collected over a period of 2 years on matches that had begun meeting more than 13 months prior to the collection of data for the study. Of those matches, 13 percent of the matches had ended within 5 months. In the MCP program, as of the second quarter of FY 2005, only 7.7 percent of matches had ended within 6 months. FYSB's research and evaluation plans and activities for MCP also appear to be ahead of the curve. At the time of the June 25, 2004 GAO report on Education's mentoring program, there was no outcome data available, and plans for an evaluation were undefined. Since then the Department of Education has required grantees to report their progress over the life of the grant on measures such as relationship duration and youth academic performance, but ongoing caseload data is not collected or available at the Federal level (unlike FYSB's data). Similarly, six years into OJJDP's funding of the JUMP program, only limited data on program performance were available. Measures were limited to "how satisfied mentors and youth were with their mentoring experience;" the focus of FYSB's data collection goes beyond program "satisfaction" to evidenced-based indicators of positive mentoring relationships and the outcomes with which they are associated. ?? Government Accountability Office. (October, 2001). OJJDP Grantee Reporting Requirements and Concerns. (GAO-02-23). Washington, DC: Government Printing Office. (not in Evidence Binder; available on request) ?? Government Accountability Office. (June, 2004). Student Mentoring Programs: Education's Monitoring and Information Sharing Could Be Improved. (GAO-04-581). Washington, DC: GPO. (not in Evidence Binder; available on request) ?? 2003 JUMP Annual Report. (2003). Washington, DC: Office of Juvenile Justice and Delinquency Prevention. Retrieved on June 22, 2005 at Information Technology International Web site: www.itiincorporated.com/AnnRpt/. (not in Evidence Binder; available on request) ?? Jucovy, L. (2003). Amachi: Mentoring Children of Prisoners In Philadelphia. Philadelphia: Public/Private Ventures and the Center for Research on Religion and Urban Civil Society. Retrieved on June 22, 2005 at www.ppv.org/ppv/publications/assets/167_publication.pdf. (Evidence Binder, 2.4)

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: FYSB understands that conducting expensive, time-intensive evaluations on a new program is not a wise use of scarce research/administrative dollars since it measures program start-up, rather than the actual merit of the established program. Therefore, FYSB has put in place research efforts that more appropriately correspond to the level of development of this program. For example, FYSB awarded a contract to a reputable, independent evaluation firm to conduct a descriptive study of the program, develop a design, and advise FYSB on how to approach evaluation when the program reaches a steady state. The descriptive study of the program is a primary topic of a Report to Congress, which is expected to be submitted by the Secretary of Health and Human Services to Congressional Leadership and Ranking Members in the fourth quarter of FY 2005. Among other topics, the draft report addresses MCP design, strategy, implementation, current operating status, and characteristics and program practices of the participating grantees. The Report was written pursuant to Section 439(g) of the Statute and based on available 1st and 2nd Quarter, FY 2005 data. Data based upon a 95% response rate show that MCP is developing mentoring relationships of the duration and intensity needed to result in positive youth outcomes. The data indicate that MCP is successfully creating mentoring relationships that research indicates will likely lead to positive youth outcomes. Until outcome data from the planned evaluation is available, FYSB is evaluating the success of this program by analyzing indicators of positive youth outcomes. MCP has the benefit of being backed by approximately 20 years of research on mentoring. Where we cannot yet report outcomes, we rely on evidenced-based standards of quality mentoring. For example, prior research has shown that mentors and youth who meet relatively often and who remain in relationships for a relatively long time, tend to have quality relationships. These quality relationships lead to the type of positive long-term outcomes, such as the behavioral and academic outcomes, that will be the focus of FYSB's evaluation. (Abt Associates Design Study)

Evidence: ?? 1,698 children began new, first-time matches during the first quarter of FY 2005, and 4,407 children were in matches during the quarter (including newly-matched children). ?? In the second quarter the caseload grew to 6,257 children in matches. ?? In the first quarter about 20 percent of children had been in relationships of at least 12 months (917 out of 4,407) or longer. The number of 12 month relationships is expected to grow rapidly as the program's existing caseload accumulates more time in relationships. The rapid influx of brand-new relationships among the FY 2004 grantees could lower this proportion for a period of time. For example, in the second quarter 16 percent of the matches had endured past the eleventh month (1,024 out of 6,257). ?? Only 3 percent of matches active during the second quarter ended prematurely after having lasted six months or less. ?? Grantees new to mentoring took time to get established, hire staff and recruit youth and mentors. By the end of the first quarter of FY 2005, 60 percent of the reporting grantees had at least one child in a match. By the second quarter, the level had increased to about 85 percent of reporting grantees. ?? Nearly 70 percent of the children in matches had 12 or more hours of regular contact (e.g., one hour meetings/contacts every week for 12 months or two hour meetings/ contacts every two weeks, etc.). About 28 percent had 24 or more hours of regular contact during the quarter. (The program standard is weekly contacts and meetings at least every two weeks.) ?? 3,185 children were on waiting lists during the quarter. 26 days was the average waiting time until a child was assigned his or her first mentor after parental consent had been established. ?? At the end of the quarter, 1,800 mentors were trained and screened but not yet matched. 259 matches ended ahead of schedule with mentors being twice as likely as parents or guardians to initiate the termination. Children and agencies were least likely to do so. 1st Quarter FY 2005 breakdowns between FY 2003 and FY 2004 grantees for key factors: FY 2003 FY 2004 Total Children in new, first-time matches 698 1,000 1,698 Children in matches (including 1st time) 2,949 1,458 4,407 Children on waiting lists 932 2,253 3,185 Matches ending ahead of schedule 184 75 259 Matches lasting 12 months or more 767 150 917 ?? Overview of Mentoring and the Mentoring Program (excerpt from evaluation design study, Abt Associates, Cambridge, MA) (FYSB document)