Detailed Information on the
Refugee Social Services Assessment

Office of Refugee Resettlement (ORR) will establish targets for unit costs as an annual measure of cost-effectiveness. Milestone: Review and analyze survey presented at the FY 2007 Annual ORR Consultation that was intended to capture states' mechanisms for reporting unit cost

Conducting an independent and quality evaluation analyzing employment and self-sufficiency outcomes in three program sites. Milestone: Share findings and promising practices of the Lewin evaluation with states and Wilson-Fish agencies during monitoring and technical assistance visits.

Collaborating with partners to set measurable goals that are linked to the budget. Milestone: Set monitoring and technical assistance criteria, including performance outcomes and budget, to determine CY09 monitoring and technical assistance schedule.

Improving strategic planning to ensure that systems are in place to identify deficiencies. Milestone: Revise formula funding allocation to pre-FY 2006 system.

Collaborating with partners to set measurable goals that are linked to the budget. Milestone: Office of Refugee Resettlement (ORR)/State Coordinators deliberate on proposed changes to annual performance measures presented by Economic Self-Sufficiency and Integration workgroups at ORR Annual Consultation.

The Budget includes funds ($2 million) for the Office of Refugee Resettlement (ORR) to conduct independent and quality evaluations.

Office of Refugee Resettlement (ORR) will establish targets for unit costs as an annual measure of cost-effectiveness. Milestone: The Lewin Evaluation contract will research the various program variables that impact performance and our ability to measure effectiveness in this area.

Conducting an independent and quality evaluation analyzing employment and self-sufficiency outcomes in three program sites. Milestone: Survey 900 refugees (300 refugees in three selected locations: Houston, TX, Miami, FL, and Sacramento, CA).

Conducting an independent and quality evaluation analyzing employment and self-sufficiency outcomes in three program sites. Milestone: Collect, process, and analyze ORR and state administrative data on refugee employment and economic self-sufficiency.

Collaborating with partners to set measurable goals that are linked to the budget. Milestone: Consult with the Office of Planning, Research and Evaluation (ACF) and Office Management and Budget (OMB) on proposed changes.

Collaborating with partners to set measurable goals that are linked to the budget. Milestone: Conduct a workshop at the FY 2008 Annual ORR Consultation to further discuss changes made to the Performance Report (ORR-6) and to provide clarifications.

Conducting an independent and quality evaluation analyzing employment and self-sufficiency outcomes in three program sites. Milestone: Final evaluation report published and released to public.

Measure: Increase the percent of refugees entering employment through ACF-funded refugee employment services.

Explanation:The target of 65% in 2012 is a technical correction to the previous target of 85%. Prior to FY 2002, this measure reported on numbers of refugees rather than percentages. The annual target for this measure is to increase by 3% over the prior year's actual performance.

Measure: Increase the percentage of 90-day job retention as a subset of all entered employment.

Explanation:

Measure: Increase the percent of entered employment with health benefits available as a subset of full-time job placements by a percent of the prior year's actual percentage outcome.

Explanation:This measure was reported as a percentage, rather than a number, starting in FY 2002.

Measure: For refugees receiving Temporary Assistance for Needy Families (TANF) or other forms of Federal cash assistance, shorten the length of time from arrival in teh U.S. to achievement to self-sufficiency. (Transitional and Medical Services, Refugee Social Services)

Explanation:This efficiency measure focuses on the 180-day (six month) self-sufficiency of refugees in the MG program. The MG program awards approximately $50 million in federal funding to grantees each calendar year, serving approximately 25,000 refugees annually. To calculate performance on this measure, the number of refugees who are self-sufficient at 180 days is divided by the federal award (in millions of dollars) to grantees for that year. The measure is adjusted for inflation with a baseline year of 2004, using the Consumer Price Index Inflation Calculator.

Is the program purpose clear?

Explanation: The goal of the refugee program is to assist refugees to attain economic self-sufficiency as soon as possible after arrival.

Evidence: Authorization in Section 412(a)(1) of the Immigration and Nationality Act

Does the program address a specific interest, problem or need?

Explanation: All persons admitted as refugees while in the U.S. are eligible for refugee benefits. Federal resettlement assistance to refugees is provided primarily through the State-administered refugee resettlement program. The Office of Refugee Resettlement (ORR) formula grants program assists refugees in obtaining the skills they need for economic self-sufficiency by providing employment services, job training, and English Language Training (ELT).

Evidence: According to the Refugee Resettlement Program FY 200 Report to Congress, the U.S. admitted 72,489 refugees and Amerasian immigrants in FY 2000. An additional 17,871 Cuban and 1,570 Haitian nationals were admitted as entrants, for a total of 91,960 arrivals. About 63 percent of these refugees spoke no English and required intensive English language and job training.

Is the program designed to have a significant impact in addressing the interest, problem or need?

Explanation: The program is designed for and specifically funded to provide employment and social adjustment services to refugees to assist them in learning about the U.S. culture and labor market and to place refugees in jobs. The Federal government provides 100 percent of the funds to State governments and private, non-profit agencies that are responsible for providing services.

Evidence: According to ORR's Fall 2000 annual survey of refugees who have been in the U.S. less than five years, about 68 percent of refugees age 16 or over were employed as of September 2000, as compared with about 65 percent for the U.S. population. The total cost of ORR formula grants to States in FY 2001 was $137 million. Social services formula allocations totaled $92 million and targeted assistance allocations totaled $44 million.

Is the program designed to make a unique contribution in addressing the interest, problem or need (i.e., not needlessly redundant of any other Federal, state, local or private efforts)?

Explanation: No other program provides formula grants to States to address the needs of refugees for employability services. Without these formula grants, States could not provide the specialized, linguistically and culturally appropriate employment, training and ELT services to newly arrived refugees that prepare them to work in the U.S. and to support themselves as soon as possible after arrival.

Evidence: ORR provides various resettlement services, cash and medical assistance, for refugees in addition to the employment services being evaluated here. Total ORR funding is around $450 million. The INS inspects and admits refugees, and the State Department provides grants for reception and placement, however, HHS is the only agency that provides resettlement services for refugees.

Is the program optimally designed to address the interest, problem or need?

Explanation: This program has achieved significant accomplishments, including: State flexibility in designing programs of assistance and services, family self-sufficiency plans for each case, on-site and desk monitoring, technical assistance, and sufficient funding to allow States to respond quickly to new refugee populations and needs.

Evidence: The program is centrally administered by ORR and ORR conducts on-site and desk monitoring of States' results. States have direct access to ORR State analysts. ORR published a final rule in March 2000 which gave States flexibility in designing their programs under ORR regulations at 45 CFR Part 400. ORR is now seeing States' responses to the final rule regulatory flexibility in terms of better coordination between cash and employment services.

Does the program have a limited number of specific, ambitious long-term performance goals that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: Although ORR currently does not have a 5-year strategic plan or long-term goals for the program, some of ORR's activites are addressed in ACF's five-year plan. More importantly, however, OMB and ORR recently developed ambitious long-term outcome goals that link to the mission of the program.

Evidence: ORR's newly developed long-term outcome goal that has been revised in the FY2004 GPRA plan is to have an 85 percent entered employment rate (EER). An EER is the ratio of refugees entering employment relative to the number of refugees receiving employment services. States with an EER of less than 50% will be expected to achieve a 5% annual increase in this rate. States with an EER of greater than 50% will be expected to achieve a 3% annual increase in this rate. Average national EERs will be calculated a) for all states, b) for all except the 2 states with the largest caseloads, and c) for each of the two cohorts listed above.

Does the program have a limited number of annual performance goals that demonstrate progress toward achieving the long-term goals?

Explanation: In FY 1996, ORR developed specific goals in consultation with the States and these are updated annually in GPRA plans. Improvement along four specific goals have been identified for refugees: entered employment, average wage at placement, employment retention, and entered employment with health benefits available.

Evidence: Some of ORR's FY 2004 goals shown in the GPRA Annual Performance Plan are: (1) Increase the number of refugees entering employment through ACF-funded refugee employment services by at least 3% annually from FY 1997 actual performance. (2) Increase the number of entered employments with health benefits available as a subset of full-time job placements by 3% annually from FY 1997 actual performance. (3) Increase the number of 90-day job retentions as a subset of all entered employments by at least 3% annually from FY 1997 actual performance.

Do all partners (grantees, sub-grantees, contractors, etc.) support program planning efforts by committing to the annual and/or long-term goals of the program?

Explanation: All States report to ORR on their program performance on a quarterly basis. Quarterly performance is tracked and compared to the Annual Outcome Goal Plan developed by ORR in partnership with each State. Desk and on-site monitoring and the provision of technical assistance are tools used by ORR to assist grantees in improving outcomes.

Evidence: Quarterly Performance Report (QPR) (ORR-6) (OMB No.0970-0036).

Does the program collaborate and coordinate effectively with related programs that share similar goals and objectives?

Explanation: Refugee resettlement policies and activities are coordinated with the U.S. Department of State, (DOS) State and community agencies, the Immigration and Naturalization Service, the Social Security Administration, the U.S. Department of Agriculture, Food and Consumer Service, as well as with TANF, Medicaid, and other programs within HHS.

Evidence: Most of the persons eligible for ORR's refugee program benefits and services are refugees resettled through the Department of State's refugee allocation system under the annual ceiling for refugee admissions established by the President through a consultative process. ORR participates on several DOS interagency workgroups and reviews reception and placement applications. ORR coordinates policy issues with DOJ/INS, SSA, and DOS as appropriate. ORR also conducts annual consultations with its resettlement partners: States, voluntary agencies and other non-profit organizations serving refugees.

Are independent and quality evaluations of sufficient scope conducted on a regular basis or as needed to fill gaps in performance information to support program improvements and evaluate effectiveness?

Explanation: ORR does not conduct long-term, independent evaluations for this program. However, ORR does complete an Annual Survey of Refugees regarding refugees' education and skills, employment potential, English competence and health. In addition, ORR conducts on-site monitoring of selected States and other grantees to help them achieve improved client employment and self-sufficiency outcomes. ORR also targets States that have large refugee populations and that receive significant refugee program funding for monitoring. In monitoring, ORR assists States and grantees to identify strategies to improve outcomes on ORR performance measures and provides technical assistance on implementing program improvements.

Evidence: Last HHS Inspector General Report dates back to 1995. No schedule of program evaluation exists.

Is the program budget aligned with the program goals in such a way that the impact of funding, policy, and legislative changes on performance is readily known?

Explanation: Due to variability in the number and timing of refugee arrivals in need of employment and social services, the budget cannot be directly aligned with program goals. States can provide employment services to refugees with these funds for up to 5 years after arrival. States' allocations are determined by the number of refugee arrivals per State.

Evidence:  

Has the program taken meaningful steps to address its strategic planning deficiencies?

Explanation: ORR does not have a system for identifying and correcting deficiencies in its strategic planning process. However, through this PART process, ORR has set specific and ambitious long-term goals that were not previously formulated. In addition, ORR is moving towards completing program evaluations focused on improving program performance.

Evidence: ORR staff participate in a number of workgroups with INS, DoS, and the Refugee Council USA to seek solutions to problems. In addition, ORR plans to hold a series of consultations with State Refugee agencies concerning long-term performance goals. Activities would begin in late winter and/or early spring 2003.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The Quartelry Performance Report (QPR) is the established reporting and data collection instrument for capturing data on States' performance. ORR uses performance data to plan program monitoring. Desk monitoring and tracking of QPR data occur quarterly. Data are validated by periodic on-site monitoring, in which refugee cases are randomly selected and reviewed. Outcomes reported by service providers are verified with both employers and refugees to ensure accurate reporting of job placements, wages and job retention.

Evidence: QPR. ORR uses its performance data to target States with low performance for on-site and/or intensive desk monitoring; and provides technical assistance to States with low performance. For example, as a result of ORR monitoring, a sub-recipient of social services in San Diego, CA was terminated as a provider due to poor perfromance and a corrective action plan was implemented for Indiana as a result of monitoring.

Are Federal managers and program partners (grantees, subgrantees, contractors, etc.) held accountable for cost, schedule and performance results?

Explanation: Federal accountability is reflected in the Senior Managers' Perfromance Contracts with the Assistant Secretary. Federal Managers identify several discrete goals on which they will be evaluated.

Evidence: The ORR Director's Performance Contract.

Are all funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: Form SF-269 is used to report actual expenditures made by the States consistent with the approved State Plans for Refugee Resettlement and in accordance with all applicable statutes and regulations. Financial analysts in ACF staff offices track grantees' draw-downs and liquidations of obligations on a quarterly basis. Grantees respond to single audits and the ORR Director responds to audit findings as the responsible entity.

Evidence: ORR staff and staff in the ACF Office of Administration examine quarterly expenditure reports. On-site reviews examine financial management systems of grantees and test transactions.

Does the program have incentives and procedures (e.g., competitive sourcing/cost comparisons, IT improvements) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: ORR does have a contract for program monitoring and evaluation of effectiveness, and contract staff assist ORR staff in monitoring programs and validating outcome data. However there are no incentives for States to improve performance. ORR indicates that States voted not to set up incentives or penalties, rather to publish data on each State's annual targets and actual performance, which are in the ORR Annual Report to Congress. The publicity serves as an incentive for improved performance.

Evidence: ACF Performance Plan. ORR Annual Report to Congress. States are required to provide information to ORR regarding expenditures to achieve outcomes quarterly and annually. ORR uses these data to compute unit costs per placement as a measure of cost effectiveness. In each Annual Report, ORR reports the range of costs per job placement for States and describes how unit costs function as a measure of cost effectiveness. ORR also uses these data to direct its annual goal plan negotiations with States by asking States to hold the unit cost constant, which often results in increased goals for the number of entered employments to be achieved.

Does the agency estimate and budget for the full annual costs of operating the program (including all administrative costs and allocated overhead) so that program performance changes are identified with changes in funding levels?

Explanation: The allocation formula for social services is set in statute at section 412 (c) (1) (B) of the INA which requires that funds for grants be allocated among the States based on the total number of refugees who arrived in the U.S. not more than 36 months before the beginning of the FY. The allocations for Targeted Assistance formula are based on section 412 (c) (2)(B)(ii) of the INA which requires that 95% of the amount of the grant award is made available to the county or other local entity that qualified for the allocation.

Evidence:  

Does the program use strong financial management practices?

Explanation: The ACF Audited Financial Statements for the past three years have demonstrated that ORR does not have any material weaknesses. ORR staff review and analyze the Quartely Performance Report and SF-269 reports submitted quarterly by State grantees. The issuance of grant awards is contingent upon submission of an Annual Services Plan to ORR. States are also subject to annual single audit requirements. This program is subject to numerous congressional earmarks, which has complicated financial management processes.

Evidence: 1) SF-269 2) ORR Annual Services Plan. 3) FY 1999, 2000, and 2001 ACF Audited Financial Statements.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: ORR does have a system for evaluating the effectiveness of its management through performance contracts, the EPMS system, and the FMFIA requirements, the ACF annual audited financial statement.

Evidence: FY 2000 Federal Managers Financial Integrity Report

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: Grantees are required to file program and financial reports quarterly which describe activities undertaken during the quarter, specifically to accomplish the yearly goals and objectives the State has proposed. Monitoring activities undertaken during the quarter are also reported.

Evidence: ORR-6, SF-269. monitoring reports, and corrective action plans

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: The program collects unduplicated annual performance data once a year. These data are published in the ORR Annual Report to Congress with State by State perforrmance comparing a State's last year's actual performance on each of ORR's six measures to the current year's performance.

Evidence: ORR Annual Report to Congress and on ORR website.

Has the program demonstrated adequate progress in achieving its long-term outcome goal(s)?

Explanation: ORR did not previously have long-term outcome goals set and established in their annual GPRA plan, however through this PART process, they established very aggressive targets.

Evidence: Revised FY 2004 GPRA Plan

Does the program (including program partners) achieve its annual performance goals?

Explanation: Annual goals are in place and States strive to achieve these goals. However, achievement of annual goals is contingent upon entering refugee populations (i.e. some populations have more barriers to employment than others).

Evidence: GPRA Plans; Annual Reports to Congress

Does the program demonstrate improved efficiencies and cost effectiveness in achieving program goals each year?

Explanation: In annual negotiations with States, ORR strives for increased outcomes and steady or decreasing unit costs per entered employment. Unit costs are tracked and reviewed based on annual performance, however meeting performance targets to reduce unit costs are not part of ORR's annual goals.

Evidence: FY 2000 Annual Reports; State reports

Does the performance of this program compare favorably to other programs with similar purpose and goals?

Explanation: There are no other programs with similar purpose and goals. This is the only domestic program funded to meet the employment needs of refugees in a linguistically and culturally appropriate manner. Other mainstream employment programs do not provide services in a way that refugees can understand. However, the costs of providing these services to refugees are not out of line with other employment programs that serve the mainstream caseload.

Evidence: The Refugee Act of 1980, 45 CFR Part 400 , and policy guidance of the Director, ORR.

Do independent and quality evaluations of this program indicate that the program is effective and achieving results?

Explanation: There are no legislative requirements nor are special funds available for this purpose. There are, however, internal assessments in place that reveal that the program is effective and achieving results. Refugee Annual Survey data from FY 1993 to FY 2001 indicate that the refugee "employment to population ratio" (EPR) increased by 169%. The 2001 EPR for refugees is equal to the EPR for the U.S. population.

Evidence: FY 2000 ORR Annual Report and Annual Survey