 Continuous Evaluation
This report benefited from the contributions of several persons at the Office of Refugee Resettlement and other offices in the U.S. Department of Health and Human Services, and from the Lewin Group Evaluation team. Pamela Green-Smith and Susan Kyle of ORR provided valuable guidance, data, reports, and knowledge throughout. They reviewed several drafts of the report and their suggestions and comments are gratefully acknowledged. Martha Newton, Nguyen Van Hanh, Josh Trent, and Ken Tota provided important conceptual and policy context, and Timothy Forbes, Gayle Smith, April Young, and James Perlmutter participated in working sessions during various phases of the study, helping us understand some of the more complex aspects of the rules and regulations regarding refugees. In addition to these officials, Emily Ball and Moushumi Beltangady reviewed drafts of the report and provided their expert comments. And Richard Jakopic, who passed away last year, is fondly remembered for his advice and input on this and many other studies.
Thanks also go to other researchers on the Lewin Group’s evaluation, under which this report was produced. Their contributions were essential to this study. In particular, Mary Farrell and Michael Fishman, both of the Lewin Group, provided critical input throughout, particularly on conceptual issues, and they carefully reviewed drafts and provided useful comments that improved the report. Data analysis conducted by Bret Barden and Michael Mueller of the Lewin Group contributed importantly to several sections. Burt Barnow of Johns Hopkins University reviewed and provided helpful suggestions on the evaluation design section. In addition to Mary Farrell and Mike Fishman, Nancy Pindus and Randy Capps of the Urban Institute and Sam Elkin of the Lewin Group shared important experience about program management and operations and their insights from the field form the basis for examples used in this report.
I. Introduction and Purpose
The Refugee Social Service (RSS) and Targeted Assistance Formula Grant (TAG) Programs provide services to refugees, asylees, Cuban/Haitian entrants, Amerasians, and victims of a severe form of trafficking with the objective of helping them achieve economic self-sufficiency soon after entering the United States. The Office of Refugee Resettlement (ORR) within the U.S. Department of Health and Human Services (DHHS) administers these programs and sponsored an evaluation to assess how program services are delivered and how refugees who receive these services fare over time. The Lewin Group and its partners, the Urban Institute, Johns Hopkins University, National Opinion Research Center (NORC), and Southeast Asia Resource Action Center (SEARAC) conducted the evaluation focusing on three sites: Houston, Texas; Miami, Florida; and Sacramento, California. This Continuous Evaluation report serves as an extension of the evaluation of the RSS and TAG programs, outlining ways that ORR can better plan for and institutionalize evaluation and accountability throughout the range of refugee resettlement programs. The intent of this paper is to present a range of options ORR might consider to complement existing performance management and evaluation strategies.
The federal government determines how many refugees will be admitted to the United States. From FY 2002 to FY 2007, the proposed annual ceiling was set at 70,000, and the FY 2008 ceiling will be set at 80,000.1 In the past five years, a total of between 27,000 and 75,000 persons have been admitted annually as refugees. In addition, certain Cuban and Haitian entrants (approximately 25,000 annually) are allowed to enter the country directly (e.g. parolees, asylum seekers). Other populations eligible for ORR-funded services include asylees (approximately 24,000 a year), certain Amerasians, and victims of a severe form of trafficking (up to 1,000 a year). For ease of reference, this report generally uses the term “refugees” to refer to all such groups that qualify for ORR services, except where delineation is necessary.
To assist refugees settle in the United States, ORR funds a number of programs that provide economic support, social services, health services, and employability services designed to aid individuals and families achieve rapid economic self-sufficiency. Most of the actual service delivery occurs through a broad network of providers including state and local agencies, mutual assistance associations ( MAAs), and voluntary agencies (Volags) that have established relationships with the Department of State for reception and placement services to refugees.
The largest federal refugee programs in terms of funding are the Refugee Cash Assistance (RCA) ($36.5 million allocated in FY 2006) and Refugee Medical Assistance (RMA) ($82.0 million allocated in FY 2006) programs, whereby the federal government reimburses states for the costs of cash and medical benefits. In addition, ORR allocates funds according to Congressionally-established formulas to states and localities to provide a broad range of employability services to help individuals obtain employment and achieve economic self-sufficiency and social adjustment as quickly as possible. RSS funding for FY 2006 was about $83.4 million and the TAG funding was about $43.7 million; with a small percentage of the total RSS and TAG funds made available for discretionary allocation by ORR.
RSS- and TAG-funded services are specifically intended to improve economic self-sufficiency and social adjustment, primarily through employability and support services. ORR has established extensive policies to monitor the results and performance of its programs in keeping with the regulations emanating from the Government Performance and Results Act (GPRA), the President’s Management Agenda, and the Program Assessment and Rating Tool (PART). However, there has been less focus on making evaluation, on a broader scale, a part of the refugee program. There have been very few internal or independent evaluations of ORR programs, and those that have been conducted have not been adequate in rigor or scope. Strong program accountability requires solid monitoring, reporting, and evaluation strategies.
Therefore, as part of ORR’s ongoing efforts to improve performance management strategies, this paper provides a framework for ORR to consider for continuously evaluating RSS- and TAG-funded employability services. The discussions and options draw from information and findings from the current Lewin evaluation of ORR programs2, review of recommendations of the Economic Self-Sufficiency Workgroup, discussions with ORR officials and staff, and review of ORR PART and GPRA materials. The Economic Self-Sufficiency Workgroup, established by ORR in 2006, consists of ORR staff, state coordinators, representatives of Wilson/Fish programs, local and national Volags, MAAs, an employment technical assistance provider, and the Department of State. The Workgroup reviewed the operating definition of self-sufficiency, and engaged in extensive discussions about current performance measures, alternative measures, reporting, timing, and other technical issues. Input from the workgroup is incorporated into ORR’s recent proposed guidelines.3 Information from the workgroup was reviewed and used in this report as well.
The intent of this paper is to present a range of options ORR might consider to complement existing performance and evaluation strategies and the proposed guidelines related to economic self-sufficiency.
II. Conceptualizing Continuous Evaluation
Over the past decade there has been an increasing focus in the federal government on managing for results. The Government Performance and Results Act (GPRA) requires all agencies to develop annual performance plans with clear goals, and then track progress towards goals set. Since 2001, the President’s Management Agenda (PMA) further specifies that each agency focus on continuously improving five areas of management:4 strategic management of human capital, competitive sourcing, improved financial performance, expanded electronic government, and budget and performance integration. The fifth area, budget and performance integration, expanded upon the GPRA concepts to promote improvements in defining, measuring, and monitoring performance results, to encourage continuous improvement, and to inform budget decisions with performance results. The Program Assessment and Rating Tool (PART) which supports the PMA, is intended to ultimately assess all federal programs, and is designed to focus on goals and outcomes to help evaluate programs’ overall effectiveness and to improve performance over time. There are four categories of factors included in PART: program purpose and design, strategic planning, management, and results. Each year, selected programs are identified by agencies and the Office of Management and Budget (OMB) for PART assessments, and PMA Scorecards and narrative reports are prepared and made available to the public. About half of the total PART score for each program is based on the results portion of the Assessment, which uses performance and evaluation data to determine progress towards achieving annual targets and long-term efficiency objectives. PART is intended to complement GPRA by operationalizing and integrating specific planning, management, and results activities to improve performance.
Program evaluation enters into PART in several ways. First, one of the dimensions in the strategic planning section inquires whether the program has independent evaluations, and whether the program has clear performance measures, including ambitious baselines and targets. Second, the management section requires evidence of program efficiency, including having measures established and procedures in place to achieve efficiency. Third, the results section requires programs to show evidence of continuous progress towards achieving goals, efficiency, and performance.
Thus, both GPRA and PART require agencies to have systems in place that each program can use for establishing goals and targets, measuring performance and results, and continuously tracking progress over time towards improved results. Program management through program planning processes and management information reporting are critical for conducting these assessments at the aggregate level, and program evaluation activities help establish a baseline of results, determine the effectiveness and efficiency of specific program activities, and track progress over time.
ORR has established extensive policies to monitor the results and performance of its programs, and uses this information to continuously improve programs, in keeping with GPRA and PART requirements. Because ORR programs are collaborative efforts among state and local agencies, Volags, and local service providers, it has been important to institutionalize an ongoing performance management and monitoring strategy into program management at the federal level and, operationally, at the state, local, and service delivery levels.
However, there has been less focus on making evaluation, on a broader scale, an integral part of the refugee program. There have been very few internal or independent evaluations of ORR programs, and those that have been conducted have not been adequate in rigor, regularity, or scope. True program accountability requires this type of regular evaluation, in addition to solid monitoring and reporting practices. This report provides a framework for more intentionally integrating the concept of evaluation into the refugee resettlement system.
In other words, the nature and types of ORR programs, along with the PART and GPRA requirements to which ORR is held, suggests a three-pronged conceptual framework for continuous evaluation, shown graphically on Exhibit 1. The general parameters of this framework are already in place, representing the confluence of ORR’s programmatic management and oversight responsibilities, its broad mission of successfully resettling refugees, and its obligations to achieve results at the national level.
Two obvious assumptions underlie the framework. First, state performance is important in the context of PART; how states achieve their goals affects the achievement of ORR’s PART goals. Second, evaluation is viewed as an integral part of overall program performance management, not as a separate activity, which is discussed in Section III.A. Formal evaluations of program outcomes and impacts are designed according to accepted empirical criteria and following standard methodologies, which are briefly described in Section III. B.
Exhibit 1. Framework for Institutionalizing Continuous Program Assessment and Evaluation into ORR Programs and the Role of the Federal Office
Program Dimension and Role of Evaluation
1 U.S. Department of State, U.S. Department of Homeland Security, and U.S. Department of Health and Human Services, “Proposed Refugee Admissions Report to Congress” for each fiscal year.
2 Mary Farrell, Bret Barden, and Mike Mueller, “The Evaluation of the Refugee Social Service (RSS) and Targeted Assistance Formula Grant (TAG) Programs: Synthesis of Findings from Three Sites,” forthcoming; Randy Capps, “Houston Case Study,” forthcoming; Nancy Pindus, “Miami Case Study,” forthcoming; Sam Elkin “Sacramento Case Study,” forthcoming.
3 “ORR Recommendations and Proposed Reporting Requirements and Guidelines for Economic Self-Sufficiency,” ORR State Letter 07-08.
4 Executive Office of the President, Office of Management and Budget, The President’s Management Agenda, Fiscal Year 2002 (no date). http://www.whitehouse.gov/omb/budintegration/pma_index.html.