Core Performance Indicators for Homeless-Serving Programs Administrated by DHHS

Review of Reporting and Performance Measurement Approaches Among Four Homeless-Specific Programs Administered by the U.S. Department of Health and Human Services

[ Main Page of Report | Contents of Report ]

Contents

1. Main Findings from Interviews with Agency Officials and Document Reviews
2. Implications and Conclusions for Development of Common Performance Measures

This chapter synthesizes the results of interviews with administrators and a review of relevant program documentation at the four DHHS homeless programs that are the focus of this study: (1) Programs for Runaway and Homeless Youth (RHY) Program, (2) the Health Care for the Homeless (HCH) Program, (3) Projects for Assistance in Transition from Homelessness (PATH), and (4) the Treatment for Homeless Persons Program. The initial research task was aimed at developing an understanding of the basic operations of these four DHHS homeless-serving programs, with a particular focus on each program’s performance measurement systems. Project staff conducted in-person discussions in December 2001 and January 2002 with programmatic, budget, and policy staff in the agencies that oversee these four DHHS programs. Appendix A provides a copy of the discussion guides used in conducting interviews with agency officials.

A. Main Findings from Interviews with Agency Officials and Document Reviews

Overall, our interviews with agency officials at the four homeless-serving programs provided: (1) background information about each program, including information about key program components and client flow; (2) principal performance measures; (3) methods used to collect performance data; and (4) program officials’ views on potential measures that might be incorporated to enhance performance monitoring. In synthesizing the results from our discussions and review of background documentation on each project, we have attempted to the extent possible to provide cross-program comparisons at a fairly detailed level of key program features and, particularly, with respect to performance measurement used by each of the programs. Exhibit 2-1 provides a comparative analysis of some of the key programmatic features of the four homeless-serving programs. Below, we highlight several key findings that emerge from this program comparison.

Program funding, allocation, role of the federal/state governments, and number and types of agencies providing services vary substantially across programs. While all four of the programs serve homeless individuals as their target population, there are substantial cross-program differences that complicate efforts to develop and implement common measures of performance and systems for collecting data across the four programs. Some underlying programmatic differences are highlighted in the exhibit:

• Authorizing Legislation: Authorizing legislation for two of the four programs comes from the McKinney Act (PATH and HCH). Authorization for RHY dates back nearly three decades to Juvenile Justice and Delinquency Act of 1974, while authorization came only about two years ago (in 2001 by Congressional directive) for the Treatment for Homeless Persons Program.
• Budget, Funds Allocation, and Matching Requirements: FY 2002 funding runs from $9 million (Treatment for Homeless Persons) to in excess of $100 million ($116 million for HCH). Three of the four programs allocate funds competitively; one of the programs allocates funds to states by formula (PATH). However, even within the three programs using a competitive process to select grantees, the methods for allocating funds and selecting grantees are quite different and quite complex. For example, under RHY’s Basic Center Program, 90 percent of funds are allocated to states based on the state population under age 18 in proportion to the national total. Regardless of the size of its youth population, the minimum allocation for a state is $100,000 ($45,000 for territories). Despite this initial allocation to states, the federal government runs a competitive grant process in which service providers (mostly local nonprofits and county agencies, though state agencies may also compete) submit grant applications. The applications are peer reviewed and awarded based on scores. If all the funding is not awarded within a state (i.e., all grant awards within a state do not add up to a state’s allocation), funds are re-allocated from that state to other states within the state’s region to fund other grant awards. By comparison, SAMHSA/CSAT issued Guidance for Applicants (GFA) for the Treatment for Homeless Persons Program for the first two rounds of grant awards. Nonprofit agencies submitted proposals and CSAT rated proposals and made awards to agencies based on how their proposals stacked up with others submitted. Finally, across the four programs, requirements for providing matching funds ranged from no match to providing $1 in non-federal funds for every $10 of federal funds to providing $1 in non-federal funds to every $3 of federal funds.
• Role of Federal versus State Governments: The federal government plays a significant role in all four of the programs — distributing funds to states (PATH) or competing grants and selecting grantees (in the case of the other three programs); providing oversight and collecting performance information; and providing technical assistance. In terms of state involvement, only under the PATH program among the four programs does the state play a significant role — distributing funds to local areas, providing technical assistance to local grantees, monitoring subgrantee performance, and submitting annual performance reports to the federal government.
• Number and Types of Grantees/Subgrantees: The number of grantees ranges from 50 grantees selected under the Treatment for Homeless Persons Program to about 640 under the three RHY programs. There is some overlap across programs in the types of local agencies receiving grant funds and providing direct services. All four of the programs rely to some extent upon nonprofit community-based organizations to recruit and deliver services (e.g., two-thirds of HCH grantees and about one-fourth of HCH grantees are CBOs). Community mental health centers account for nearly two-thirds of PATH grantees; CHCs represent about half of current HCH grantees; and RHY Basic Center Program grantees include a broad network of (about 400) youth shelters operated by public and nonprofit entities.

While there is a similar focus on homeless individuals across the four programs, there are differences in terms of the number and types of individuals served, definitions of enrollment, and duration of involvement in services. Not surprisingly, all four programs serve homeless individuals — though programs target different subpopulations of the homeless. The RHY programs target youth (both runaway and homeless), while the other three programs target services primarily on adult populations (though other family members are often also served). The HCH program funds initiatives that serve a broad range of homeless individuals (especially those unable to secure medical care by other means). The PATH and Treatment for Homeless Persons Programs serve a somewhat narrower subgroup of the homeless population than the other programs: the PATH program focuses on homeless individuals with serious mental illness; and the Treatment for Homeless Persons program targets homeless persons who have a substance abuse disorder, or both a diagnosable substance abuse disorder and co-occurring mental illness or emotional impairment.

The definition of “enrollment” and “termination” in the programs and duration of involvement in services all vary considerably by program. In a program such as PATH — which is considered to be a funding stream at the local (operational) level — it is often difficult to identify a point at which someone is enrolled or terminates from PATH. In a program such as HCH, a homeless individual becomes a “participant” when he/she receives clinical services at an HCH site. Length of participation in HCH is highly variable — it could range from a single visit to years of involvement. HCH program grantees would like to become the medical home for each individual until a point at which they are no longer homeless and can connect with another health care provider (to serve as the medical home). Much like any other private practice doctor, there is not generally a point in time in which an individual is terminated — rather, a case file is set up on the individual and at some point they simply do not show up (or may come in only sporadically for services).

Even within a program like RHY — which is composed of three program components — there is considerable variation in what constitutes enrollment and duration of involvement. For example, in RHY’s Street Outreach Program (SOP) — a program designed to get youth off the street and into a safe situation (and linked to needed services) — involvement is very brief (often a single contact) and presents little opportunity for collecting information about the individual. In contrast, RHY’s Transitional Living Program (TLP) provides residential care for up to 18 months under the program and a broad range of other services to move homeless youth toward self-sufficiency and independent living. RHY’s third program component — Basic Center Program (BCP) — offers up to 15 days of emergency residential care, help with family reunification, and other services. Hence, BCP’s involvement with homeless youth is longer and more intensive than SOP, but much shorter and less intensive than TLP.

Finally, of the four programs, enrollment in the Treatment for Homeless Persons Program appears to be most clearly defined. Homeless individuals are considered participants when the intake form (part of the Core Client Outcomes form) is completed on the individual (though there is no standardized time or point at which this form is to be completed by program sites). Involvement in the program is extended over a year or longer — with follow-up surveys being conducted with participants at six and 12 months after intake into the program.

While actual numbers of individuals “served” or “participating” are difficult (if not impossible) to compare because of varying definitions across programs, the sizes of programs appear quite different. For example, HCH (with 142 grantees nationwide) reports that “about 500,000 persons were seen in CY 2000.” Under its BCP program component (with a network of about 400 youth shelters nationwide providing services), the RHY program estimates that it “helps” 80,000 runaway and homeless youth each year. According to figures reported annually by states, the number of homeless individuals with serious mental illness who were PATH clients in FY 2000 was about 64,000 (though as noted earlier, because PATH is regarded as a funding stream rather than a distinct program, it is often difficult to isolate an individual as a “participant” or being “served” by PATH). Finally, through the first two rounds of funding, the 36 grantees funded under the Treatment for Homeless Persons Program anticipate serving about 7,700 individuals (over the three-year grant period).

Wide range of program services offered through the four programs. As shown earlier in Exhibit 2-1, despite their many differences, there is a fair degree of convergence in the goals of the four DHHS homeless-serving programs. All four of the programs are aimed at improving prospects for long-term self-sufficiency, promoting housing stability, and reducing the chances that participants will become chronically homeless. Each program has more specific goals that relate to the populations served and the original program intent – for example, RHY’s BCP component has as one of its goals family reunification (when appropriate); HCH aims to improve health care status of homeless individuals; PATH aims to engage participants in mental health care services and improve mental health status; and the Treatment for Homeless Persons Program aims at engaging participants in substance abuse treatment and reducing/eliminating substance abuse dependency.

Exhibit 2-1 (shown earlier) provides an overview of services delivered through the four programs. Common themes cutting across the programs include emphases on flexibility, providing community-based services, creating linkages across various types of homeless-serving agencies, tailoring services to each individual’s needs (through assessment and case management), and providing a continuum of care to help break the cycle of homelessness. For example, the Treatment for Homeless Persons Program emphasizes linkages between substance abuse treatment, mental health, primary health, and housing assistance; HCH emphasizes a multidisciplinary approach to delivering care to homeless persons, combining aggressive street outreach, with integrated systems of primary care, mental health and substance abuse services, case management, and client advocacy. Of the four programs, the RHY program (in part, because it targets youth) provides perhaps the most unique mix of program services — and even within RHY, each program component provides a very distinctive blend of services (e.g., street outreach [the Street Outreach Program] versus emergency residential care [Basic Center Program] versus up to 18 months of residential living [Transitional Living Program]).

The four homeless programs feature substantially different approaches to performance measurement, collection of data, and evaluation. Given the variation in the structure of these four programs, it perhaps comes as no surprise that their approaches to information collection, performance measurement, and evaluation are quite different (see Exhibit 2-2). With respect to GPRA measures, three of the four programs have explicit measures; there are no GPRA measures specific to HCH. GPRA measures apply to the BPHC’s Health Centers Cluster of programs as a whole, of which HCH program is part.⁽¹⁰⁾ The measures used for the three other programs range from process to outcome measures. The Treatment for Homeless Persons Program has outcome-oriented GPRA measures, as well as a data collection methodology (featuring intake and follow-up client surveys) designed to provide participant-level data necessary to produce the outcome data needed to meet reporting requirements. For example, the GPRA measures for adults⁽¹¹⁾ served by the Treatment for Homeless Persons Programs are the percent of service recipients who — (1) have no past month substance abuse; (2) have no or reduced alcohol or illegal drug consequences; (3) are permanently housed in the community; (4) are employed; (5) have no or reduced involvement with the criminal justice system; and (6) have good or improved health and mental health status. In contrast, the measures employed by PATH are process measures (rather than outcome-oriented): (1) percentage of agencies funded providing outreach services; (2) number of persons contacted, (3) of those contacted, percent “enrolled” in PATH. Of the three main GPRA measures used in the RHY program, just the first one is outcome-oriented: (1) maintain the proportion of youth living in safe and appropriate settings after exiting ACF-funded services; (2) increase the proportion of BCP and TLP youth receiving peer counseling through program services; and (3) increase the proportion of ACF-supported youth programs that are using community networking and outreach activities to strengthen services.

As displayed in Exhibit 2-2, how performance data are collected and the quality of the data collected also varies across the four programs. Three of the four programs have states (PATH) or grantees (HCH and RHY) submit aggregate data tables either annually or semi-annually. For example, under the PATH program, states submit 16 tables annually, including — (1) federal PATH funds allocated to the state, (2) total FTEs providing PATH supported services, (3) PATH providers by type of organization, (4) state/local matching funds, (5) PATH portion of local provider budgets, (6) PATH clients as a percentage of homeless clients in all services, (7) number of organizations providing PATH services by type of service and funding, (8) number and percent of PATH outreach contacts that eventually become enrolled in services, and (9) number and percent of PATH clients by: age, gender, race, principal diagnosis, dual diagnosis, veteran status, client’s housing status, and length of time homeless. States send these tables via the Internet to the federal program office (HPB/CMHS), which abstracts data from each state to generate figures needed on each of the three GPRA measures. Data provided is aggregate (rather than at the participant-level) for PATH (as is the case for HCH and RHY).

The Treatment for Homeless Program (the newest of the four programs) takes an entirely different approach to collection of data than the other three programs. Each of the grantees collects participant-level data at three points of the client’s involvement in the program (using standardized data collection form across all sites, referred to as the Core Client Outcomes form) — at intake, 6 months after intake, and 12 months after intake. This generates the data needed by the program to address the outcome-oriented GPRA measurement (e.g., percent of participants who have no past month substance abuse). In addition, it is possible on an individual participant basis to make comparisons on the various outcome measures between the time of intake and follow-up to determine pre/post change for each participant.

All four of the programs use (or are in the process of developing and implementing) some type of automated database for transmission of performance data to their federal administering agencies. In the case of PATH, HCH, and RHY standardized data tables are produced by each state (PATH) or grantee (HCH and RHY) and submitted via the Internet. CSAT has recently implemented a web-based application, which enables Treatment for Homeless Persons Program grantees to submit participant-level records via the Internet.

There are several other issues with regard to the collection of performance data that affect their appropriateness for GPRA reporting and evaluating program performance:

• The reliability and quality of the data collected and submitted to federal offices varies by program. For example, RHY is substantially revising and streamlining its data system (RHYMIS) in response to past problems with data completeness and quality. A recent report noted RHYMIS data are unreliable because of chronic low levels of grantee reporting (less than 50% of grantees submitted reports for all four quarters in FY 1999); also, many youth served by RHY centers are not counted as “admitted to services” in RHYMIS because services are funded by non-federal sources.
• The HCH program is embedded in the Health Cluster of programs at BPHC (and is a relatively small program despite its in excess of $100 million funding, when compared to the Community Health Centers funding). Annual reports submitted by grantees receiving more than one Health Cluster source of funds combine HCH participants with participants of other Health Cluster programs – and, hence, it is not possible (except on several of the tables provided by grantees) to produce disaggregated counts for HCH participants.
• Sophistication with data collection and reporting varies considerably across and within programs. For example, RHY’s BCP program funds about 400 youth shelters. A recent RHY report indicated that funded agencies range from large, multi-service agencies with fairly sophisticated data collection capabilities to small single-service agencies just beginning to use MIS technology to track service delivery.
• Intensity and duration of participant involvement in programs ranges considerable across and within programs. As noted earlier, short or episodic involvement of participants in programs (such as that of some participants of the RHY and HCH programs) limit opportunities for in-depth collection of data from participants.

Program Use Data for a Variety of Purposes. Data collected by the four programs are used for a broad set of purposes, particularly reporting to Congress and others about the program, budgeting purposes, deciding on how to allocate funds to grantees, and to support evaluations of the programs and technical assistance efforts. Programs do not use the data at this time for performance rewards — though in some cases, the data has an effect on which grantees are funded in future rounds. As described by agency officials, the following are the main ways in which data currently collects are being used:

• HCH: Data are used mainly to report to the Department and Congress on whom the program serves and the types of services provided. No rewards or sanctions result directly from data collected — though technical assistance may be provided for those with poor performance. Some funding decisions are based, in part, on trends in users over several years time.
• PATH: The data collected during the grant application process and through the annual reporting system are used to generate cross-state data tables and U.S. totals to analyze program and participant characteristics. These data are used to report to Congress on the program and to generate data on the three GRPA measures. The federal office carefully reviews annual reports and grant applications — providing comments where appropriate to states.
• Treatment for Homeless Persons Program: Though the program just recently started, data collected will ultimately be used to report to Congress on the performance of the program. No performance bonuses are planned.
• RHY: Data collected from grantees is used by the Family and Youth Services Bureau (FYSB) to report annually on the program. Data are not used to distribute performance awards; data collected may be used to guide decisions in issuing future grants to existing or past grantees.

B. Implications and Conclusions for Development of Common Performance Measures

Initial discussions with agency officials at the four homeless-serving programs and review of readily available program documentation suggested that despite having a common focus on serving homeless individuals, the four programs that are the focus of this study have many differences. Upon closer examination of the programs, the differences appear to be greater than the similarities – for example, the four programs serve different subpopulations of the homeless, providing a different range of services over varying lengths of participant involvement, to achieve often different results. As might be expected given these programmatic differences, approaches to measuring and reporting on program performance and the problems associated with collecting high-quality data are also quite different. In particular, sites vary substantially across the following dimensions:

• Actual GPRA measures used — ranging from process to outcome-oriented;
• The specific data items collected and the extent to which pre/post outcome data are collected;
• Whether data are maintained and submitted to the federal office in aggregate or at the participant-level;
• Reliability and completeness of data provided; and
• Whether new performance measures and data systems are currently being designed and/or implemented.

The implication of these differences is that it will be a difficult and delicate task to come up with a common set of performance measures across the four programs, which are also applicable to other DHHS programs serving homeless individuals. In addition, while federal agency officials are very willing to discuss their programs and share their knowledge of how they approach data collection and reporting, their willingness and ability to undertake change (e.g., potentially incorporating new, more outcome-oriented GPRA measures) is uncertain. From our discussions, it appears that changes in how programs collect data and report on performance will require substantial efforts on the part of agency officials and programs. For example, with regard to RHY — which is currently involved in an effort to implement a streamlined data system — it would not only require change at the federal administering agency, but how over 600 grantee organizations collect and manage data.

In the next chapter of this report, we examine the potential relevance of homeless administrative data systems (HADS) for enhancing data collection and performance measurement in DHHS homeless-serving programs. Chapter 4 then returns to the main focus of this study — examining the potential for implementing a set of common performance measures across these four homeless-serving DHHS programs.

Endnotes

(10) HCH is clustered with several other programs, including Community Health Centers [CHCs], Migrant Health Centers, Health Services for Residents of Public Housing, and other community-based health programs.

(11) As shown in Exhibit 2-2, GPRA measures are slightly different for youth.

Where to?

Top of Page | Contents

Main Page of Report | Contents of Report

Home Pages:
Human Services Policy (HSP)
Assistant Secretary for Planning and Evaluation (ASPE)
U.S. Department of Health and Human Services (HHS)