By:
Greg A Greenberg, PhD
Northeast Program Evaluation Center, VAMC West Haven, CT,
Yale University Department of Psychiatry, New Haven, CT
and
Robert A Rosenheck, MD
Northeast Program Evaluation Center, VAMC West Haven, CT,
VA New England Mental Illness Research, Education, and Clinical Center;
Yale University Department of Psychiatry and School of Epidemiology and Public
Health, New Haven, CT
February 13, 2007
This report is available on the Internet at:
http://aspe.hhs.gov/hsp/homeless/CICH/networks/
Printer Friendly Version in PDF format (53 pages)
Abstract (in PDF format - 3 pages)
Sections
References
Appendix A: Network Participation Survey (in
PDF format - 10 pages)
Footnotes
Peggy Halpern, Paul Dornan, Pete Dougherty, Anne Fletcher, Cynthia High, Mark Johnston, Robyn Raysor, and Gay Koerber participated in the Federal Funders CICH Evaluation Group representing HUD, DHHS, and VA provided essential oversight and review on earlier drafts of this paper. [Correspondence]
We wish to acknowledge specifically the CICH evaluation site coordinators who coordinated data collection at their sites: Joyce Jones and Daniel White (Chattanooga), Eugene Herskovic (Chicago), JuanitaWilson (Columbus), Richard DiBlasio (Denver), Daniel Robbin and Elaine Stein (Ft. Lauderdale), John Nakashima (Los Angeles), Phyllis Larimore (Martinez), Julie Irwin (New York), Vincent Kane and Kimberly Lewis (Philadelphia), Lawrence Brennan (Portland), and Charlene Nason (San Francisco).
Brandi Williams coordinated data management at the VA Northeast Program
Evaluation Center (NEPEC) and Dennis Thompson provided computer programming support.
Enhancing service system integration has long been an objective of policymakers, payers, and human service providers. As far back as the Model Cities Program of the 1960s processes that facilitate system integration have been thought to improve the capacity of systems to address the needs of individuals with multiple problems and have been widely believed to increase the accessibility, continuity and coordination of care (Morrissey et al. 1994; Provan & Milward 1995; Randolph et al 2002; Cocozza et al. 2000; Foster-Fishman et al. 2001; Provan & Milward 1995). Emphasis on implementation of evidence-based practices has been a growing and parallel trend in health care (Guyatt et al. 2004) and has been rigorously promoted in several recent policy summaries of literature on the treatment of mental illness (Drake et al. 2001; Torrey et al. 2001; DHHS 1999; New Freedom Commission on Mental Health 2003; SAMHSA 2003). It has been suggested that more integrated systems may also allow for more rapid dissemination of evidence-based practices, with greater model fidelity(2) (McKinney et al 1993; Goldman et al. 2001).
There is broad consensus that increasing the level of service system integration should lead to improved client outcomes. System integration can be defined broadly as the provision of services with high levels of coordination, communication, trust, and respect among service agencies so that they are better able to work together to achieve common objectives. While there is some supporting evidence from cross sectional data (Provan & Milward 1995; Rosenheck et al. 1998; Rosenheck, Morrisey, Lam, et al. 2001), two large prospective evaluations of system integration initiatives found that even when the levels of integration were increased there was no substantial improvement in treatment outcomes at the individual client level. One explanation for this finding may be that improved system integration may be necessary but not sufficient to improve client outcomes and that concomitant implementation of evidence-based clinical practices may also be necessary (Isett and Morrissey 2006; Goldman et al. 2001; Lehman et al. 1994).
In 1986, the Program on Chronic Mental Illness launched by the Robert Wood Johnson Foundation and the U.S. Department of Housing and Urban Development sought to evaluate whether more highly integrated systems of care were more effective in addressing needs of persons with severe mental illness (SMI). Results of this nine-site evaluation found that integration efforts were associated with measurable improvements in inter-agency collaboration and increased continuity of care (in part as a result of greater availability of case management services) but were not associated with improved client outcomes, such as symptoms, social relationships, and quality of life (Morrissey et al 1994; Lehman et al. 1994).
Similar results were reported from the 18-site evaluation of the Center for Mental Health Service's Access to Community Care and Effective Services and Supports (ACCESS) program. This five-year demonstration program, implemented in 1993, evaluated the impact of efforts to enhance system integration on outcomes of homeless persons with serious mental illness (Cocozza et al. 2000; Rosenheck et al. 2001) and represented a major improvement over the evaluation design of the Program on Chronic Mental Illness by including a matched sample of comparison sites that did not implement integration strategies. While intervention sites in ACCESS showed greater increases in measures of system integration than comparison sites (Morrissey et al. 2002), clients at these sites did not show greater improvement in housing or symptom outcomes (Rosenheck et al. 2002). However, correlational findings from the ACCESS evaluation showed that sites that had more integrated service systems, regardless of whether they were intervention or comparison sites, had superior 12 month housing outcomes at (Rosenheck et al. 1998) but that this relationship may have been mediated by the level of community social capital, an indicator of overall civic culture (Putnam 1994; 2000; Rosenheck et al. 2001) rather than by specific integration interventions (Rosenheck et al 2002).
In 2002 the federal government made a major commitment to address the problem of chronic homelessness (Sullivan 2002), which was defined in the CICH Notice of Funding Availability (NOFA) as, "an unaccompanied homeless individual with a disabling condition who has either been continuously homeless for a year or more OR has had at least four (4) episodes of homelessness in the past three (3) years." It was expected that by focusing on housing and service needs of this target population it would be possible to disrupt repetitive cycles of recurrent homelessness and free resources to meet the needs of the vast majority of people experiencing homelessness who, with limited assistance, can often exit homelessness relatively quickly (Interagency Council on Homelessness 2003).
A major service demonstration project that emerged from this effort is the Collaborative Initiative to Help End Chronic Homelessness (CICH), a $55 million in federal dollars effort ($35 million funding in 2003, with $20 million added in subsequent years) jointly funded by the Department of Housing and Urban Development, the Department of Veterans Affairs, and the Department of Health and Human Services. After a competitive Request for Applications, CICH awards were made to 11 jurisdictions to provide comprehensive assistance to chronically homeless persons, and to help them move from the streets and emergency shelters into stable housing.
CICH is the first national evaluation of client outcomes among chronically homeless individuals targeted to receive comprehensive housing and support services through a collaboration of HUD, HHS, & VA. Neither the Robert Wood Johnson Foundation Program on Chronic Mental Illness nor the ACCESS program targeted this particularly vulnerable and costly subgroup. Furthermore, no previous initiative focused on collaboration among the three federal agencies primarily responsible for assisting and providing care for homeless persons or emphasized both system integration and implementation of evidence-based practices.
In this report we present system-level observational evaluation data on the CICH program. We describe changes in performance measures over time, by agency type (i.e., lead, housing service provider, veteran service provider, etc) and across sites, as well as examine interrelationships among salient measures. CICH focused on improving outcomes for chronically homeless people by making funding available to provide five core services at each site: (1) permanent supportive housing, (2) mental health treatment, (3) substance abuse treatment, (4) primary health care, and (5) veteran health services. Services were to be provided through a local network of agencies, coordinated by a local "lead" agency that would oversee distribution of funds; facilitate joint planning and coordination across agencies; promote the use of evidence-based practices; and foster development of homeless management information systems. This evaluation was not designed to evaluate the causal impact of specific interventions but to present descriptive data on service system activities and characteristics, clinical service delivery, and client outcomes.
The CICH program represents an extension of efforts to integrate services for people experiencing homelessness fostered for many years by the Department of Housing and Urban Development's (HUD) Continuum of Care (COC) initiative. Since 1996, through its competitive application process for supportive housing programs, HUD has promoted the development of networks of agencies that together constitute a Continuum of Care. These networks are organized around the centralized distribution and coordination of federal funding to provide a comprehensive array of services for homeless Americans (Burt et al 2002). In addition to encouraging interagency collaboration, CICH took note of but did not mandate the development of "Housing First" service models, in which people experiencing homelessness are moved into permanent housing as quickly as possible, and once housed, are provided long-term support to facilitate access to services thereby preventing a return to homelessness (Tsemberis et al. 2000, 2004; Kowal 2006).
This report examines four questions reflecting central objectives of the CICH initiative at the service system level. First, to what degree is CICH associated with implementation of practices that encourage system integration; with improvements in coordination of service delivery and planning among participating agencies over time; and with increased trust and respect between providers? Second, was the initiative associated with changes in the type of housing provided at CICH sites, with the implementation of homeless information management systems, or with the availability of evidence-based mental health practices? Third, did some sites and some types of agencies show greater change in measures of system-wide performance than others? Lastly, to complement the focus on organizational integration we examine whether relationships specifically characterized by exchanges of funds are associated with greater levels of inter-agency integration, collaboration and trust, both cross-sectionally and over time.
While this report thus focuses on an examination of system-level activities in CICH, another report, "HUD/HHS/VA Collaborative Initiative to Help End Chronic Homelessness National Performance Outcomes Assessment Client Outcomes Report," will present data on service delivery and client outcomes in the CICH program and a third report will address the relationship of client outcomes and system integration.
[ Go to Contents ]
The CICH program began between March and August 2004 at 11 sites: Los Angeles, Martinez (Contra Costa), and San Francisco, California; Philadelphia, Pennsylvania; New York, New York; Denver, Colorado; Chicago, Illinois; Columbus, Ohio; Chattanooga, Tennessee; Fort Lauderdale, Florida; and Portland, Oregon (See Table 1 for a brief description of each site) and are scheduled to run for a three year period after start up. Sites varied in the number of chronically homeless that they would serve, from 50 to 100 people.
Each site identified core agencies to participate in an annual key informant survey involving representatives of the lead agency and partnering agencies that provided housing assistance, mental health care, substance abuse services, primary care, and the local Veterans Health Administration facility. A "network definition survey" was administered at the start of the initiative (in November/December 2003) to identify participating agencies and key informants at each site. Key informants were those identified by program leaders as the most knowledgeable about the activities of each agency at each site.
A second more extensive "network participation survey" was then administered in three waves, between November 2003 to March 2006, to the key informants identified at each of the participating agencies. The first survey wave occurred before CICH began (from November 2003 to March 2004). The second and third survey waves occurred at the end of the first and second years of operation (from November 2004 to February 2005 and from January to March 2006, respectively). Surveys were sent to all key informants prior to the interviews, which were conducted over the telephone by national evaluation staff at the Northeast Program Evaluation Center. In some sites one agency provided more than one key service (e.g., the mental health agency also provided substance abuse service) and only one survey was conducted for that agency. At other sites, in contrast, more than one agency provided a particular service and key informants at both agencies were interviewed. Thus, the number of agencies at each site ranged from five to nine. Furthermore, at some agencies more than one key informant was identified and interviews were jointly held.
Five dimensions of CICH service systems were assessed: 1) system connectedness and integration (Coccozza et al., 2000; Interagency Council on Homelessness 2003; Morrissey et al., 2002); 2) emphasis on providing permanent supported housing services (Interagency Council on Homelessness 2003); 3) development of homeless services management information systems (Magnobosco-Bower JL 2001); 4) use of evidence-based mental health practices (DHHS 1999; New Freedom Commission on Mental Health 2003; SAMHSA, 2003); and 5) the existence of interagency fiscal relationships. Because each measurement domain could be addressed in several ways, multiple measures were used in each domain.
Most of the measures were based on items in which each key informant characterized the overall service network at their site from the perspective of their agency. However, elaborating on methods developed by Morrissey et al. (2002), 16 questions asked key informants about the relationship of their agency to each of the other participating agencies in their local network (see items 32-45 in Appendix A). While respondents answered questions about the performance of their network as a whole only once, they gave four to nine responses (depending on the number of agencies in the core CICH network) to questions concerning their dyadic relationship with each other agency. Creation of scales using these 16 items involved multiple steps. First, we conducted a separate exploratory factor analysis of the 16 items. Factor analysis indicated that data could be summarized in four factors that explained 74% of the variation in the 16 measures (see Table 2 for factor loadings for individual items). One of the four factors represented joint planning and cooperation with other agencies, and a second addressed trust and respect between agencies. For these two factors the relevant items were averaged to create scales characterizing each dyadic relationship. The other two factors addressed interagency funding flows: and were used to create a measure indicating whether a fiscal relationship existed between each pair of agencies.
In the sections below we describe specific measures in each of the five domains.
1) System connectedness and integration: Six measures were used to assess connectedness and integration, four of which were based on key informant ratings of the overall CICH network and two that were based on dyadic (i.e. interagency) relationship scales, as described above.
The first system-level measure assessed the extent to which each network implemented specific interventions that were designed to result in greater system integration (Morrissey 2002). This measure is the average implementation rating given to 20 potential interventions, 12 of which were developed to evaluate the implementation of integration strategies in the ACCESS program (Cocozza et al. 2000; Morrissey et al. 2002) and the remainder of which were developed specifically for the CICH initiative. Each item represented the degree to which each of the 20 strategies was implemented, on a score from zero to three with zero representing "none" or no effort and three representing "a lot". Examples of these strategies are presence of: a) a system integration coordinator position, b) an interagency coordinator body, c) a formal strategic plan, d) co-location of services, e) cross-training, or f) client tracking systems (see Appendix A for a full list of these strategies — items 46-65) (Cronbach's coefficient alpha=.88; Range .2 - 3).
The second system-level integration measure assessed the extent to which the local homelessness coalition provided material resources, or political or institutional support for the implementation of the CICH initiative. This "coalition participation" measure is the average of five items which addressed how important the coalition had been to the CICH network in providing: 1) support for the development of the initial application; 2) material resources, 3) political and institutional support; 4) help in implementing the initiative; and 5) guidance in the shaping of the goals of the initiative (Cronbach's coefficient alpha=.84 for all five items). Items were scored from zero to three with zero representing "not at all" and three representing "very" or "a lot" (see appendix A for details on the specific items in this scale — 86-90).
The third measure of system connectedness and integration was a global assessment of how well the agencies at a site worked together to solve problems. This scale is the average of four items that focus on how well agencies jointly identified, understood and fixed service delivery problems, as well as how effectively they worked together to address client service needs (Cronbach's coefficient alpha=.83). Here too, each item was scored on a zero to three scale where zero represented "not at all" and three represented "very well" (see Appendix A items 18a-18d).
The fourth system integration scale assessed the total number of different types of agencies that were involved with the CICH network. Key informants at each surveyed agency were asked about agencies that were on the initial grant application for the CICH network and agencies as well as types of local organizations that were not included among the core agencies (e.g., soup kitchens, law enforcement agencies, private businesses, etc.) (Cronbach's coefficient alpha=.75; Range 3 - 12). Items in this scale were dichotomized so that answers of two or three ("somewhat involved" or "very involved") were scored as a one, and other responses were scored as zero (see Appendix A items 11a-11l).
The last two scales were based on items derived from the dyadic measures of the relationships between pairs of agencies and addressed a) joint planning and cooperation and b) trust and respect within each dyadic relationship. First, as suggested by the factor analysis described above, nine items were averaged to constitute a scale representing inter-agency joint planning and coordination. Specific items in this scale addressed: cooperation in serving clients, goal congruence, client referral, cooperative planning, co-location of staff and services, information sharing and communication (see items 32-36 and 40-43 in appendix A for details).
The second dyadic interagency scale, a measure of interagency trust and respect, was constructed as the average of two items addressing the extent to which this relationship was characterized by: 1) trust and 2) respect.
Items in all dyadic scales were scored from zero to three with zero representing "none" and three representing "a lot".
2) Emphasis on providing diverse housing services. Two measures were derived from an exploratory factor analysis of eight items and addressed: a) degree of emphasis in the network, as a whole, on providing various types of housing; and b) changes in the types of housing emphasized by each CICH network (for further details see items 13a-13d and 15a-15d in Appendix A). Factor analysis indicated that these items could be summarized in two scales, one reflecting an emphasis on emergency shelter as well as transitional and unsupported housing and the other reflecting a change in these emphases (see Table 3 for factor loadings for individual items).
3) Development of homeless services management information systems (HMIS). Two measures were created to evaluate the degree to which systems existed at the network level for management of client, service delivery and outcome information. The first measure was the average of three items that documented whether a system was available to generate data on: 1) client characteristics, 2) housing and service delivery, or 3) client outcomes. These three items were scored from zero or one, with zero representing "no" and one representing "yes".
The second measure was a single item which asked if an HMIS specific to the CICH initiative had been implemented. There were five possible answers to this question: 1) neither planned for, nor currently available for use, 2) being planned or developed, 3) in use by some network members, 4) in use by most network members, or 5) in use by all network members. The first answer was scored as zero while the last answer was scored as four (see items 19, 21, 23, and 25 in Appendix A for more details).
4) Evidence-based practices. A single scale was used to measure the degree to which each of the 11 CICH networks was viewed by its core agencies as having implemented evidence-based practices. This scale was based on the 18 practices identified in 2003 by the Substance Abuse and Mental Health Services Administration (SAMHSA) to be solidly evidence-based in a report entitled "An Overview of Mental Health and Substance Abuse Services and Systems Coordination Strategies" (SAMHSA 2003). Examples of these practices are: a) multidisciplinary treatment teams or intensive case management (e.g. Assertive Community Treatment [ACT]); b) self-help programs; c) the housing-first model of supportive housing; d) opiate substitution; and e) family psycho-education (for further details see items 66 to 82 in Appendix A). Items for each of the 18 evidence-based practices measured the extent to which these practices were implemented at each site and were scored from zero to three, with zero representing "none" or no effort and three representing "a lot". The values for the 18 items, one for each evidence-based practice, were averaged for each survey respondent to create this scale (Cronbach's coefficient alpha= .86; Range .11 - 3).
5) Resource flows and influence over resource flows. Finally, a scale was constructed to measure whether an active fiscal relationship existed between each pair of agencies. This measure was based on four items based on the factor analysis described above. First, two intermediate scales were created that characterized the transfer of funds or influence over the transfer of funds between agencies. The first of these dyadic scales (sending or influencing funds sent), was based on responses to two questions concerning the extent to which each agency: a) directly sent funds to each other agency or b) influenced the flow of funds to other agencies from third parties. The second of these scales (receiving funds or influencing receipt of funds) was based on answers to a related pair of questions concerning the extent to which each agency: a) received funds directly from each other agency or b) influenced the receipt of funds from each other agency. The answers to these individual items were scored from zero to three with zero representing "no funds exchanged" and three representing "a lot of funds exchanged". These two intermediate scales were constructed by averaging each pair of items. In the second step, a final, dichotomous measure was created that had a value of one if either of the two intermediate scales had a value of one or greater (i.e., a rating of at least "a little" on either of the funds transfer measures). Otherwise the scale had a value of zero (no funds transfer). For every pair of agencies this scale indicates whether there existed any resource transfer, or influence over resource transfer from a third party between the two agencies in the dyad.
Separate analyses were conducted to address each of the central study questions.
System Integration: The first question concerned the degree to which the CICH project was associated with the implementation of interventions designed to increase system integration, and/or with resultant increases in system coordination, or in trust and respect among agencies at each site. To address this question. a series of analyses were conducted in which the dependent variables were the six system connectedness and integration measurements discussed previously, and the independent variable was a categorical variable representing the year of the project (time). This variable had a value of one to three, with one indicating measurement conducted before the project started and values of two or three indicating measurements that occurred in subsequent years.
In these analyses random effects were modeled using an unstructured covariance matrix, thereby adjusting standard errors for the correlated nature of the data in these models (i.e., for the potential correlation of observations from the same agency for different years). This technique is often referred to as hierarchical linear modeling (HLM)(3) (38).
The second part of the first evaluation question was whether the implementation of practices designed to facilitate system integration was associated with observed increases in system coordination as well as in trust and respect among providers at each site. To investigate this issue we examined the correlation of measures reflecting the implementation of integration strategies, goals and activities with the measures of system coordination and the measure of trust and respect. Correlations at the agency level were done using the PROC CORR procedure of the SAS ® software system.
Other System Changes: Analysis of data to answer the second study question also focused on system change, in this case, in the types of housing provided, in the degree to which information systems were implemented, and in the extent to which evidence-based mental health practices were deployed. The analyses of these data were similar to those described for measures of integration. Separate HLM models were examined in which the dependent variables were the measures of system performance and the independent variable represented the year of the study.
Site and Agency Change: The third system-level question concerned variation across sites and agency types in integration efforts and outcomes, and in other system-level performance measures.
In the analysis of the degree to which the 12 system performance measures changed to different degrees across sites, analysis of covariance was used to test the relationship between a categorical variable with 11 levels representing the site of the responding agency, a variable representing time, and the variable of primary interest, a term that represented the interaction of site and time. These analyses thus examined whether there were significantly different degrees of change over time between sites, on each dependent measure. For those dependent measures in which the interaction term was significantly related to the dependent measure (with an alpha level set to less than 0.15 because of the small number of observations), we further examined each measure over time to identify specific sites at which there was statistically significant change over time. The PROC GLM procedure of SAS ® was used to conduct these analyses.
The same analytic approach was used to examine variation across agency types over time on these measures. In these analyses a measure that represented the type of respondent agency was substituted for the measure of site. This measure is a categorical variable that represents each of the seven types of agencies, i.e., the lead agency; the local Veterans Health Administration facility; or an agency that provided housing assistance, mental health care, substance abuse services, primary care, or another service.
Fiscal Integration. The final system level question addressed the association between the presence of a funding relationship between each pair of agencies and the degree of joint planning and cooperation, as well as trust and respect. In this analysis, we first used HLM to examine two models in which the independent measure was the dyadic indicator of the existence of a fiscal relationship. Random effects were modeled for site, agency, and time. An additional set of models addressed the interaction of fiscal relationship and time. These models explored whether there were differences in the amount of change in the dependent measures between dyads with and without a fiscal relationship.
Analyses were based on two types of measures, as described above, one that used each participating agency at each of the 11 CICH sites as the unit of analysis, and the other that used dyadic relationships among the agencies as the unit of analysis. An average of 6.7 agencies (standard deviation=1.66) were surveyed at each site in each wave. Data were available for 80 agencies surveyed in wave one; 72 in wave two; and 70 in wave three.(4)
Data were available for a substantially greater number of dyadic relationships — 528 in wave one; 474 in wave 2; and 462 in wave 3. Data were thus available for an average of 44.4 dyadic relationships (standard deviation=19.8) per site in each survey year. Table 4 provides descriptive information on each system level measure and Table 5 presents a summary of bivariate correlations among the 12 measures.
Results for both the implementation of system integration practices, and the integration outcome measures showed positive trends (Table 6). Over the study period there was a significant increase of 15% in the measure of implementation of practices designed to encourage system integration. While there was no significant changes from wave one to wave three in the level of involvement of the local homeless coalition, or in how well the agencies at each site globally rated the way they worked together, this mostly likely reflects a ceiling effect since participating sites had achieved high baseline scores (2.4-2.6 out of a possible 3.0) prior to program implementation, i.e., during and even prior to the development of their CICH proposals.
There were significant increases in two other key measures of system integration (Table 6). Strong results were observed on the measure of joint planning and coordination, which increased by 25% over the study period and showed highly significant change (p<.0001) (see Figure 1). The dyadic measure of trust and respect also increased by a statistically significant 3.5%. The smaller magnitude of change on this measure similarly reflects a high baseline level (2.6 out of a possible 3.0) that left little room for improvement on the underlying metric.
Figure 1.
System Integration over Time
Analyses of correlations between the implementation measure and the level of system integration actually achieved showed that greater implementation of integration practices was highly and significantly associated with greater system integration and better system performance on virtually all measures (see first column of Table 5).
Table 7 presents data that address the second study question; i.e., was the implementation of the CICH initiative associated with changes in the type of housing provided at CICH sites, in the implementation of homeless management information systems (HMIS), or in the use of evidence-based mental health practices?
There was no significant change in the degree to which agencies at CICH sites emphasized the provision of emergency, transitional, and affordable housing without support, as contrasted with permanent supported housing, nor was there significant change in the assessment of change in types of housing emphasized at CICH sites. The lack of change in housing emphasis may reflect ceiling effects since many sites selected for CICH were already committed to developing permanent supported housing.
There was, however, a significant increase of 20% in the reported ability of CICH agencies to obtain information about clients served and services delivered to them by the CICH network. There was also a significant 54.5% increase in the measure of implementation of a homeless management information system.
The 13.7% increase in the measure of the use of evidence-based mental health practices progressed monotonically from wave one to wave three and was also highly statistically significant (p=.0002).
There were few differences among sites, or among types of agencies, in the magnitude of changes in system-wide performance measures.(5) Significant variation in change by site was observed on the dyadic measure of joint planning and coordination (see Table 8 columns 3-5).
Analysis of the site changes over time showed five of the eleven sites with significant increases in the level of joint planning and coordination, in contrast to the other six sites that did not show significant change (Table 8). Closer examination of the data indicated that although on average these six sites started at higher level of joint planning and coordination than the other five sites, they all scored less than two (with three being the highest possible score).
Examination of differences in change across types of agencies revealed significant differences in changes on both the measure of joint planning and coordination and on the measure of trust and respect. Five of the seven agency types showed significant change on the joint planning and coordination measure (Table 8 second panel). The lead agency alone showed a significant increase in trust and respect in comparison to the other agencies.
There was no significant increase or decrease in the prevalence of fiscal relationships(6) among agencies participating in CICH (see last row of Table 9) suggesting that participating in CICH was not associated with the formation of these types of relationships. Table 7 presents comparisons between pairs of agencies (i.e., dyads) with and without a fiscal relationship. Cross sectionally, agencies in dyads characterized by fiscal relationships had substantially higher ratings on the measure of joint planning and coordination (55.5% higher) and on the measure of trust and respect (8.6% higher) than those dyads without a fiscal relationship. However, there were no significant differences in the magnitude of change over the study period between pairs of agencies with and without fiscal relationships. Increases in system integration observed over time cannot therefore be attributed to the development of stronger fiscal relationships.
[ Go to Contents ]
This report highlights several positive trends in the characteristics and activities of CICH networks over the course of this initiative. The most notable trend was the significant increase over the study period in the implementation of practices that encourage system integration, as well as in levels of system integration themselves, particularly on the measure of joint service planning and coordination. We also found that the implementation of practices intended to encourage system integration was significantly and positively correlated with measured levels of integration. Future reports will evaluate the effects of system integration on service delivery and client outcomes.
It is further encouraging that there was a significant increase in the availability of information on client and service delivery and in the implementation of homeless management information systems as well as in the use of evidence-based mental health practices. Each of these findings reflects movement towards meeting CICH program goals.
There were no significant changes in ratings of the type of housing provided or in the prevalence of dyadic fiscal relationships. The CICH sites were selected as the best candidates out of a field of over 100 applicants and as a result of this selection process these sites appear to have been focused on the provision of permanent supported housing even before the initiative began.
Significant variation among sites or agency types in the amount of change they experienced over the study period was limited to two dyadic measurements of joint planning and coordination and trust and respect. These modest cross-site differences partly reflect lack of statistical power due to the small number of cases as well as to ceiling effects on these measures since these sites were selected for their demonstrated capacity to function as a coordinated network of agencies.
While a major emphasis in CICH was placed on encouraging system integration it was also of interest that agencies with ongoing fiscal relationships had significantly higher levels of joint planning and coordination as well as trust and respect, although these relationships do not explain the changes in integration measures observed over the course of the initiative.
As with previous evaluations of initiatives intended to increase system integration (i.e., the Program on Chronic Mental and the Access to Community Care and Effective Services and Supports program) we found that the CICH initiative was associated with increasing levels of coordination and communication as well as trust and respect among service agencies. Given appropriate motivation, technical support, and resources, participating agencies achieved increased levels of system integration and furthered the implementation of both homeless management information systems and evidence-based practices.
This preliminary evaluation report does not address the issue of whether the increases in system integration, or in the implementation of either information systems or evidence-based practices, were associated with improved client outcomes. Although substantial data will be available on client outcomes, the small number of sites and the high level of integration at the beginning of the project may limit our eventual ability to demonstrate such relationships.
Several limitations of this evaluation need mention. Most importantly, all data reported here were based on interviews with a small number of key informants. We did not have access to objective measures of site performance, a weakness that is broadly inherent in research on service systems due to their complexity, the large number of individuals and organizations that comprise them, and the many exogenous environmental factors that influence system operation (Rosenheck et al. 2001).
Secondly, this study did not use an experimental evaluation design through which sites would have been randomly assigned to a treatment or a control group exposed to different interventions, nor did we use a quasi-experimental study design, such as matched site comparisons. Although the pre- and post-implementation data for this evaluation are suggestive, other factors including reporting biases may have been responsible for the measured system changes.
While these limitations prevent us from concluding definitively that the CICH initiative caused the observed system changes at these 11 evaluation sites, the data presented are clearly consistent with such an inference, and suggest that site level initiatives in the CICH program successfully accomplished the program objectives.
[ Go to Contents ]
Burt, Martha, Dave Pollack, Abby Sosland, Kelly Mikelson, Elizabeth Drapa, Kristy Greenwalt, and Patrick Sharkey. (May 2002) Evaluations of Continuums of Care for Homeless People. Washington D.C.: U.S. Department of Housing and Urban Development Office of Policy Development and Research.
Cocozza, J.J; Steadman, H.J; Dennis, D.L; Blasinsky, M; Randolph, F.L; Johnsen, M; Goldman, H. (2000). Successful systems integration strategies: the ACCESS program for persons who are homeless and mentally ill. Administration and Policy in Mental Health 27(6): 395-407
Department of Health and Human Services (DHHS). Mental Health A Report of the Surgeon General. Rockville, MD: U.S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, Center for Mental Health Services, National Institute of Health, National Institute of Mental Health, 1999.
Drake RE, Goldman HH, Leff HS, et al. "Implementing Evidence-Based Practices in Routine Mental Health Service Settings" Psychiatric Services 52 (2001): 179-182.
Foster-Fishman, P.G., Salem, D.A., Allen, D.A., & Fahrbach, K. (2001). Facilitating interorganizational exchanges: The contributions of interorganizational alliances. American Journal of Community Psychology, 29(6), 875-905.
Goldman, Ganju, Drake, et al. "Policy implications for implementing evidence-based practices." Psychiatric Services 52(1) Dec 2001 52(12).
Guyatt, G., D Cook, B Haynes. Evidence based medicine has come a long way. BMJ. 2004; 329 (7473); 990-1 30 Ocbert.
McKinney, M M J P Morrissey, and A D Kaluzny. Interorganizational exchanges as performance markers in a community cancer network. Health Serv Res. 1993 October; 28(4): 459-478.
Interagency Council on Homelessness. January 27 2003 Federal Register http://www.hud.gov/offices/cpd/homeless/apply/2002nofa/nofa.doc [Docket No. FR-4805-N-01] Accessed June 1, 2006
Isett, Kimberley; Morrissey, Joseph. Assessing Delayed Effects of a Multi-Site System Intervention for Homeless Persons with Serious Mental Illness. Administration and Policy in Mental Health, Volume 33, Number 1, January 2006, pp. 115-121(7)
Kowal, Jessica. (June 30, 2006) "Homeless alcoholics receive a permanent place to live, and drink." New York Times. Accessed on Internet.
Lehman AF, Postrado LT, Roth D, McNary SW, Goldman HH. "Continuity of Care and Client Outcomes in the Robert Wood Johnson Foundation Program on Chronic Mental Illness." Milbank Quarterly, 1994;72(1):105-22.
Magnobosco-Bower JL. An evaluation of state public mental health system performance for adult persons with serious mental illness: Effects of state political culture and state mental health planning & implementation characteristics on state public mental health system comprehensiveness. Doctoral thesis. Columbia University, 2001
Morrissey, Joseph P., Michael Calloway, et al. Local Mental Health Authorities and Service System Change: Evidence from the Robert Wood Johnson Foundation Program on Chronic Mental Illness. Milbank Quarterly Volume 72 Number 1, 1994
Morrissey JP, Calloway MO, Thakur N, Cocozza J, Steadman HJ, Dennis D; ACCESS National Evaluation Team. Integration of service systems for homeless persons with serious mental illness through the ACCESS program. Access to Community Care and Effective Services and Supports. Psychiatr Serv. 2002 Aug; 53(8):949-57.
New Freedom Commission on Mental Health, Achieving the Promise: Transforming Mental Health Care in America. Final Report. DHHS Pub. No. SMA-03-3832. Rockville, MD 2003.
Provan Keith G. Brinton Milward. A Preliminary Theory of Interorganizational Network Effectiveness: A Comparative Study of Four Community Mental Health Systems. Administrative Science Quarterly, Vol. 40, 1995. 1-33.
Putnam RD, Leonardi R, Nanetti RY: Making Democracy Work: Civic Traditions in Modern Italy. Princeton University Press, New Jersey, 1994
Putnam RD: Bowling Alone: The Collapse and Revival of American Community. New York, Simon and Schuster, 2000.
Randolph, Frances, Margaret Blasinsky, M.A., Joseph P. Morrissey, Ph.D., Robert A. Rosenheck, M.D., Joseph Cocozza, Ph.D., Howard H. Goldman, M.D., Ph.D. and the ACCESS National Evaluation Team Overview of the ACCESS Program. Psychiatr Serv 53:945-948, August 2002
Rosenheck RA, Morrissey J, Lam J, Calloway M, Johnsen M, Goldman HH, Calsyn R, Teague G, Randolph F, Blasinsky M and Fontana A (1998) Service System Integration, Access to Services and Housing Outcomes in a Program for Homeless Persons with Severe Mental Illness. American Journal of Public Health 1998; 88(11):1610-1615.
Rosenheck RA, Morrissey J, Lam J, Calloway M, Stolar M, Johnsen M, Randolph F, Blasinsky M and Goldman HH. Service delivery and community: Social capital, service systems integration, and outcomes among homeless persons with severe mental illness. Health Services Research 2001;36(4):691-710.
Rosenheck RA, Lam J, Morrissey JP, Calloway M, Marilyn Stolar, Randolph F, Blasinsky M, Johnsen, M, Steadman H, Cocozza J, Dennis DD, Goldman HH. Service systems integration and outcomes for mentally ill homeless persons in the ACCESS program. Access to Community Care and Effective Services and Supports. Evidence from the ACCESS Program. Psychiatric Services. 2002:53(8):958-966
SAMHSA. Best practices: An overview of mental health and substance abuse services and system coordination strategies. URL: http://alt.samhsa.gov/grants/content/2003/Resource%20material.doc (2003). Accessed June 1, 2006
Sullivan, Brian. (July 18, 2002). "News Release: White House Interagency Council announces new strategy to combat chronic homelessness: New plan will better coordinate federal response to homelessness." Interagency Council on Homelessness. Accessed at http://www.hud.gov/news/release.cfm?content=pr02-078.cfm on July 13, 2006.
Torrey, William C M.D., Robert E. Drake, M.D., Ph.D., Lisa Dixon, M.D., M.P.H., Barbara J. Burns, Ph.D., Laurie Flynn, A. John Rush, M.D., Robin E. Clark, Ph.D. and Dale Klatzker, Ph.D. Implementing Evidence-Based Practices for Persons With Severe Mental Illnesses. Psychiatric Services 52:45-50, January 2001
Tsemberis, Sam. and Ronda F. Eisenberg, M.A. Pathways to Housing: Supported Housing for Street-Dwelling Homeless Individuals With Psychiatric Disabilities. Psychiatric Services 51:487-493, April 2000
Tsemberis S, Gulcur L, Nakae M. Housing First, consumer choice, and harm reduction for homeless individuals with a dual diagnosis. Am J Public Health. 2004 Apr; 94(4):651-6.
[ Go to Contents ]
2. Model fidelity refers to the degree to which a program is implemented in conformance with an evidence-based treatment model (such as, Assertive Community Treatment), i.e. has elements and levels of intensity that experts have found to be part of effective models and does not have elements that are not thought to be part of effective models.
3. The PROC MIXED procedure of the SAS® software system Version 6.12 (SAS Institute, Cary, NC) was used for this analysis. Random effects were modeled for site in models in which the dependent measures were considered at the agency level and with respect to both site and agency for those models in which the dependent measure was at the dyad level.
4. The decline in the number of respondents by 10 from wave one to wave three at five sites, a 16.7% decline overall, primarily reflects the integration of mental health and substance abuse programs (60%) but also weakening involvement of general health programs (30%), and a consolidation of VA facilities at one site (10%).
5. An alpha level of less than 0.15 was used to determine statistical significance for these analyses because of the relatively small number of observations for testing the significant interactions between site/agency and time.
6. The existence of a fiscal relationship between two agencies was indicated by either the transfer of funds between the two agencies or the ability of one agency to influence the flow of funds to the other agency from third parties.
[ Go to Contents ]
To obtain a printed copy of this report, send the title and your mailing information to:
Human Services Policy, Room 404EFax: (202) 690-6562
Home Pages:
Evaluation of the CICH: Project Page
Assistant Secretary for Planning and Evaluation
(ASPE)
U.S. Department of Health and Human Services
(HHS)
Last updated: 03/20/08