Many communities in Appalachia still do not enjoy the same overall quality of life as in other parts of the country, with respect to material prosperity, health, education, safety, recreation, and other aspects of community well-being.  Further, there is a strong case to be made that these struggling communities are in need not just of external financial assistance or infusions of new industry and business, but also of a kind of transformation from within wherein whole communities come together to envision their future and awaken to their potential for collective action and improvement.  The ultimate goal of community capacity building is to recognize and develop untapped resources to improve the living conditions and quality of life of people in communities. Community capacity building involves:

• Purposive and planned action on the part of a representative cross-section of the community;
• The mobilization and participation of a broad, diverse coalition of citizens within a community;
• The generation of awareness of community issues and problems, as well as a sense of commitment, common purpose, and empowerment on the part of community members;
• The strengthening of human capital by equipping people with the skills, know-how, and creativity necessary to carry out common goals; and
• The establishment of dense collaborative networks across agencies, organizations, and individuals. 

Since 1995, the Appalachian Regional Commission (ARC) has invested nearly $12.5 million in 168 community capacity-building projects. [Many of the projects received additional funding from other agencies and organizations.] This report summarizes findings from an evaluation study of 100 community capacity-building projects funded by the ARC between 1995 and 2003.  Chapters 1 and 2 provide background information about the study and community capacity building.  Chapter 3 portrays the various community contexts in which the capacity-building projects were embedded, including information on grant recipients, community characteristics, assets, liabilities, and available resources. Chapter 4 presents findings on project activities and their implementation, including barriers encountered and how these were overcome.  Chapter 5 examines project accomplishments and impact, and chapter 6 addresses performance measurement.  The final chapter presents lessons learned and recommendations for ARC regarding its future capacity-building projects. 

Study Overview

In 2003, the ARC commissioned an evaluation of its civic capacity-building program.  The purpose of this evaluation was as follows:

• To document the range of outputs and outcomes that capacity-building and community leadership projects set for themselves in their proposals to ARC and to assess the extent to which these projects were able to accomplish their goals;
• To measure the extent to which these projects made progress toward the objectives [the objectives for ARC Goal 3 are (1) the percentage of Appalachian residents participating in leadership development programs aimed at community improvement will substantially increase, and (2) all communities and community organizations will have access to capacity-building activities to enhance their ability to marshal resources, plan, and develop solutions to local problems] and performance goals [the performance goals for ARC Goal 3 are (1) support 4,000 participants in leadership development and/or civic capacity programs, and (2) provide support to develop leadership and civic capacity programs for 10 additional counties per year] that pertain to ARC Goal 3;
• To document community outcomes that occurred as a result of these projects;
• To assess the implementation of a sample of these projects, with an emphasis on identifying obstacles and steps taken to overcome these obstacles;
• To identify potentially promising practices that might be adapted elsewhere; and
• To make policy recommendations aimed at improving ARC's efforts to monitor and assist its civic capacity-building and community leadership projects.

Within this context, two important purposes of the evaluation were to assess factors associated with successful capacity-building projects and to recommend a wide range of performance measures that might be used to document the impact of successful initiatives.  The study examines projects' implementation and accomplishments, assesses the extent to which they met their own objectives, and makes policy recommendations for a performance measurement system.

The evaluation of ARC's capacity-building projects included four integrated activities:

• A review of the literature regarding theoretical and applied perspectives on capacity-building and associated best practices;
• A review of applications and final reports to gain a better understanding of the purpose, scope, and accomplishments of the 100 projects in the study sample;
• Telephone interviews with a representative sample of 25 projects to obtain broad information about project-related activities and accomplishments; and
• Site visits with 12 projects to obtain more detailed information about project-related implementation experiences, accomplishments, impacts, and performance measures.

Project Context

All capacity-building projects are embedded in particular communities and, as such, must navigate unique contexts, factoring in interrelated sets of specific aims, assets, liabilities, and resources. Many of the communities in this study faced a common set of challenges:  geographic isolation, persistent poverty, unemployment, and declining population, often attributed to young people moving away for educational and employment opportunities that do not exist in their home communities.  Other problems mentioned by grant recipients include drug trafficking and addiction, teen pregnancy, and high rates of high school dropouts. 

The most common goals reported by the 100 grant recipients in the study were developing or expanding the pool of potential community leaders, reducing the community's sense of isolation, preserving natural resources, fulfilling or improving municipal services, providing previously unavailable opportunities to local youth, and improving the economic health of their communities.  Some projects reported interrelated goals.  For example, projects geared toward youth were often described as an attempt to stem the exodus of young people from the community, with the community's long-term economic health in mind.        

Each community had its own combination of assets and liabilities. The site visits and telephone interviews indicated that communities faced a variety of limitations or obstacles as they embarked on their capacity-building and community development work, such as the local political structure, lack of empowerment, fear of change, and limited resources.  Another pervasive problem for the communities in the study was limited resources, most notably funding, but also including nonmaterial resources such as technical expertise and available staff.  Even communities with resources that could be marshaled in the drive for change did not necessarily maintain a formal mechanism through which such resources could be distributed efficiently and effectively.

Community capacity building has at its core an assets-based approach to community development, emphasizing the identification and mobilization of community resource toward shared goals.  Community organizations, innovative leaders, concerned citizens, and natural resources are just a few examples of community assets that can be directed toward community improvement efforts.  In particular, the importance of community organizations—via collaboration, innovative leaders, and concerned citizens willing to get involved in community projects—was a prevalent theme echoed throughout the telephone interviews and site visits. 

Resources are the financial, organizational, and human inputs that communities rely on to conduct project activities.  Resources can include funding, ideas, expertise and technical assistance, time, materials, technology and equipment, and staff, among others.  Grant recipients who participated in the telephone interviews and site visits often stated that they took advantage of existing resources in the community to facilitate project activities. 

Projects varied considerably in the amount of ARC funding they received, from a low of $1,137 for a recycling program in Georgia, to a high of $335,000 for the West Virginia Flex-E-Grant Program. This wide range of funding suggests the breadth of activities that ARC funded capacity-building projects comprise, and also reflects different funding strategies used by ARC to foster capacity building in Appalachia.  Projects received approximately $5.3 million in matching funds from federal, state, local, and other sources. Most projects received aid from outside organizations in the form of technical assistance, outreach, and/or additional funds and other resources. 

Project Implementation

The approaches employed by the ARC-funded capacity-building projects reflect the diversity of the communities in which they operated.  Projects conducted numerous activities suited to particular project goals, and used strategies attuned to the particular needs, assets, liabilities, and available resources of their communities.  The activities conducted by ARC-funded community capacity-building projects were divided exhaustively into 11 activity types, with an other category for activities that were not categorized elsewhere. [Examples of activities categorized as other include the establishment of a 24-hour toll-free number for victims of domestic abuse, recycling activities, health screenings, and the renovation and expansion of a local history museum.]  For organizational purposes, similar activity types were grouped into four overall strategies:  vision and direction, involvement, skills and knowledge, and support.

Looking across all projects, the single most prominent activity—cited by 51 percent of projects—was conducting a group instructional activity such as a workshop or course (see table E-1).  Other common activities included strategic planning (35 percent), meetings, conferences, and forums (28 percent), and technical assistance and consultation (26 percent). 

Source: Document review of ARC grantees.

Almost two-thirds (62 percent) of projects relied on strategies designed to enhance skills and knowledge—e.g., organizing and conducting group instructional activities, conducting one-on-one instructional activities, and facilitating the distribution of instructional materials.  Most of these training programs focused on the promotion of leadership skills (38 projects), but other topics were also addressed—e.g., economic development (8 projects), civic development (5 projects), and technical issues such as strategic planning and grant writing (5 projects) (see table 4-4 in Chapter 4).  Most (51 of 62) projects used group instruction—and this group instruction was most frequently aimed at adult community members (17 projects), community and business leaders (15 projects), and youth (14 projects) (see table 4-4).  

The problems that grant recipients encountered were generally not severe enough to prevent projects from implementing their approach.  The most commonly cited problems included time and staffing constraints, attracting participant interest in the project, the isolation of and competition between communities, and limited resources.  Projects were able to troubleshoot problems related to implementation, as demonstrated by the fact that the overwhelming majority of these projects remain active.   

Project Accomplishments and Impact

Site visit and telephone interview findings provided significant qualitative evidence that the ARC-funded projects were largely successful at building capacity at different levels. There were three main types of capacity outcomes:  individual, organizational, and community. At the individual level, projects enhanced the skills and knowledge of community members, increased awareness of community issues, and developed people's sense of empowerment. Many projects benefited organizations by increasing collaboration and the sharing of ideas and strategies for community development, and by enhancing their efficiency and effectiveness, as well as their stability and growth. Projects also benefited communities more broadly by improving strategic planning, enhancing the sense of community self-reliance and pride, increasing civic and political participation, and improving infrastructure and educational opportunities.  

Study findings reveal that many projects had outcomes and achievements at more than one level of capacity, suggesting a richness and efficiency of approach on the part of projects with respect to capacity building. Qualitative evidence from across the site visits and telephone interviews support the view that many projects had far-reaching effects on their communities. Some involved significant changes in orientation and attitude, toward both communities in general and social and political duty and service in particular.  Many projects led to greater awareness of community issues, a greater sense of community pride and self-reliance, and a stronger commitment to community service. Also, many projects were successful at convincing people of their own capacity for self-improvement and change. Besides these important psychological and attitudinal changes, projects gave rise to more concrete benefits, including the development of individual skills and knowledge, increased collaboration, the strengthening of community organizations and infrastructure, increased volunteerism, and improved planning.       

Performance Measurement

A systematic and accurate assessment of project achievement requires a performance monitoring system that can be used to demonstrate that projects carried out their proposed activities, and the extent to which those activities ultimately benefited individuals and communities. For such a system to work effectively, activities and related benefits must be clearly defined and measurable.  Further, projects should have realistic plans to obtain valid and reliable data for documenting progress toward their outcomes.

The benefits of ARC's community capacity-building projects to individuals, organizations, and communities were far reaching and significant.  One could argue that such enhanced capacity has paved the way for longer term economic, environmental, and social benefits, as well as increased community assets and decreased liabilities. However, these conclusions about program impact are primarily based on the observations, reflections, and judgments of project staff and participants themselves, rather than on more concrete and documented forms of evidence. Claims of project success could not be substantiated with hard data.  In some cases, the desired outcomes that projects described in their proposals were not sufficiently clear, specific, and measurable—and therefore could not provide data-based tangible evidence of project success.  In other cases, projects did not invest in the data collection activities required to demonstrate the immediate or long-term impacts of the efforts. 

Moreover, the definition of clear and measurable outcomes is notoriously difficult for community capacity-building projects. One reason is that many of the benefits resulting from such efforts, such as enhanced community pride, empowerment, and community self-reliance, are not easily quantifiable. Further, many of the more quantifiable outcomes are hard to measure in a cost-effective manner.  For small-scale and/or short-term projects with limited resources for data collection, it is even more difficult to assess whether the activities ultimately contributed to longer term economic, environmental, or social outcomes.  In any event, over the long term, these obstacles will ultimately hinder the Commission's ability to document the range of benefits resulting from its community capacity-building projects.

Most (85 percent) of the 179 outcomes proposed by the 30 projects were capacity-based (i.e., individual, organizational, or community). [In fact, of the 30 site visit and telephone interview sites included in this analysis, only one project did not propose at least one capacity outcome.  This particular project proposed three economic/increased employment outcomes, all involving an increase in new jobs for project participants. The project might have also proposed an increase in individual skills as an outcome, but did not.] The remaining 26 outcomes (15 percent) were development outcomes (economic, environmental, or social), suggesting that most projects were setting their sights on shorter-term capacity-building goals (see table E-2). 

Of the 179 outcome statements, 55 (30 percent) were classified as individual, 25 (14 percent) as organizational, and 73 (41 percent) as community outcomes. Of the 26 developmental-based outcome statements, 11 outcome statements (6 percent) were economic, 7 (4 percent) were environmental, and 8 (4 percent) were social outcomes. Across the 179 outcome statements that we reviewed, the most commonly proposed included enhanced skills (22 percent), increased civic participation (15 percent), improved planning (11 percent), and increased collaboration (9 percent).

*No outcomes statements were proposed.

Most (70 percent) of the 179 outcomes proposed by the case study and telephone interview projects were successfully achieved (see table 6-3).  However, projects appeared to have difficulty obtaining data that could be used to ascertain whether an outcome—especially developmental outcomes—had been achieved.  This mirrors our finding from the site visits that the evidence provided in support of project success was mostly anecdotal.  In fact, most of the 30 case study and telephone interview projects did not collect any data about project implementation and impact.  Interviews with project staff revealed several reasons for this—e.g., the difficulty of collecting data, a lack of resources or funding for data collection activities, lack of expertise and experience in data collection and evaluation methods, and the belief that such data collection was not formally required by ARC.  In addition, only a few of the projects appeared to believe that the collection of such data would further their own immediate interests (e.g., to attract future funding, improve activities), and some viewed data collection as an external imposition.

Recommendations

The weight of both the qualitative and quantitative evidence indicates that a clear majority of projects succeeded in achieving real results. Nevertheless, if the ARC wants to be in a position to demonstrate the impact of its community capacity-building projects, it will have to do a better job in the future of measuring, tracking, and reporting performance of the program.  Our recommendations are designed to build upon our previous recommendations by identifying specific steps that ARC might take to enhance the application and reporting guidelines for its community capacity-building projects.

Reinforce the ARC application materials provided to community capacity-building applicants.  First, we recommend that ARC reinforce the general blueprint set forth in earlier ARC publications by developing additional materials aimed at helping applicants consider the range of steps required to execute their proposed approach and document the resulting community benefits.  Exhibit E-1 presents seven guiding questions that community capacity-building applicants might address in their proposals.  Beginning with project purpose, these questions are designed to help applicants consider the range of issues they will need to address over the life of their project—most notably the link between their proposed approach and the capacities they are trying to enhance, the community conditions they are seeking to improve, and the data they will use to document project success.

Provide prospective applicants with examples of community capacity-building outcomes.   In an effort to help prospective community capacity-building projects address these questions in their proposals, we recommend that the ARC provide applicants with supplemental materials that demonstrate the range of outputs and outcomes that might be attributed to a generic community capacity-building project.  The logic model presented throughout this report provides an example of the type of tool that might be posted on the ARC web site. The model, which portrays the relationship between a project's intended purpose, activities, and outcomes, can be used to provide applicants with illustrative examples of intermediate and long-term outcomes that are commonly associated with community initiatives (see appendix A). 

We further recommend that the Commission work more closely with individual applicants to help them specify (1) the goals and outcomes that their projects are designed to address, (2) the numeric benchmarks against which their progress can be assessed—and the timeframe within which these benchmark will be achieved (see exhibit E-2), (3) a description of the methodologies that will be used to assess whether the numeric benchmark was achieved (see exhibit E-3), and (4) a description of how and when the data will be reported to the ARC.  While some of this could be accomplished by posting a tool like the logic model on the Commission's web site, many projects will likely benefit from having extended discussions with ARC staff (or other experts recommended by the ARC) on topics pertaining to performance measurement.  [The introduction to the logic model includes an overview of how to transform goals and outcomes into measurable benchmarks.  See the electronic attachment for a full working version of the logic model.]

Provide projects with written materials on potential data collection and analysis practices.  While collecting data can be difficult and time consuming, findings from the literature review and site visits suggest that some of the outcomes that projects propose can be addressed through analysis of easily accessible existing records—e.g., county tax records, school enrollment records, police reports, employment and unemployment statistics, organizational meeting minutes, medical records, and hotel receipts.  These data can often be found in county offices or the local chamber of commerce, as well as through such state and federal agencies as the Bureau of the Census, Department of Education, Department of Agriculture, Department of Housing and Urban Development, and Department of Commerce.  By publicizing these existing sources, ARC may be able to help projects devise strategies for obtaining and analyzing public data that can be used to demonstrate improvements in a given condition (e.g., unemployment, hotel occupancy rates) over time.  For other outcomes, projects may need to be encouraged to collect new data through one-time surveys, pre/post surveys and assessments (e.g., to measure changes in participants' skills or knowledge), and interviews (conducted in person or over the phone). 

The ARC may need to help individual grant recipients develop low-cost plans for collecting and analyzing data.  While working with individual projects—collectively or in a workshop setting—can be time consuming, we believe that the potential rewards are substantial.  By their very nature, community capacity-building projects should be making continuous use of data to inform their consensus-building and decision-making efforts.  As such, the Commission's efforts to help community capacity-building projects make maximum use of data would represent a valuable investment with many long-term benefits.

Reinforce ARC's reporting structure. If the ARC is to be in a position to identify innovative and successful community capacity-building practices, its staff will need to be able to systematically access more detailed information about the implementation and impact of its projects.  We therefore recommend that the ARC develop uniform closeout report guidelines that are to be used by all of its community capacity-building projects.  Exhibit E-4 provides an example of the topics that the ARC's community capacity-building projects might be required to address in their final reports.  Collected over time, we believe this information would enhance the ability of program staff to assess the implementation and impact of the Commission's community capacity-building projects.  While some grant recipients might continue to rely primarily on anecdotal information, the use of standard reporting guidelines—coupled with the requirement that projects report on their intermediate and long-term outcomes—would likely enhance the Commission's ability to obtain consistent data that can be used to assess project and program success.  In addition, interviews with case study respondents suggest that projects would actually welcome more structured reporting requirements, so long as those requirements are not onerous or unrealistic.