Detailed Information on the
Appalachian Regional Commission Assessment

Revise Memorandums of Understanding between ARC and other federal agencies in order to improve access to information, to better describe the collaborative work relationship, and to articulate reporting requirements necessary for ARC to meet current OMB A-136 reporting requirements.

Reduce the number of ARC project close-outs by 10%.

Update and improve the agency website in accordance with federal transparency guidelines and to enable other agencies to use ARC research and information as a tool and to encourage increased collaboration in community and economic development throughout the region.

Measure: ADHS miles opened

Explanation:Goal is to maximize investment in Appalachian Development Highway System, which is intended to reduce the region's economic isolation.

Measure: Households provided with basic infrastructure services

Explanation:Households served with basic water and sewer infrastructure

Measure: Number of new jobs created (cumulative)

Explanation:Enhanced employability is a key aspect of improved regional development.

Measure: Number of citizens benefitting from enhanced education and job-related skills

Explanation:Key element for improving regional economy

Measure: Number of participants in job training and education programs that demonstrate results (i.e., expand worker skills, obtain a job, increase in educational attainment and achievement)

Explanation:Workforce training and enhanced educational performance of the region's students are key to helping the region compete in the global economy

Measure: Average grants processing time

Explanation:Time from receipts of grant application in proper order to disposition

Measure: Direct 50% of funds to distressed counties/areas

Explanation:Direct half of ARC grant funds to benefit distressed counties or areas

Measure: Number of new jobs created/retained

Explanation:Enhanced employability is a key aspect of improved regional development.

Measure: Number of households provided with basic infrastructure services

Explanation:

Measure: ADHS miles opened

Explanation:Goal is to maximize investment in Appalachian Development Highway System, which is intended to reduce the region's economic isolation.

Measure: ADHS miles opened

Explanation:Goal is to maximize investment in Appalachian Development Highway System, which is intended to reduce the region's economic isolation.

Is the program purpose clear?

Explanation: Congress established ARC in 1965 to reduce the substantial socioeconomic gaps between Appalachia and the rest of the nation. The establishing legislation states "It is, therefore, the purpose of this Act to assist the region in meeting its special problems, to promote its economic development, and to establish a framework for joint Federal and State efforts toward providing the basic facilities essential to its growth and attacking its common problems and meeting its common needs on a coordinated and concerted regional basis." Although ARC has made progress in economically developing the region, substantial gaps still exist (see 1.2).

Evidence: A 1964 study discussed the long-standing deficits in Appalachia (Report of the Presdient's Appalachian Regional Commission). This report endorsed the Federal-State partnership model that eventually became the basis of the ARC. This report is available at www.arc.gov/index.do?nodeId=2255. See also findings and statement of purpose in the Appalachian Regional Development Act (ARDA), available at www.arc.gov/index.do?nodeId=1243.

Does the program address a specific and existing problem, interest or need?

Explanation: The Appalachian region historically has lagged behind the rest of the country in terms of employment, income, education, health, and quality-of-life. These problems have produced concentrated high poverty areas, persistent unemployment, low incomes, inadequate health care, educational disparities, and out-migration. Although investments in the region and growth in entitlement programs have increased parity between the region and the rest of the nation, the region still lags behind. For example: 1) ARC counties have a higher unemployment rate than the national average, and 145 counties exceed it by 150%; 2) ARC counties trail the rest of the nation by 18% in per capita income; 3) number of residents with a college degree is 70% of the national average; 4) Appalachian residents have higher rates of strokes, heart disease, diabetes, and other preventable diseases relative to the rest of the country; and 5) the region has substantial infrastructure needs, as 30% of households are not connected to centralized wastewater treatment and 15% in Central Appalachia lack both public water and wastewater services.

Evidence: FY 05 congressional justification. Overview of Appalachian statistics, www.arc.gov/index.do?nodeId=26.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: Concerns regarding the duplication of effort exist and are well-documented. GAO recently identified 73 federal programs that can be used for economic development activities, or for activities that could be considered related to economic development. These programs cover rural and urban populations in communities across the country and include an element of local planning in the use of funds. The multiplicity of federal programs imposes transactions costs on localities attempting to shift through the array of federal programs and creates limitations on creatively packaging federal resources. However, ARC does perform a unique function in coordinating federal resources to the region. Poor and highly distressed counties may be at a disadvantaged in identifying opportunities and making effective business cases for grant funds. Often they are unable to provide matching funds that are required by other grant-making agencies and institutions. In other cases, the funding needed to initiate an innovative solution is so small that it falls below the minimum amount provided by some agencies. ARC's consensus model ensures close collaboration and gives the Commission a non-federal character that distinguishes it from typical federal executive agencies and departments.

Evidence: Sept. 2000 GAO study. - Multiple Federal Programs Fund Similar Economic Development Activities. Ten agencies and 27 subagency units administer 73 programs that can be used to support one or more of the six activities directly related to economic development -- planning; constructing or renovating non-residential buildings; establishing business incubators; constructing industrial parks; constructing roads and streets and constructing water and sewer systems. For example, ARC, the Economic Development Administration, US Department of Agriculture and the Department of Housing and Urban Development all help finance infrastructure investments such as waste water treatment facilities. ARC differs from other Federal, State, local, and private efforts because it is based on a collaborative model involving partnership with other federal, state, local and private organizations. A 14-member Commission governs the partnership. It is comprised of: a federal co-chair, appointed by the President and confirmed by the Senate, and the thirteen Appalachian state Governors

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: ARC's program design is free of major flaws that limit effectiveness or efficiency. ARC is a federal/state partnership, with the Federal co-chair having veto power. The states contribute 50% of the operating costs. The competitive grants structure and the ability of the Federal co-chair to veto any project together ensure that projects that are most likely to help the region reach its long-term goals are achieved. ARC also ensures that localities are aware of other Federal, State, and private resources available to further develop the community and ARC provides the assistance necessary to ensure that the communities are able to apply for the available resources.

Evidence: Two independent studies found that ARC's coordinated investment strategy has paid off for the Region in ways that have not been evident in other parts of the country without a regional development approach. A study in 1995 funded by the National Science Foundation compared changes in Appalachian counties with their socioeconomic twin counties outside the Region over a 26-year period. This analysis, controlled for factors such as urbanization and industrial diversification, found that the Appalachian counties grew significantly faster than their economically matched counterparts outside Appalachia. A more recent similar analysis by East Carolina University compared Appalachian counties with matched non-Appalachian counties in the southeastern states, with similar findings.

Is the program effectively targeted, so that resources will reach intended beneficiaries and/or otherwise address the program's purpose directly?

Explanation: ARC classifies all Appalachian counties by economic condition, using factors such as poverty, unemployment, and per capita income. Counties are arrayed on a continuum, including economically distressed counties, transitional counties, competitive counties, and attainment counties. At least 50% of ARC's grant funds flow to activities that benefit the region's distressed counties. It is important to note that project funds are used in distressed areas, even if those areas are in a non-distressed county, as competitive and transitional counties have pockets of distress within them. Distressed counties are eligible to receive up to 80% of the project cost from ARC, whereas transitional counties can only obtain up to 50% and competitive counties up to 30%. ARC funds do not support projects in attainment counties.

Evidence: Congressional Justification, Performance & Accountability Report, ARC Strategic Plan, establishing legislation. See www.arc.gov/index.do?nodeId=100.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: ARC's long-term goal is to bring the region into parity with the rest of the nation, defined as the number of distressed counties relative to the nation. Long-term measures associated with these strategic goals include: (1) number of jobs created or retained, (2) number of citizens benefiting from enhanced education and job-related skills as a result of ARC investment, (3) number of households served with new or improved water and sewer infrastructure and (4) number of miles of the Appalachian Development Highway System completed. Overall, ARC's strategic plan focuses on the key outcomes of ARC. However, performance measures could be improved to include stronger indicators of economic and social change. For example, the number of jobs created could be supported by measures documenting trends in wage growth. The Appalachian Regional Commission is working with OMB and other federal agencies to define common performance measures for community and economic development programs.

Evidence: Congressional Justification, Performance & Accountability Report, ARC Strategic Plan. See www.arc.gov/index.do?nodeId=100.ARC has recently adopted a new Strategic Plan for 2005-2010. The four goals underpinning this plan include: (1) Increase job opportunities and per capita income in Appalachia to reach parity with the nation, (2) Strengthen the capacity of the people of Appalachia to compete in the global economy, (3) Develop and improve Appalachia's infrastructure to make the region economically competitive, and (4) Build the Appalachian Development Highway to reduce Appalachia's isolation. ARC developed targets and strategies to meet the long-term goals. For example, to support the goal of creating or retaining 120,000 jobs by 2011, ARC has developed strategies that include developing workforce training programs, improving access to investment capital for local businesses, and identifying local and regional assets for development.ARC funded a study of water and sewer infrastructure gaps in FY 2003.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: ARC's long-term targets are based on existing baselines and conditions, the potential for success in ARC's strategies, and the existence of exogenous conditions. The 2011 timeframe acknowledges that the problems are deeply-rooted and endemic, yet fixable. However, for most performance measures there is significant 'double-counting' of performance. For example, while ARC contributes less than 6 percent of federal dollars to projects encouraging job creation and retention and ensuring adequate water and sewage infrastructure, ARC claims 100 percent credit for number of jobs created and number of households served. While federal agencies should be in no way penalized for leveraging other federal dollars, ARC efficiency measures should consider all federal dollars. ARC, EDA, USDA, HUD and OMB are currently discussing appropriate metholodolgies for reporting performance.

Evidence: Congressional Justification, Performance & Accountability Report, ARC Strategic Plan. See www.arc.gov/index.do?nodeId=100.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: ARC has recently refined its strategic plan and created a 'logic model' to align annual performance measures and strategies with long-term goals. Annual measures and targets include: (1) Create/retain 20,000 jobs for Appalachians, (2) position 20,000 Appalachians for enhanced employability (workforce training), (3) provide 20,000 households with basic infrastructure services, (4) provide broadband service to 5 communities for every $1 million invested, (5) build 25 miles of the Appalachian Development Highway System (ADHS), and (6) achieve a 4:1 average private sector investment ratio for projects. ARC is also using two efficiency measures: (1) ADHS miles completed per $100 million invested and (2) average grants processing time. The first efficiency measure attempts to assess whether the completion of the Appalachian Development Highway System is proceeding in a cost-efficient manner. Targets are estimated based upon terrain and route characteristics and cost of highway structures. To help measure the cost-effectiveness of federal economic development programs, ARC should track the cost per job created, measured by the amount of federal funds needed to create or retain one job. However, before any such measure is used, ARC should coordinate with other federal agencies to ensure a consistent methodology is being applied.

Evidence: "Moving Appalachia Forward" Appalachian Regional Commission Strategic Plan 2005-2010. Draft, October 1, 2004. FY 2003 Performance and Accountability Report FY 2005 Budget Justification

Does the program have baselines and ambitious targets for its annual measures?

Explanation: See the concerns raised on "double-coutning" in response to question 2.2.

Evidence:  

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: ARC ensures that its partners work toward its strategic goals through several mechanisms. First, ARC established and works with 72 local development districts (LDDs), which serve as multi-county planning and development organizations. These LDDs serve as the liaison between ARC and the localities and regions. Several ARC evalulates potential projects through a competitive process, and a primary consideration is whether a proposed project would further ARC's strategic goals and objectives. Third, grantees are required to include performance measures that support ARC goals and to report these measures as part of the grant approval and monitoring process. Finally, when more than one Federal agency funds a project, a Memorandum of Understanding ensures that one agency has the lead for supervising the project and all agencies agree to the expected outcomes of the project.

Evidence: State development plans, strategic plan, memoranda of understanding with LDDs and Federal agencies.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: Evaluations focus on the extent to which the projects have achieved their objectives. Evaluations have addressed a wide range of outcomes, including the efficiency and economic benefits of the ADHS, the impact of public works projects on income and job creation, the educational benefits of ARC's education programs, and the results of ARC's economic diversification program. In addition, a 'twin counties' study compared actual changes in economic growth in Appalachia with the changes that would have occurred irrespective of ARC's programs. By comparing Appalachian counties with non-Appalachian counties with similar characteristics, the study assessed the extent to which ARC counties grew faster than their 'twins' by measuring growth rates over 22 years and considering 20 variables (e.g., per capita income, earnings by place of work, and population). While the study found that the counties of Appalachia grew faster than their control-group twins, the study did not uncover relationships between ARC's programs and economic development in the region. Additional research assessing trends in individual counties over time might provide greater understanding about the effectiveness of specific programs.

Evidence: In the last five years, ARC has conducted 32 evaluation and research studies that address program results and strategies. These evaluations have used a variety of techniques, and the most useful have established the counterfactual condition of what would have happened without ARC's involvement through a quasi-experimental or comparative framework. Evaluations have been conducted by independent outside researchers (commissioned by ARC to complete the evaluations) and have covered ARC's work over several decades. Isserman, A. and T. Rephann. The Economic Effects of the Appalachian Regional Commission: An empirical assessment of 26 years of regional development planning. APA Journal. (Summer) 1995: 345-363.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: Budget requests are tied to annual and long-term performance goals. For example, the FY05 budget links ARC's funding requests to specific strategies and performance measures and is allocated based on program evaluations. The expected economic benefits are quantified and presented with the performance measures. The budget also provides full costing of each performance goal.

Evidence: FY2005 Congressional Justification, available at www.arc.gov/images/newsandevents/publications/fy05budget/05budget.pdf

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: ARC has conducted a year-long strategic planning revision process that has involved Federal, State, local, and regional representatives coming to agreement on a new strategic plan through a series of field forums. The new strategic plan became available in the summer of 2004. This plan takes into account the changing economic and community conditions in the region. The plan also reflects ARC's priorities (established through the field forums), focuses more on outcomes, and provides a tighter linkage between long-term goals and annual goals. ARC is also working with the Interagency Collaborative on Community and Economic Development (ICCED) and OMB on a cross-cut assessment of federal community and economic development programs. This assessment will, among other things, establish a common set of measures to assess program performance in this area.

Evidence: "Moving Appalachia Forward" Appalachian Regional Commission Strategic Plan 2005-2010. Draft, October 1, 2004. FY 2005 Budget Justification

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: ARC regularly collects performance data from grantees and validates these data through 50-60 site visits each year (about 10 percent of projects). Site visits are conducted two years after the award of a grant to give grantees time to finish projects and track outcomes. In addition, ARC uses ARC.net, its Management Information System (MIS) to track critical project performance information. ARC staff reviews performance measurement data generated by programs throughout the fiscal year to analyze trends and validate data. ARC used this information to inform its recent strategic planning revisions and routinely shares such information with program partners through 'best practices,' conferences and on-site validation visits with grantees. A recent example of how ARC has used performance information to improve program management and direction include recent evalautions on health disparities in Appalachia that resulted in ARC investing in a joint research project with the National Institutes of Health (NIH). In recent years, ARC's Policy Development Committee has also used research, evaluations, validation visits and staff monitoring to develop and revise program guidelines for revolving loan funds, tourism development, export trade and telecommunications.

Evidence: Congressional Justification, Annual Performance Plan, and other strategic planning documents. Best practices listed on ARC's online resource center at www.arc.govPerformance information is also shared in a number of best practice forums and conferences including:"The New Appalachia: Ideas that Work" (1999); (2001)The New Appalchia Conference Programs. Recent conferences has focused on (1) capacity building and collaboration, (2) education, and (3) telecommunications.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: ARC routinely monitors the performance of all project managers to ensure they are focused on outcomes and adhere to the milestones and schedules outlined in planning documents. ARC withholds funds from underperforming projects and will cease funding projects if performance does not improve.

Evidence: FY 2003 Performance and Accountability Report Appalachian Regional Developmeht Act of 1965 (as amended March 2002).

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: As a grant program, ARC is not expected to obligate all funds in the year in which they are appropriated; this is the nature of no-year funds. Since grant projects often take several years to complete, funds that are obligated in a timely manner can still straddle fiscal years. ARC works closely with Federal partners and states to obligate funds as quickly as possible and to ensure that funds are spent for intended purposes. Award recipients must produce periodic financial status reports, and the IG conducts field audits on 15 grants per year.

Evidence: ARC annually approves 100 percent of project dollars, but some of these funds must be obligated and deobligated by its Federal partner agencies. See ARC spending and audit reports.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: ARC uses a strong "filtering process" that ensures that the grants that provide the greatest benefit-cost ratio to the Appalachian region are funded. First, states prepare investment plans that align with the ARC strategic plan. This ensures that priorities are aligned. States work with LDDs to identify the most efficient grant investment opportunities next. Finally, ARC reviews performance and related information about all proposed grants; the Federal Co-Chair may veto any grant that is not aligned or does not appear to offer a strong return-on-investment. In addition, ARC uses several efficiency measures in its strategic planning process. ARC considers the ADHS one of the most critical aspects of its work, and an efficiency measure related to it is ADHS miles completed per $100 million in investment. In addition, ARC uses average grant processing time as a matter of efficiency. Finally, the agency works closely with other Federal agencies (such as EDA, USDA, and HUD) to coordinate funds and not duplicate efforts, ensuring the most efficient use of its monies. ARC is also working with other Federal agencies on establishing a common methodology for calculating the cost per job created or retained as a result of program investment.

Evidence: See ARC strategic plan, ARC congressional justification, ARC Project Guidelines and MOUs with other Federal agencies.

Does the program collaborate and coordinate effectively with related programs?

Explanation: ARC has always emphasized collaboration with public and private resources to accomplish its mission. ARC was established to leverage resources and seek out partners to address Appalachia's problems. ARCs operations are based around a partnership model, with state and local governments working with the Federal government in the decision-making and governance structure. ARC coordinates extensively with other Federal agencies. About half of past ARC grants have been administered under agreement with 12 other Federal agencies. This achieves consistency in program objectives, creates efficiency in resource allocation, and aids in compliance with applicable laws such as environmental, safety, and labor requirements.

Evidence: ARC establishing legislation, ARC congressional justificationand MOUs with other Federal agencies.

Does the program use strong financial management practices?

Explanation: ARC's accounting system has been certified for government use by the Joint Financial Management Improvement Program. Policies and procedures are in place to ensure that payments are properly made to the intended parties. ARC undergoes an annual financial audit, and ARC's grant recipients' financial activities are independently reviewed by the IG periodically. ARC recieved a clean opinion on its latest financial statements (for FY03) and has no material internal control weaknesses.

Evidence: ARC provides comprehensive information about its financial management practices and performance in its Performance Accountability Report. Pages 58-83 of the PAR speak to ARC's financial report, including the report of the independent audit on page 59. The ARC website includes the OIG semi-annual reports. Financial management information is publicly available and transparent.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: ARC conducts formal and information meetings with division managers to identify vulnerable areas and potential control weaknesses, and has an internal management control committee to conduct reviews. ARC conducts program evaluations on an ongoing basis to examine the effectiveness of its programs and progress in achieving outcomes. Program management devidiencies, when identified, are included in these reveiws and acted upon. Recommendations included in the latest IG report are currently being implemented.

Evidence: For example, one nonmaterial weakness related to tracking of advanced payments for grantees came up in the last financial audit. In response, ARC has implemented improved procedures and data management to eliminate this condition. See ARC congressional justirfication, IG audits, and the Performance and Accountability Report (PAR)

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: ARC awards grants based on a combination of formula allocation and competition. Each fall, ARC allocates its annual appropriation from Congress among the Commission's member states. Funds are distributed by formula in three broad categories: highways, area development, and LDD support. After states have received their allocations, the governors of those states work with LDDs to prepare strategy statements of how they plan to use ARC funds. These statements link state priorities to ARC's goals and include lists of projects that the governors will submit to ARC for funding to implement their development strategies. Each state's process is competitive. The Federal Co-Chairman then reviews and must approve the state spending plans; he or she has veto power over projects that are not in the best interest of the region. Each proposed project receives a thorough review by ARC program analysts. To be approved by ARC, the projects must both support the local state development plan and ARC's strategic goals.

Evidence: FY 2003 Performance and Accountability Report Appalachian Regional Developmeht Act of 1965 (as amended March 2002). ARC Project Guidelines

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: In general, grantee performance is carefully monitored, although improvements are still warranted. ARC's validation visits are a critical component of this process, and this provides ARC a chance to confirm program outcomes and better understand the consequences of its programming and make policy or procedural changes as the need arises. In situations where a project fails to meet proposed goals, ARC considers mitigating circumstances and looks for possible trends in an effort to assist other projects facing similar circumstances. Analysis from the field validation visits is compiled in an annual internal report. However, recent IG reports have cited a need to improve oversight of ongoing grants to ensure grantees meet reporting and documentation requirements. For example, IG reports cite instances where grantees have been given further funding although they had not yet submitted required status reports for previous expenditures and cases where inactive funds remain allocated to expired grants. The IG has recommended that ARC develop policies and procedures to obtain the accurate status of funds held by grantees at the end of the fiscal year. The IG has noted, however, that ARC has begun to take appropriate to bring program managers more directly into the oversight function and has not cited these areas as material.

Evidence: FY 2003 Performance and Accountability Report Inspector General's Semiannual Report to Congress (October 1, 2003-March 31, 2004). Appalachian Regional Developmeht Act of 1965 (as amended March 2002). ARC internal audit reports.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: ARC collects grantee performance information using both interal and external methods. The agency collects grantee performance data and publishes it in its Performance and Accountability Report. Annually, ARC contracts with outside organizations to evaluate ARC programs. In addition, ARC places grantee performance information on its website (www.arc.gov).

Evidence: FY 2003 Performance and Accountability ReportGrantee information available on ARC's website

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: ARC has recently refined its long-term measures and baselines have been established for FY 2004. However, data on socioeconomic conditions in Appalachia and results from recent performance evaluations suggest that ARC has made progress in addressing the needs of Appalachian communities. Since ARC was established the region's poverty rate has been cut in half, the infant mortality rate reduced by 67 percent, the percentage of adults with a high school education has doubled, and over 1.6 million new jobs have been created. A recent study comparing Appalachian counties with a similar set of counties outside the region shows that the counties of Appalchia grew faster than their 'control-group twins.' (Isserman & Rephann, 1995). More recently, the number of severely distressed counties has decreased from 121 in 2003 to 91 in 2004. However, it is difficult to attribute these changes directly to ARC investment, particularly since ARC's impact is relatively small--approximately $60 million (not including ADHS) out of $25 billion in Federal dollars going to the region. In the eight states participating in ARC's educational programs, college matriculation rates have increased 15- 35 percent. This evidence suggests that ARC's programs are having an impact on the region.

Evidence: FY 2003 Performance and Accoutability Report Isserman, A. and T. Rephann. The Economic Effects of the Appalachian Regional Commission: An empirical assessment of 26 years of regional development planning. APA Journal. (Summer) 1995: 345-363. Other ARC long-term measures assess the number of jobs created or retained, number of Appalachina benefiting from enhanced education and job-related skills, number of households with basic infrastructure services and the expansion in regional access as the Appalachian Highway System is completed. While these measures are important and must be tracked, success will also ultimately depend on longer-term socioeconomic trends in the region (e.g., per capita income, poverty rates and college graduation rates). Per PART guidance, ARC received a "small extent" because discussion are ongoing regarding developing an appropriate methodology for tracking performance. Furthremore, it is very difficult to establish a link between ARC and regional economic and social changes.

Does the program (including program partners) achieve its annual performance goals?

Explanation: ARC has generally met or exceeded annual goals. However, as noted in answer to question 2.3, ARC and other federal agencies are currently discussing appropriate performance measures and methodology for community and economic development programs. Therefore, ARC has received a 'small extent' to this question. As ARC is only approximately 6 percent of funding for area development projects, performance measures should accurately reflect ARC contribution to outcomes and outputs. One alternative would be for ARC to calculate the federal cost per output or outcome, thereby crediting ARC with leveraging private investment, but also allowing for comparisons among federal community and economic development programs. However, as the Performance and Accountability Report demonstrates ARC is quite successful in meeting their annual targets and in general is tracking the right types of outputs and outcomes.

Evidence: Annual measures include: Education and Workforce Training ' Number of employees receiving basic education and skills training and the number of participants obtaining or retaining employment as a result of labor force training projects ' Number of students participating in school readiness, drop-out prevention, school-to-work transition and GEG programs and number of students documenting success in those program areas. Region's Physical Infrastructure ' Number of households with basic services and infrastructure for water, sewerage, and waste management. Jobs and Income ' Number of jobs created and retained Transportation ' Number of miles builds of the ADHS ' ADHS miles completed per $100 million investment Other ' Average private sector investment ratio ' Grants processing time NOTE: ARC's Strategic Plan and FY06 performance plan have realigned measures and strategic goals from FY03 PAR

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: ARC has shown some increases in efficiency. For example, grants processing times have been reduced from 60 days to 45 days in 2004, with the goal to further reduce to 30 days. In addition, ARC tracks and measures efficiencies in the cost to complete the ADHS. While the number of miles completed per $100 million has decreased slightly, this is consistent with anticipated costs to complete the highway in areas with more difficult terrain and the cost of expensive highway structures. Finally, ARC has exhibited organizational efficiencies by becoming a flatter organization and allowing other federal agencies to manage grant projects. Staff has been reduced from a high of 125 FTE to 52 FTE. A measure tracking unit costs such as the cost per job created or retained and the cost per infrastructure investment would also help ARC track programmatic efficiencies over time.

Evidence: FY 2003 Performance and Accountability Report

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: ARC's performance can be compared with other regional authorities (Denali Commission and Delta Regional Authority) as well as Federal partner programs (HUD, EDA, USDA-RD). While no comparative study has evaluated the relative strengths and weaknesses of community and economic development programs, ARC's strategic plans, competitive grant processes, performance measures and evaluations compare quite favorably with other similar agencies. As such, ARC has received a 'large extent' to this question.ARC's strategic planning process and organizational structure provide some unique advantages to its programs. Due to its partnership model, ARC provides a good forum to address the socioeconomic issues facing the region. ARC partners with federal, state, and local organizations and is quite effective at leveraging private and other federal funding to the region. Its 'bottom-up' approach that fits with ARC's overarching strategic framework helps ensure projects address local priorities but also regional strategic goals. However, comparisons between ARC and other federal programs could be improved, however, if agencies used similar methodologies for reporting performance. For example, it is difficult to assess per unit costs among the different federal programs.

Evidence: FY 2003 Performance and Accountability Report and Draft Strategic PlanPerformance and Accountability reports for EDA, HUD, USDAStrategic plans for Denali Commission and Delta Regional AuthorityIsserman, A. and T. Rephann. The Economic Effects of the Appalachian Regional Commission: An empirical assessment of 26 years of regional development planning. APA Journal. (Summer) 1995: 345-363.The Impact of CDBG Funding (October 2002) found at: www.huduser.org/publications/commdevl/cdbg_spending.htmlCost Per Job Associated with EDA Investments in Urban and Rural Areas (Pennsylvania State University, 2002)Public Works Program: Performance Evaluation (Rutgers University, 1997) Public Works Program: Multiplier and Employment-Generating Effects. (Rutgers University, 1998)

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: Prior evaluations that have been completed include evaluations of ARC's telecom projects, vocations education/workforce training projects, educational projects, entrepreneurial initiative, infrastructure/public works programs, and ADHS. Evaluations generally use a comparative framework to compare what happened in ARC counties with what happened in similar but non-ARC counties, to estimate the counterfactual condition.

Evidence: ARC program evaluations have analyzed costs, benefits, and results, with the following key findings: ' For every $1 invested, the Appalachian Development Highway System (ADHS), returned $1.18 in efficiency benefits, and $1.32 in economic development benefits, (Appalachian Development Highways Economic Impact Studies, July 1998); ' ARC funded infrastructure and public works projects resulted in a benefit cost ratio for direct job creation of 5.4 to 1 and indirect job creation of 8.9 to 1 and personal income rose $9 per public dollar invested. (Evaluation of the Appalachian Regional Commission's Infrastructure and Public Works Program Projects, June 2000); ' Almost three quarters of ARC funded educational projects met or exceeded expectations which included goals for educational attainment, job skills and wages, and family/individual well-being (Evolution of The Appalachian Regional Commission's Educational Projects, March 2001).