Program Code | 10003723 | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
Program Title | National Park Service - Visitor Services | ||||||||||
Department Name | Department of the Interior | ||||||||||
Agency/Bureau Name | National Park Service | ||||||||||
Program Type(s) |
Direct Federal Program |
||||||||||
Assessment Year | 2005 | ||||||||||
Assessment Rating | Moderately Effective | ||||||||||
Assessment Section Scores |
|
||||||||||
Program Funding Level (in millions) |
|
Year Began | Improvement Plan | Status | Comments |
---|---|---|---|
2006 |
Examine cost and benefits of different interpretive techniques, such as waysides, guided walks, and exhibits. |
Action taken, but not completed | The performance standards work group has been established as a part of the I & E Renaissance. The FMSS work group developed products and tested in four parks. Implementation for Asset Code 7500 Interpretive Media will begin service wide for FY 2012. |
2006 |
Determine a process and schedule for an independent program evaluation. |
Action taken, but not completed | Blue Ribbon Evaluation Panel met 10/06 and produced a report. Independent Volunteer Program Evaluation completed 2/07 Seventeen pilot park evaluation studies identified and asked for $875,000 and 1 FTE in FY 2010. |
2006 |
Develop (jointly with the Forest Service) a measure on how parks contribute to societal health by encouraging outdoor recreation. |
No action taken | |
2006 |
Demonstrate how NPS holds superintendents and other managers accountable for achieving program results, such as by incorporating into a manager's personnel evaluation a set of specific performance targets that "cascade" down from program goals. |
Action taken, but not completed | Performance goals have been added to each employee's evaulation. These goals are tied to the supervisor's performance plan. Goals will be used in FY 2009. |
Year Began | Improvement Plan | Status | Comments |
---|---|---|---|
2006 |
Demonstrate how NPS uses performance measures to improve management, adjust priorities, and allocate resources. |
Completed | Performance goals added to each employee evaluation. Tied to the supervisor's performance plan. |
2006 |
Establish baseline and targets for measure on customer satisfaction with value for fee paid. |
Completed | A baseline of 92% was established in FY05. Targets and actuals reported in FY2008 Green book. |
Term | Type | ||||||||||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Long-term | Outcome |
Measure: Visitor Understanding - Percent of visitors that understand and appreciate the significance of the park they are visitingExplanation:Average result of annual visitor survey card (VSC) surveys.
|
|||||||||||||||||||||||||||||||||||||||||||||||||||
Annual | Output |
Measure: Customer satisfaction with value for fee paidExplanation:Baseline and targets to be established FY05
|
|||||||||||||||||||||||||||||||||||||||||||||||||||
Long-term | Outcome |
Measure: Visitor Safety (fatalities) - Number of fatalities annually in national parksExplanation:From 1971-2002, parks had an average of 154 visitor fatalities annually. Goal is to reduce by 3% annually.
|
|||||||||||||||||||||||||||||||||||||||||||||||||||
Annual | Outcome |
Measure: Visitor Satisfaction - Percent of visitors satisfied with appropriate park facilities, services and recreational opportunitiesExplanation:Average result of annual visitor survey card (VSC) surveys.
|
|||||||||||||||||||||||||||||||||||||||||||||||||||
Long-term | Outcome |
Measure: Public Health -- Percent of park visits that involve physical activity.Explanation:Demonstrates how parks can contribute to better societal health by encouraging outdoor recreation. Need to establish baseline and targets.
|
|||||||||||||||||||||||||||||||||||||||||||||||||||
Annual | Output |
Measure: Increased visitation levels at targeted sites.Explanation:Demonstrates increased public awareness of lesser known recreation areas. Need to establish baselines and targets
|
|||||||||||||||||||||||||||||||||||||||||||||||||||
Annual | Efficiency |
Measure: Cost of collection as a percent of total revenue collectedExplanation:Compiled annually for NPS fee management purposes
|
|||||||||||||||||||||||||||||||||||||||||||||||||||
Annual | Efficiency |
Measure: Cost per interpretive visitor contactExplanation:The cost of providing an interpretive contact per per person is adjusted annually for inflation.
|
|||||||||||||||||||||||||||||||||||||||||||||||||||
Annual | Output |
Measure: Facilitated Programs - Attendance at facilitated programs in millionsExplanation:Compiled annually for Servicewide Interpretive Report (SIR) since FY99
|
Section 1 - Program Purpose & Design | |||
---|---|---|---|
Number | Question | Answer | Score |
1.1 |
Is the program purpose clear? Explanation: The organic act of 1916 created the National Park Service with the purpose to conserve park resources and "provide for enjoyment of the same in such manner and by such means as will leave them unimpaired for future generations." The Visitor Services program fulfills this purpose by providing safe recreational opportunities and helping the public understand the meaning and relevance of park resources. Evidence: NPS Organic Act (1916). NPS Management Policies (2001). NPS Director's Orders 6 and 83. Harpers Ferry Center's mission statement. |
YES | 20% |
1.2 |
Does the program address a specific and existing problem, interest, or need? Explanation: Parks have almost 300 million visitors annually. This program provides visitors the information they need to have a safe, enjoyable visit and a better appreciation of park resources. Public health and safety programs have helped to reduce visitor fatalities, even while visitation increases. Visitor services have contributed to consistently high visitor satisfaction levels. Evidence: NPS Organic Act (1916). NPS budget justification (FY06). NPS Director's Order 52A. National Leadership Council's Mission Statement for the National Park Service |
YES | 20% |
1.3 |
Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort? Explanation: No other program has responsibility for serving visitors in national parks. Other Federal agencies may provide similar services, but on different lands with different objectives and responsibilities (e.g., multiple-use management). Some State and local entities may have a similar mission to the NPS, but not for Federal lands, for the Nation as a whole, or for resources deemed nationally significant by law or proclamation. Evidence: Document: "Excellence in NPS Interpretation. NPS Organic Act (1916). NPS Budget justification (FY06). NPS has sole jurisdiction over national parks and sole responsibility for providing services to national park visitors. |
YES | 20% |
1.4 |
Is the program design free of major flaws that would limit the program's effectiveness or efficiency? Explanation: Program is free of major design flaws, as illustrated by consistently high visitor satisfaction ratings. NPS successfully leverages resources through the use of volunteers and seasonal employees. Evidence: Visitor satisfaction surveys. Volunteer in Parks (VIP) reports. Comprehensive Interpretive Plans. |
YES | 20% |
1.5 |
Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries? Explanation: Program is generally designed well, with services provided through a variety of means to park visitors and, via internet, to non-visitors. Health and risk management programs focus on providing as safe a visiting environment as possible. Interpretive and educational programs focus on helping visitors to understand the significance of park resources. NPS can do more to examine the costs and benefits of different interpretive techniques, such as waysides, guided walks, and exhibits. Evidence: NPS visitor surveys show consistently high satisfaction levels and overall understanding of park resources. Website recorded nearly 400 million hits in 2004, up from 112 million in 2001. Visitor fatalities have declined from an average of 154 annually from 1971 to 2002 to 106 in 2004. |
YES | 20% |
Section 1 - Program Purpose & Design | Score | 100% |
Section 2 - Strategic Planning | |||
---|---|---|---|
Number | Question | Answer | Score |
2.1 |
Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program? Explanation: Existing long-term measures focus on three basic outcomes: (1) visitor satisfaction; (2) visitor safety; and (3) visitor understanding for park resources. NPS is developing a fourth measure on how parks contribute to societal health by encouraging outdoor recreation. All support the overarching goal of making the parks fully accessible for the public to enjoy safe and meaningful visits. Evidence: Visitor satisfaction and visitor understanding are measured through annual surveys (see Visitor Survey Card annual summaries) and compiled in various reports (e.g., Servicewide Interpretive Report). All measures are tracked through the Performance Measure Database System (PMDS). |
YES | 12% |
2.2 |
Does the program have ambitious targets and timeframes for its long-term measures? Explanation: NPS has established targets for all three existing long-term measures. Improvement in visitor satisfaction will be difficult, given the high levels currently achieved, but NPS does have targets to increase visitor safety and visitor understanding. A new reporting system in 2005 will capture visitor accident/incident information at 388 park units more accurately and allow NPS to set targets for reducing visitor injury and fatality rates more effectively. Evidence: Visitor Survey Card (VSC) annual summaries. PMDS reports. Servicewide Interpretive Report (SIR). Visitor Services Project (VSR) overview. New reporting system for visitor safety is the Incident Management Analysis and Reporting System (IMARS). |
YES | 12% |
2.3 |
Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals? Explanation: Annual measures directly support the long-term goals listed above. Although NPS can measure outputs (e.g., number of visitors contacted by park staff or participating in facilitated programs), it needs to better demonstrate that it uses these measures to improve management. Visitor satisfaction and visitor understanding measures would be more effective as annual measures (in addition to serving as long-term measures) if they could more responsively demonstrate the link between specific NPS actions and subsequent changes in visitor responses. Evidence: Visitor Survey Card (VSC) annual summaries. PMDS reports. Servicewide Interpretive Report (SIR). Visitor Services Project (VSR) overview. NPS crosswalk to DOI strategic plan measures. (Visitor satisfaction and visitor understanding measures are both annual and long-term measures, but are shown as only one for the purposes of this PART.) |
YES | 12% |
2.4 |
Does the program have baselines and ambitious targets for its annual measures? Explanation: NPS has established targets for all annual measures, except customer satisfaction with value for fee paid. Although NPS can measure outputs, it needs to better demonstrate that it uses these measures to improve management. Visitor satisfaction and visitor understanding measures would be more effective as annual measures (in addition to serving as long-term measures) if they could more responsively demonstrate the link between specific NPS actions and subsequent changes in visitor responses. Evidence: Visitor Survey Card (VSC) annual summaries. PMDS reports. Servicewide Interpretive Report (SIR). Visitor Services Project (VSR) overview. NPS crosswalk to DOI strategic plan measures. (Visitor satisfaction and visitor understanding measures are both annual and long-term measures, but are shown as only one for the purposes of this PART.) |
YES | 12% |
2.5 |
Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program? Explanation: Program has not demonstrated how it links partner activities to program goals through performance requirements, such as performance-based service contracts. NPS has many partners, including more than 140,000 volunteers, 64 cooperating associations, and various universities and associations. Yet, NPS has not shown how partners measure and report on their performance as it relates to accomplishing program goals. Evidence: No sufficient evidence provided. |
NO | 0% |
2.6 |
Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need? Explanation: NPS has not conducted non-biased evaluations on a regular basis to fill in gaps in performance information or improve the effectiveness of the program. Although the program collects survey information, that constitutes its primary performance measure and is not an evaluation of program effectiveness. Park-specific evaluations lack sufficient scope to evaluate the program overall. NPS should contract or authorize an outisde entity to conduct an independent evaluation. Evidence: Visitor Service project conducts park-specific evaluations of visitor services, but no evidence was provided of program evaluations that are of sufficient scope, quality, and independence. |
NO | 0% |
2.7 |
Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget? Explanation: NPS lacks effective budgeting that defines the link between performance targets and budget resources. Although budget justifications cite performance goals, they do not make clear the impact of funding decisions on expected performance. Nor does NPS report all the direct and indirect costs needed to attain projected results. Evidence: NPS 2006 Budget justifications |
NO | 0% |
2.8 |
Has the program taken meaningful steps to correct its strategic planning deficiencies? Explanation: Steps to correct strategic planning deficiencies include: (1) requiring each park to have a current Comprehensive Interpretive Plan with annual and long-term work plans and targets; (2) developing additional measures to track efficiency and promote outdoor recreation; (3) creating a new IT system (IMARS) for incident reporting; and (4) a long-term strategic education plan and the formation of an educational council. Evidence: IMARS overview. Comprehensive Interpretive Planning. Rethinking the National Parks for the 21st century. Renewing our Education Mission. A Comprehensive Study of Visitor Safety in the National Park Service, pp71-72. |
YES | 12% |
Section 2 - Strategic Planning | Score | 62% |
Section 3 - Program Management | |||
---|---|---|---|
Number | Question | Answer | Score |
3.1 |
Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance? Explanation: NPS collects credible performance information, but has not demonstrated how the agency uses it to manage the program and improve performance. NPS has not shown how it systematically uses visitor satisfaction and visitor understanding measures to adjust program priorities, allocate resources or take other appropriate management actions. Nor has it provided examples of steps taken to enact improvements cited by a specific evaluation. NPS is developing systems, such as IMARS, to better track and evaluate progress in visitor safety, but that has not been implemented. Evidence: GPRA Survey results (FY03). Comprehensive Study of Visitor Safety in the NPS (2002). IMARS announcement (2004). NPS FY05 and FY06 Budget justifications. Visitor Service Progam summary report. Servicewide Interpretive Report summary (SIR)(2004). |
NO | 0% |
3.2 |
Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results? Explanation: NPS has not demonstrated that it holds superintendents and other managers responsible for achieving program results. Personnel performance evaluations do not incorporate program performance with clearly defined and quantifiable performance targets for each manager. Some evaluations have workload measures, but they do not "cascade" down from program goals. NPS has not shown how agreements with volunteers or cooperating associations incorporate and directly contribute to program performance goals. Evidence: Illustrative personnel evaluations site program performance goals and list specific workload measures for a given manager, but they do not show how one links to the other. Illustrative cooperative agreements list some requirements, but do not show how they contribute to program performance goals. |
NO | 0% |
3.3 |
Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose? Explanation: All visitor service functions prepare annual financial plans to allocate and track expenditures against intended use. Nearly all funds are available for only one year, so obligations are made in a timely manner. Program uses a project agreement system to outline the products, schedule, budget, and roles and responsibilities involved with projects, such as exhibits built by Harpers Ferry Center. Evidence: Table showing enacted vs. obligated history (FY00-FY04). Servicewide Interpretive Report summary (FY04). Example of Harpers Ferry Center project tracking sheet. |
YES | 14% |
3.4 |
Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution? Explanation: NPS efficiency measures show that the costs per visitor contact and the cost of fee collections are going up (mainly due to higher labor costs), but NPS is using volunteers and other steps to leverage resources and limit the rate of growth. NPS also gathers data annually to track the cost of full and part time NPS personnel in providing interpretive services, and makes appropriate personnel adjustments. NPS has also made IT investments, such as Harpers Ferry Center's project tracking system, to better measure project performance. NPS still needs to do more benchmarks against similar programs and competitive sourcing or other cost comparisons. It must also scrutinize those parks with above-average fee collection costs to find ways to allocate more fee receipts toward projects. Evidence: Volunteer-in-Parks annual activity and expense report. Harpers Ferry Center project tracking information system. Servicewide Interpretive Report (SIR) (2003-2004). See efficiency measures on cost per interpretive visitor contact and cost of fee collection. |
YES | 14% |
3.5 |
Does the program collaborate and coordinate effectively with related programs? Explanation: NPS collaborates frequently with other Federal, State, and local providers of visitor services. Examples that involve meaningful actions include: Recreation One-Stop, an e-gov initiative to improve access to recreation information; a consolidated reservation system for Federal agencies providing campgrounds and other services; Volunteer.gov, a web-based recruitment resource for public agencies; Lewis and Clark Bicentennial celebration; and Blue Ridge Parkway regional destination center. All of these involve joint planning or resource allocations. Evidence: See www.recreation.gov or www.volunteer.gov. Project Agreement for Blue Ridge Parkway regional destination center. Lewis and Clark webpage at www.nps.gov/lecl. |
YES | 14% |
3.6 |
Does the program use strong financial management practices? Explanation: NPS has "several weaknesses" in its financial management systems and has reportable conditions on its financial reporting controls, according to its FY04 financial audit. NPS lacks the ability to integrate financial and performance systems to support day-to-day operations, so these weaknesses have a direct relation to the visitor services program. Evidence: FY04 NPS financial audit. NPS historically has had weak financial management practices, which adversely affects management in many program areas. |
NO | 0% |
3.7 |
Has the program taken meaningful steps to address its management deficiencies? Explanation: The program has had various internal and external reviews, including a 2002 review by private consultants of the Harpers Ferry Center core business and processes and a 2003 GAO report on cooperating associations and other non-profit partners. NPS has under review a draft Director's Order for risk management and a new strategic plan for public health programs. NPS has also created a new certification program for interpreters. Evidence: "Aiming for Excellence" HFC Strategic Plan (2002). "Connecting People to Parks" Interpretation and Education FY05 work plan. GAO report on cooperating agencies and nonprofit partners (GAO-03-585, 2003). Public Health Service strategic plan (2004). Director's Order #50 on public risk management programs (2005 draft). |
YES | 14% |
Section 3 - Program Management | Score | 57% |
Section 4 - Program Results/Accountability | |||
---|---|---|---|
Number | Question | Answer | Score |
4.1 |
Has the program demonstrated adequate progress in achieving its long-term performance goals? Explanation: NPS is on track to meet its long-term goals for visitor services. The current high levels of visitor satisfaction (95%) and visitor understanding (86%) demonstrate that nearly all visitors are pleased with the services being offered and appreciate the significance of the park resources being protected. Visitor fatalities are also below historic averages, which suggests safer visits as well. Evidence: Table on visitor satisfaction levels and visitor survey card reports (1998-2004). Crosswalk on NPS/DOI measures. Table on number of NPS fatalities (1971-2004) |
YES | 25% |
4.2 |
Does the program (including program partners) achieve its annual performance goals? Explanation: NPS met the annual goals for facilitated programs, visitor satisfaction, and visitor understanding, but still needs to develop output measures for increasing visitation levels at targeted sites and customer satisfaction with value for fee paid. NPS also needs to implement its planned IT system (IMARS) for collecting data on visitor fatalities, injuries, and illnesses. It must also establish performance goals for the recently established risk management program. Evidence: Table on visitor satisfaction levels and visitor survey card reports (1998-2004). Crosswalk on NPS/DOI measures. Table on number of NPS fatalities (1971-2004). Interpretation and Education annual measures (2004-2008). IMARS announcement (2004). (Visitor satisfaction and visitor understanding are both annual and long-term measures, but are shown as just one for the purposes of this PART.) |
LARGE EXTENT | 17% |
4.3 |
Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year? Explanation: The cost per visitor contact and the cost of fee collections have gone up, mainly due to higher labor costs, but NPS has taken steps to control the rate of growth, such as increasing by 5% annually the number of volunteer hours donated. NPS estimates (based on independent estimates of the value of donated labor) that contributions from volunteers have increased from $72 million in FY02 to $86 million in FY04. NPS still needs to do more competitive sourcing or other cost comparisons. It must also scrutinize those parks with above-average fee collection costs to find ways to allocate more fee receipts toward projects. Evidence: Facilitated program summary (FY03-04). Volunteer-in-Parks summary (FY02-08). NPS visitation summary. See efficiency measures on cost per interpretive visitor contact and cost of fee collection. |
LARGE EXTENT | 13% |
4.4 |
Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals? Explanation: NPS pioneered the delivery of visitor services by land management agencies. A comparison of visitor satisfaction surveys with other US and Canadian agencies shows that the national park visitor generally is more satisfied with the quality of services, although this could also reflect the relatively higher satisfaction with national park resources. This is difficult to determine, given that no benchmark comparison has been completed by an independent entity. NPS has also developed the Interpretive Development Program (IDP), a course on professional standards for interpretation, that has drawn the interest of other Federal and state agencies, universities, professional and international organizations. Evidence: Visitor satisfaction surveys for the U.S. Fish and Wildlife Service, U.S. Forest Service, British Columbia Provincial Parks and Parks Canada. Interpretive Development Program summary (www.nps.gov/idp/interp). In 2000 the National Association of Interpretation adopted language from the IDP standard as its definition of interpretation. |
YES | 20% |
4.5 |
Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results? Explanation: NPS has not conducted non-biased evaluations on a regular basis to improve the effectiveness of the program. Although the program collects survey information, that constitutes its primary performance measure and is not an evaluation of program effectiveness. Park-specific evaluations lack sufficient scope to evaluate the program overall. A 2004 GPO report did evaluate the relationship between NPS and cooperating agencies, but that is only a small piece of visitor services overall. NPS should contract or authorize an outside entity to conduct an independent evaluation. Evidence: Visitor Service Project conducts park-specific evaluations of visitor services, but no evidence was provided of program evaluations that are of sufficient scope, quality, and independence. GAO report on cooperating agencies and nonprofit partners (GAO-03-585, 2003) |
NO | 0% |
Section 4 - Program Results/Accountability | Score | 75% |