Detailed Information on the
Space Shuttle Assessment

Develop outcome-oriented measures to assess the effectiveness of the transition between the Space Shuttle and exploration programs.

Return the Shuttle safely to flight and continue using it to support the Space Station

Plan to retire the Shuttle by the end of the decade, when its role in assembling the International Space Station is complete.

Develop outcome-oriented short and long-term measures for the Space Shuttle Program

Measure: Latest date of final Shuttle flight, following completion of Shuttle's role in ISS assembly

Explanation:Definition of "completion of Shuttle's role in ISS assembly" is TBD, pending outcome of ongoing studies.

Measure: Maximum number of Type A mishaps through Shuttle retirement

Explanation:Type A mishap: damage to property at least $1M or death

Measure: On-orbit mission success

Explanation:Mission success criteria are those provided to the prime contractor (space flight operations contract) for purposes of determining successful accomplishment of the performance incentive fees in the contract.

Measure: Maximum number of type A or B mishaps

Explanation:Type A mishap: damage to property at least $1M or death; Type B mishap: damage to property at least $250K or permanent disability or hospitalization of 3 or more people

Measure: Year-to-year reduction in Space Shuttle annual value of Shuttle production contracts for Orbiter, External Tank, Solid Rocket Boosters, Reusable Solid Rocket Motor, Space Shuttle Main Engine and Launch & Landing, while maintaining safe flight. (New measure, February 2007)

Explanation:Metric reflects the phasing out of resources required for production after we have completed building and purchasing necessary flight hardware.

Measure: The number of dedicated Space Shuttle Kennedy Space Center facilities, while maintaining safe flight. (New measure, February 2007)

Explanation:These facility numbers actually represent blocks of facilities. The one facility closure in FY07 is Pad-B which is 43 facilities. The 22 facilities listed above are in fact 342 facilities if you count individual buildings, antenna farms, etc.

Measure: Deliver scheduled operating hours for all Shuttle missions

Explanation:Operating hours are defined as the time when the Space Shuttle Orbiter reaches orbit and is able to start operations until it begins to stow and prepare for landing. At least 90% of this usable time must be achieved to meet the metric.

Measure: Year-to-year reduction in Space Shuttle sustaining engineering workforce for flight hardware and software, while maintaining safe flight.

Explanation:The sustaining engineering workforce supports the program's ability to respond to and handle anomalies in a timely manner. This workforce will transition in support of Constellation System.

Is the program purpose clear?

Explanation: The primary mission of the Space Shuttle Program is to transport people, materials, and equipment to the International Space Station (ISS) and to support ISS assembly until Space Shuttle retirement.

Evidence: The Space Shuttle transports people, materials, and equipment to the International Space Station (ISS). See NASA's Implementation Plan for Space Shuttle Return to Flight and Beyond at www.nasa.gov

Does the program address a specific and existing problem, interest, or need?

Explanation: National Space Policy directs NASA to complete assembly of the International Space Staton, including the U.S. components that support U.S. space exploration goals and those provided by foreign partners, planned for the end of the decade. The Space Shuttle provides the only current U.S. capability to support assembly of the ISS.

Evidence: A Renewed Spirit of Discovery: The President's Vision for U.S. Space Exploration (www.whitehouse.gov/space/renewed_spirit.html).

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: The Space Shuttle provides the only current capability for assembly of the International Space Station (ISS). NASA is working to develop a Crew Exploration Vehicle that will be capable of ferrying astronauts to the ISS, and plans to acquire cargo transportation as soon as practical and affordable to support mission to and from the ISS, but neither of these activities will have the capability or timeliness to replace the Shuttle's role in ISS assembly. The Shuttle will be retired when its unique role in ISS assembly is complete.

Evidence: A Renewed Spirit of Discovery: the President's Vision for Space Exploration(www.whitehouse.gov/space/renewed_spirit.html) describes the roles of the Space Shuttle, Crew Exploration Vehicle, and acquired cargo transportation.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: As a result of the Columbia accident, the Space Shuttle system has undergone major modifications to address the Columbia Accident Investigation Board Return to Flight recommendations as well as several internal actions intended to further remove risk to safe flight. The successful Shuttle Design Certification Review was held on TBD, 2005. Discover was launched on TBD, 2005 and certified free of major design flaws.

Evidence: "NASA's Implementation Plan for Space Shuttle Return to Flight and Beyond" Volumne 1 10th Edition, is periodically updated to show our status on safely returning to flight (www.nasa.gov/returntoflight/news/.

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: Following return to flight, the Space Shuttle will primarily be used to complete assembly and logistics flights to and from the ISS. The beneficiaries of this effort will be the customers and communities supported by ISS research activities. In addition, through educational outreach & technology transfer the Shuttle Program is able to reach other beneficiaries. NASA has canceled planned Space Shuttle upgrades that did not efficiently support the program's revised goals following the announcement of the President's vision for space exploration.

Evidence: Annual Performance & Accountability Report FY2004 (ifmp.nasa.gov/codeb/docs/NASA_FY2004_PAR.pdf); NASA Fiscal Year 2006 Budget Estimates (www.nasa.gov/about/budget/)

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The Shuttle program has two long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program

Evidence: Measures attachment to the PART.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The program has challenging but realistic quantifiable targets and timeframes for its two long-term measures. Completing the Shuttle's role in space station assembly and retiring the Shuttle by 2010 and doing so without any Type A (major) mishaps is challenging, realistic, and quantifiable.

Evidence: Measures attachment for the PART.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The program has five annual measures that generally focus on outcomes and adequately measure the program's progress towards reaching their long-term goals. The program's efficiency measure is still under development, pending conclusion of an ongoing study.

Evidence: Measures attachment to the PART

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The program has baselines and ambitious targets for some of its annual measures. For example, the goals of 100% mission success and zero major mishaps are ambitious. The program, however, does not have baselines and ambitious targets for all of its annual measures. The goal for the average number of in-flight anomalies is not ambitious, based on past experience. In addition the targets for the program's efficiency measure are not yet available. [This answer may change after the new measure is finalized.]

Evidence: Measures section of the PART.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: Space Shuttle contractors have been an important part of the Shuttle upgrades work in the late 1990s and more recently, have been integral to NASA's Return to Flight and Space Shuttle transistion planning efforts. The performance of the Space Shuttle contractors in contributing to successful mission execution is closely monitored in accordance with the provisions in Space Shuttle agreements, including the Space Flight Operations Contract (SFOC). In addition to the normal close working relationships at the mission execution level, all government, industry, and academic partners are vested in the ISOS process which provides a forum for addressing the Space Shuttle program's investment strategy and strategic planning.

Evidence: The Shuttle monthly performance data is located in the Management Information System (MIS) http//nasa-mis.nasa.gov; Integrated Space Operations Summit 2005 CD http//onemis.nasa.gov/ISOS

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: Numerous independent reviews are conducted on the Space Shuttle program to help ensure flight safety, assess programmatics, and evaluate performance.

Evidence: The Shuttle program has been reviewed by the Aerospace Safety Advisory Panel, GAO Audits, & Non-Advocacy reviews. Most recently the program has benefited from the recommendations of the Columbia Accident Investigation Board, and the Stafford Covey Return to Flight Task Group.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: NASA's Integrated Budget and Performance Document provides, for every budget line item, an associated performance measure and strategic objective. The Shuttle program has a ISOS process that identifies, evaluates and prioritizes investments that will improve safety of flight. Additional measures to show risk reductions for safety upgrades are included.

Evidence: FY06 IBPD is located at www.nasa.gov/pdf/55412main_29%20SSP.pdf ISOS Summit III CD 2005 onemis.naa.gov/ISOS

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: With the release of the Vision for Exploration, the Shuttle program has clear strategic goals: return to flight, complete assembly of the International Space Station, then phase out the Space Shuttle fleet by 2010. In accordance with the recommendations of the Columbia Accident Investigation Board (CAIB), NASA has made significant improvements to the Shuttle planning and managmeent in order to improve safety and mission success. The top-down policy guidance from the Vision and the CAIB-recommended organizational and management changes are fed into the ISOS process, which taps expertise throughout government, industry, and academia to prioritize the Shuttle planning and investments for successful mission execution as well as transistion and phaseout.

Evidence: Vision for Space Exploration, NASA's Implementation Plan for Space Shuttle Return to Flgiht and Beyond, www/nasa.gov/returntoflight/news; ISOS Summit 2005 CD, http://one-mis.nasa.gov/ISOS

Has the agency/program conducted a recent, meaningful, credible analysis of alternatives that includes trade-offs between cost, schedule, risk, and performance goals, and used the results to guide the resulting activity?

Explanation: Through ISOS panels & the Space Operations Leadership Council, NASA had the appropriate forum to weigh and evaluate the investment direction for phase-out of the Shuttle program. Based on this evaluation, the Space Shuttle program cancelled the Cockpit Avionics Upgrade, the Space Shuttle Main Engine Advanced Health Management System Phase 2, and the Vehicle Health Monitoring System. The Agency is currently conducting two studies that will impact the current Shuttle program. After the release of the study results, the Shuttle program will be reevaluated for tradeoffs between cost, schedule, etc. (Update to be provided after the Agency studies are completed)

Evidence: ISOS Summit March 2005 CD, http//onemis.nasa.gov/ISOS Shuttle/Station Configuration Options Study - Ongoing

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: During the efforts to return the Space Shuttle to flight, the Space Shuttle Program did not initially have sufficient information due to technical immaturity and exploring multiple path solutions from key program partners to be able to estimate the true cost of returning the Shuttle to flight. Estimates evolved with technical plan maturity, which led to the need to transfer funds from other NASA programs. NASA is taking significant steps to improve the gathering of this information, including more rigorous use of earned value management systems.

Evidence: Vision for Space Exploration, NASA's Implementation Plan for Space Shuttle Return and Flight and Beyond, www.nasa.gov/returntoflight/news

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: The Shuttle Program has various incentives on contracts tied to cost, schedule and performance. Each contract has surveillance plans that each technical manager uses to monitor performance. Federal managers' performance plans have key factors that deal with cost/schedule/performance and SES bonuses are tied to performance metrics.

Evidence: A Requirements and Procedures for Certification of Flight Readiness document is in place that establishes responsibility for key government & contractor managers who sign & are accountable for each & every Shuttle launch. Schedule & costs are controlled through the Program Requirements Control Board. The Space Shuttle Program will receive flight rate credit from United Space Alliance (USA) for the flights not flown. The flight rate credit from not flying was used to partially offset the Columbia investigation and recovery effort.

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: The Space Shuttle program tracks all funds in the Agency Integrated Financial Mangement System. Annual NASA R&D funds are available for obligation for two years, and fully obligated by the end of the period. Operating Plans for the program year are submitted to Congress and revised as needed over the two year time period.

Evidence: The Space Shuttle Program (SSP) utilizes the official Agency accounting system and complies with all financial management rules and regulations. Major contractors are required to submit monthly and quarterly (533) reports which include monthly and forcasted quarterly accurals. SSP holds monthly and quarterly program status reviews to assess project cost and schedules. The Defense Contracting Audit Agency (DCAA) performs annual incurred cost audits on major shuttle contracts to ensure that all costs are allowable, accountable, and reasonable.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: A Yes answer requires the program have at least one efficiency measure with baseline and targets. The program does not have an efficiency measure with baseline and targets. (This answer may change after the new efficiency measure targets are finalized) The program does implement IT improvements and has contract incentives that encourage and reward the contractor for safe, high quality, cost effective performance in fulfilling the contract requirements in alignment with Shuttle program goals. Contract performance/award fee evaluation processes are in place to encourage and reward the contractor for safe, high quality, cost effective performance in fulfilling the contract requirements in alignment with the Shuttle Program goals.

Evidence: IT improvement was the implementation of the NMIS web interface (located at http://nasa-mis.nasa.gov/nasa_mis) that provided management adequate status of key performance indicators.

Does the program collaborate and coordinate effectively with related programs?

Explanation: In the Integrated Space Operations Summit, NASA brought in various stakeholders in the Shuttle program to determine appropriate future investments. The Space Shuttle program collaborates with several federal agencies and is heavily participating with the Office of Space Exploration to support the NASA Vision. The Space Shuttle program remains closely intertwined with the International Space Station program.

Evidence: Space Shuttle program and Department of Defense collaborate in scheduling of payload manifesting and the use of range facilities at Cape Canaveral. Also, Space Shuttle program has several agencies on the Mishap Investigation Board such as DOD and DOT. NASA has worked with many federal and state agencies in the Columbia Recovery effort (Police Departments, Forest Service, Coast Guard, EPA, Urban Search & Rescue, NIMA, NTSB, FEMA, National Guard, Park Service, NWS, NRL, et. al.) NASA has an agreement with National Geospatial-Intelligence Agency (NGA) for on-orbit vehicle assessments.

Does the program use strong financial management practices?

Explanation: The most recent Independent Auditor report for NASA identified four material weaknesses, all of which are repeats, as well as noncompliance with the Federal Financial Management Improvement Act.

Evidence: NASA's FY 2004 Performance and Accountability Report (available at www.nasa.gov/about/budget/index.html) includes the communication from the NASA Inspector General and the report of the Independent Auditor. In addition, the GAO has published numerous reports identifying shortcoming in NASA's new financial management system as well as its financial management processes (example is GAO-04-754T released on May 19, 2004).

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The Space Shuttle program continues to address and take action to correct identified deficiencies. All recommendations of the Columbia Accident Investigation Board (CAIB) are being responded to, including those pertaining to management deficiencies. In addition, the program conducted its own analyses and identified additional management and technical deficiencies that the Agency is correcting.

Evidence: Actions to correct identified deficiencies in the Space Shuttle program following the Columbia accident are detailed in NASA's Implementation Plan for Space Shuttle Return to Flight and Beyond (www.nasa.gov/pdf/116880main_rtf_imp_10th.pdf) In addition, the Shuttle program is reviewed in several fora that are intended to identify and correct deficiencies in program management. These include: - The Agency Program Management Council, which meets quarterly to discuss both technical and management issues associated with programs and projects. - The Space Shuttle Council, led by the Shuttle Program Manager, which meets quarterly to discuss program and management issues. - The Aerospace Safety Advisory Panel, which provides ongoing reviews, evaluations, and analyses on elements of NASA's safety and quality systems, including those that support the Space Shuttle program.

Is the program managed by maintaining clearly defined deliverables, capability/performance characteristics, and appropriate, credible cost and schedule goals?

Explanation: Following the loss of Space Shuttle Columbia, the Space Shuttle program was initially unable to set appropriate credible cost and schedule goals. The targeted date for the Space Shuttle return to flight has slipped more than a year from the original target date. Until late Spring of 2005, the planned manifest for the Space Shuttle program suggested that the Shuttle would fly 28 times before 2010, although experience suggests that this timeframe was unrealistic. The Shuttle's projected budget for the remainder of its service life was also not well defined and contained estimated savings that were not clearly justified. Over the past year, the estimated costs for return to flight appear to have stabilized, and NASA has recently been taking steps to develop a realistic future schedule and budget for the Shuttle.

Evidence: Vision for Space Exploration, NASA's Implementation Plan for Space Shuttle Return to Flight and Beyond, www.nasa.gov/returntoflight/news

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: By responding to the recommendations of the Columbia Accident Investigation Board, the Space Shuttle program demonstrated progress for returning to flight to meet the long term goals of completing the Space Station and retiring by 2010 without additional major mishaps.

Evidence: NASA's Implementation Plan for Space Shuttle Return to Flight and Beyond at www.nasa.gov

Does the program (including program partners) achieve its annual performance goals?

Explanation: During the performance period, the Space Shuttle program was grounded, pending a successful return to flight. Without the Shuttle flying, the program is unable to meet some of its annual performance goals. The Space Shuttle program meets its annual measures to a small extent, in that the program has operated without serious mishaps since the Columbia accident.

Evidence: Measures tab of the PART.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: Since the Columbia accident, the Space Shuttle program has emphasized reducing risk, rather than striving for cost-efficiency. The program is interested in achieving efficiencies, but wants to ensure that the efficiencies are not achieved at the cost of increased risk. NASA anticipates that the program will be able to achieve some cost efficiencies while maintaining safety as the Shuttle approaches retirement and spending for certain functions, such as production of hardware, can be reduced.

Evidence:  

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: The Space Shuttle is the only U.S. space vehicle currently capable of providing heavy lift assembly support for the International Space Station.

Evidence:  

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: The Shuttle program has undergone several independent & quality evaluations that show the program has achieved some positive results this year.

Evidence: Aerospace Safety Advisory Panel, GAO Audits, & Non-Advocacy reviews. Most recently we've benefited from the recommendations of the Columbia Accident Investigation Board, and the Stafford Covey Return to Flight Task Group. The Stafford Covey Return to Flight Task Group reported that the program had successfully responded to most of the recommendations of the Columbia Accident Investigation Board.

Were program goals achieved within budgeted costs and established schedules?

Explanation: The cost and time required to return the Shuttle safely to flight were both underestimated. Technical requirements for returning the Shuttle to flight have matured considerably this fiscal year and costs for returning the Shuttle to flight have remained stable since November 2004. The program has made major strides towards returning the Shuttle to flight, but has not been able to do so within the initial predicted cost and schedule.

Evidence: Vision for Space Exploration, NASA's Implemention Plan for Space Shuttle Return to Flight and Beyond, www.nasa.gov/returntoflight/news