Detailed Information on the
Solar System Exploration Assessment

Report for major missions on: estimated mission life cycle cost upon entering development; key schedule milestones associated with each mission phase for those missions formally approved for formulation; mission cost and schedule progress achieved in each phase before entering the next; and any plans to re-baseline life-cycle cost and schedule.

NASA will define its requirements, approach, projected schedule, and budget profile (with proposed offsets) for Deep Space Network upgrades, in time for the FY 2009 budget submit to OMB.

NASA will explore options for modifying the current approach to its competed planetary science programs to allow for a healthy mix of missions of various size and scope, potentially including missions to the outer planets.

Measure: Progress, as evaluated by external expert review, in learning how the Sun's family of planets and minor bodies originated and evolved.

Explanation:

Measure: Progress, as evaluated by external expert review, in understanding the processes that determine the history and future of habitability in the solar system, including the origin and evolution of Earth's biosphere and the character and extent of prebiotic chemistry on Mars and other worlds.

Explanation:

Measure: Progress, as evaluated by external expert review, in identifying and investigating past or present habitable environments on Mars and other worlds, and determining if there is or ever has been life elsewhere in the solar system.

Explanation:

Measure: Progress, as evaluated by external expert review, in exploring the space environment to discover potential hazards to humans and to search for resources that would enable human presence.

Explanation:

Measure: Accomplishment of key development and technology activities. FY 2006: Successfully return Stardust science samples to Earth; successfully launch Dawn spacecraft; complete successful Martian orbit insertion for MRO; complete Mars Science Laboratory Preliminary Design Review. FY 2007: Perform MESSENGER mission second Venus flyby; complete Juno Preliminary Design Review (PDR); complete 2009 Mars Science Laboratory (MSL) Critical Design Review (CDR); successfully launch Phoenix 2007 spacecraft; Begin Mars Reconnaissance Orbiter (MRO) primary science phase. FY 2008: Complete the Mercury Surface, Space Environment, Geochemisry and Ranging (MESSENGER) Mercury Flyby 1, Begin Juno instruments detailed design, Begin 2009 Mars Science Laboratory (MSL) Assembly, Test, Launch Operations (ATLO), and Land the Phoenix spacecraft on the Martian surface and begin science operations. FY 2009: Complete the Juno Critical Design Review (CDR), Complete the Discovery mission Preliminary Design Review (PDR), Complete the Mars Science Laboratory (MSL) Launch Readiness Review (LRR), and Select the next Scout mission.

Explanation:

Measure: Cumulative percentage of baseline cost overrun for projects under development.

Explanation:On average, projects in development will not exceed their baseline cost by more than 10% cumulatively.

Measure: Cumulative percentage of baseline schedule slip for projects under development.

Explanation:On average, projects in development will not exceed their baseline schedule by more than 10% cumulatively.

Measure: Percentage of scheduled operating hours delivered for all operating spacecraft.

Explanation:The term operations and research facilities is described as the ability to meet all Level-1 requirements, and deliver (discover, track, characterize, transmit, receive and archive) science data to the user community.

Measure: Percentage of budget for research projects allocated through open, peer-reviewed competition.

Explanation:This target only applies to the Planetary Science Division budget line that is referred to as "research and analysis."

Measure: Number of days within which NASA Research Announcement research grants for the program are awarded, from proposal due date to selection, with the ultimate goal of 130 days.

Explanation:

Is the program purpose clear?

Explanation: The program has a discretely defined purpose that relates directly to the NASA vision and mission statements. Its goals and objectives are linked to specific elements of the Science Mission Directorate, the NASA Strategic Plans, and the President's Vision for Space Exploration.

Evidence: The Vision for Space Exploration instructs NASA to "Conduct robotic exploration across the solar system for scientific purposes and to support human exploration. In particular, explore the moons of Jupiter, asteroids, and other bodies to search for evidence of life, to understand the history of the solar system, and to search for resources." Accordingly, the 2006 NASA Strategic Plan establishes a goal to "advance scientific knowledge of the solar system, search for evidence of life, and prepare for human exploration." The purpose of the program is to conduct robotic exploration in support of this goal. The solar system exploration strategy further defines the purpose of the program by detailing five scientific objectives: (1)Learn how the Sun's family of planets and minor bodies originated and evolved; (2) Understand the processes that determine the history and future of habitability in the solar system, including the origin and evolution of Earth's biosphere and the character and extent of prebiotic chemistry on Mars and other worlds; (3) Identify and investigate past or present habitable environments on Mars and other worlds, and determine if there is or ever has been life elsewhere in the solar system; (4) Explore the space environment to discover potential hazards to humans and search for resources that would enable human presence. The four objectives can be understood as addressing, in different ways, the fundamental goal of understanding how our solar system became, and planetary systems in general become, habitable and how habitability is sustained.

Does the program address a specific and existing problem, interest, or need?

Explanation: The program allows us to explore the solar system in search of understanding "how our solar system became, and planetary systems in general become habitable, and how habitability is sustained." This topic is not only of interest to the scientific community but also to the general public (and has been for centuries).

Evidence: That the program addresses an existing interest is documented in the National Research Council's "New Frontiers in the Solar System Survey: An Integrated Exploration Strategy." Further, in "Review of Goals and Plans for NASA's Space and Earth Sciences" (currently in pre-publication form only), the National Academy of Sciences "agrees with the breakdown of goals listed in the roadmap and notes that they are consistent with, and support, the goals outlined in the NRC decadal survey." The Vision for Space Exploration observes: "Today, humanity has the potential to seek answers to the most fundamental questions posed about the existence of life beyond Earth."

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: No other US agency launches scientific missions to other planets. While the National Science Foundation supports some astronomical studies of the solar system, its work is conducted from ground-based assets. Some universities also conduct limited studies of the solar system, including studies funded in part or in total by NASA. There are no other efforts by any federal, state, local or private entity in the U.S. of the magnitude and scope of NASA's solar system exploration program. It is a unique, one-of-a-kind program that seeks to achieve both near and long-term science goals by studying solar system objects and phenomena in situ.

Evidence: The program utilizes multiple space missions to collect a broad spectrum of scientific data. THe program also pursues and develops both enabling and enhancing technologies to provide new capabilities to collect data and achieve unique scientific advances. No other program directed at planetary science supports such a broad range of published goals and objectives. There is no redundancy between the National Science Foundation's (NSF) astronomical, planetary and other space science objectives. NSF science objectives focus on ground-based planetary astronomy and are curiosity based, not mission based, while NASA's solar system exploration program science objectives are generally pursued via space-based investigations, and directed toward mission science objectives derived from our FACA advisory committees and stated in Program Commitment Agreements and the Integrated and Budget Performance Document.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The scientific design of the program and its ability to effectively and efficiently achieve its goals has been optimized by considering and incorporating the advice and counsel of a broad community of experts who have been intimately involved for a number of years. These experts are from NASA and other federal agencies, universities, industry and our international partners. Their input is codified in a roadmap, which outlines the program's missions and objectives as endorsed by the National Academy of Sciences, NASA advisory committees, and the Solar System Exploration Subcommittee (now succeeded by the Planetary Science Subcommittee) under the NASA Advisory Council.

Evidence: The Planetary Science Roadmap (http://science.hq.nasa.gov/strategy/comm.html#roadmaps) lays out direction for ten years to ensure the program's optimal design. The science community advises NASA on the appropriateness of this roadmap to ensure the Program adequately addresses the goals of the scientific community, and that it will enable NASA to effectively achieve these goals.

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: The rigor with which the program is designed, structured, managed and funded ensures that resources will reach only the intended beneficiaries and will address the program's purpose directly. The five science objectives outlined in the Planetary Science Roadmap guide the activities of the program and provide the context through which specific research objectives are formulated, science investigations are defined, and missions that address them are planned. Only the activities included in the roadmap are funded, thereby ensuring that all resources are spent to achieve the Planetary Science goals.

Evidence: The scientific purpose of each mission is well documented (see the Integrated Budget and Performance Document and the Strategic Plan, and Planetary Science roadmap (http://science.hq.nasa.gov/strategy/comm.html#roadmaps)) and is linked to specific goals and objectives. Funds are issued to the appropriate entity at the mission level or below. Above a certain level, Federal law prohibits the redirection of resources issued for one program to another program without Congressional approval.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The program's long-term measures focus on outcomes and meaningfully reflect the program's purpose.

Evidence: The program has four long-term performance measures that direct progress towards the overall purpose of the Planetary Science theme, which is to advance scientific knowledge of the solar system, to search for evidence of life, and to prepare for human exploration.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The program has ambitious targets and timeframes for its long-term measures.

Evidence: The timeframe for the long-term measures, as outlined in the Planetary Science Roadmap (http://science.hq.nasa.gov/strategy/comm.html#roadmaps), the Mars Exploration Architecture (currently in draft form) and the 2007 Integrated Budget and Performance Document is decadal. Progress against these goals is measured yearly.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The program has specific annual performance measures that demonstrate progress toward achieving the program's long-term goals.

Evidence: The program's annual performance measures support and indicate progress toward addressing its long-term measures. The program long-term measures are supported by annual measures that serve as indicators of effective program management.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The program has baselines and ambitious targets for its annual measures.

Evidence: The program management annual measures have ambitious targets regarding cost and schedule growth, availability of operating spacecraft, percentage of funds awarded competitively, and the time taken to process grant awards.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: Partners (NASA Centers, JPL, contractors and subcontractors, universities, international organizations and other Federal agencies) are directly involved in planning and establishing the program's goals and objectives. Consequently, they fully support and are committed to the achievement of both the annual and the long-term goals of the program. Both regularly scheduled and ad hoc reviews provide management insight into whether Partners are adhering to and supporting the program's goals and objectives.

Evidence: Program goals are made clear to partners. Program missions must document their goals, objectives, and technical deliverables (as they relate to Theme goals) in program plans and commitments, signed agreements between NASA HQ, Program Offices, and the lead NASA center. These documents are available to all partners. The program uses instruments available to government agencies to enter agreements with other entities to obtain commitments to working toward and reporting on progress in achieving the annual and/or long-term goals of the program. Letters of Agreement, Memoranda of Understanding, and Technical Assistance Agreements have been signed with major international partners and other Federal agencies. Contracts and grants with industry and universities have been signed and task-level agreements between the Planetary Science division and the Jet Propulsion Laboratory and other NASA centers have also been reached. The program conducts award fee reviews, mid-year performance reviews and ad hoc reviews to determine and verify partner commitment. Independent contract and programmatic reviews are conducted routinely.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: The program effectiveness and program relevance are subjected to regular reviews and evaluations. Annually, an external expert review panel reviews program performance as part of the development of the GPRA-mandated Agency performance report. The Congress has also mandated that the National Academy of Sciences review the performance of each division of NASA's Science Mission Directorate every five years; the first such review for the Planetary Science Division is anticipated to occur in 2007-08. The Planetary Science Subcommittee of the NASA Advisory Council's Science Committee provides inputs each year on community opinions and recommendations. At the mission level, Non-advocate reviews and independent cost reviews are conducted to qualify a project to move from formulation to implementation. For operating missions, external peer-review panels weigh the scientific merits of continued operation of missions that have fulfilled their designed lifetimes. In addition, every three years, a broad community of experts from NASA, other federal agencies, universities, industry and international partners evaluates the program and offers strategic advice and counsel that leads to a revision of the NASA Science Strategic Plan.

Evidence: The National Academy of Sciences reviewed the program as part of its Decadal Survey to help NASA prioritize missions and science objectives for the next ten years. The Planetary Science Roadmap was created to lay out an effective path to achieve the vision set forth in the Decadal Survey. Independent external reviews by the NASA Advisory Council are conducted annually to evaluate progress toward meeting scientific outcomes. In addition, minutes and findings of NASA Advisory Council Science Subcommittees are posted at: http://science.hq.nasa.gov/strategy/subcomm.html. In addition, the NASA Advisory Council, its Science Committee and the Planetary Science Subcommittee meet three to four times per year to conduct reviews of science and program implementation strategies. Independent program assessment reviews are conducted at critical junctures for all missions that are in development or in space operations. Examples include mission confirmation review, critical design reviews and launch readiness reviews.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: In order to be approved for funding, each mission must demonstrate how it supports the program's long term performance measures, which are assessed and validated by external reviews. Budget requests for each mission are dependent upon the successful completion of the current year's planned activities and anticipated future requirements. The life-cycle cost requirement for each mission (in full cost) is included in the budget request (Integrated Budget and Performance Plan) in a complete and transparent manner.

Evidence: The Integrated Budget and Performance Document displays important status data for each mission, lists the budget requirements for life cycle cost, and identifies the specific long-term outcomes and annual performance goals supported by that mission. Performance against each annual measure is reported to Congress each year in the Performance and Accountability Report and budget overruns are reported as required by Congress per the NASA Authorization Act of 2005.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The program regularly reviews its strategic planning and utilizes a number of different mechanisms to determine and correct any deficiencies.

Evidence: Experts review program progress, leading to revision every three years of the Science Mission Directorate Strategic Plan, which the National Academy of Sciences then reviews. Planetary Science strategies, missions, and objectives are also reviewed by the Science Committee of the NASA Advisory Council and its Planetary Science Subcommittee. Changes in strategic planning are incorporated into the SSE Roadmap and the NASA Science Strategic Plan.

Has the agency/program conducted a recent, meaningful, credible analysis of alternatives that includes trade-offs between cost, schedule, risk, and performance goals, and used the results to guide the resulting activity?

Explanation: There are no alternatives to robotic visits to planetary bodies to obtain the data, imagery, and samples required for scientific research. However, NASA asked the National Research Council to provide an inventory of top-level scientific questions and a prioritized list of flight and supporting ground-based investigations to address those questions. The resulting decadal survey for solar system exploration was delivered to NASA in 2003. Since then, NASA has commissioned the development of science community-based roadmaps for solar system and for Mars exploration. These are being used to develop the Planetary Science portion of the NASA Science Strategic Plan. When a program/project is formulated, its concepts, technology requirements, operations concepts, internal management controls, budget and institutional requirements are evaluated by independent bodies. During this period of formulation, design trade studies are conducted in order to reconcile trade-offs between competing performance factors. Programs/projects are subjected to independent reviews throughout their life-cycle to evaluate their ability to meet commitments. Included in these reviews are recommendations for proceeding with, modifying or terminating the program or project, or for enhancing overall technical and programmatic performance.

Evidence: The National Research Council Decadal Survey was delivered in 2003 and is available at www.nap.edu . The community roadmaps are available at http://science.hq.nasa.gov/strategy/comm.html#roadmaps. Summaries of program decisions taken, presented to subordinate bodies of the NASA Advisory Council, are available at http://science.hq.nasa.gov/strategy/subcomm.html. An example of a trade-off made is the Dawn mission restart. The Dawn termination and reinstatement reviews were recently held, where the mission scientific merit, technical issues and cost were independently reviewed and assessed. Furthermore, the Science Mission Directorate held a review on the Phoenix mission, in which it was determined that Entry Descent and Landing technology will continue to be a challenge. In order to ensure mission success, additional testing was approved.

If applicable, does the program assess and compare the potential benefits of efforts within the program and (if relevant) to other efforts in other programs that have similar goals?

Explanation: Although some other entities conduct astronomical studies of the solar system, no other entity does so by studying the solar system from space-based assets. In this regard, the program is unique and thus such a comparison cannot be made. However, NASA does work closely with other programs which study the solar system to make sure that effective use is made of all available data. NASA is collaborating with ESA in the Cassini Huygens mission, the Mars Express mission, and the Venus Express mission. These collaborations are closely coordinated to be supportive and synergistic.

Evidence:

Does the program use a prioritization process to guide budget requests and funding decisions?

Explanation: Independent outside organizations review the program and help set scientific priorities in line with Agency goals and objectives. Missions are then chosen to accomplish these scientific priorities. The Planetary Science roadmap (currently in work) presents these missions in a time-ordered sequence subject to future science and technology developments. The NASA Science Plan will provide the current set of prioritized missions through 2016. Budget requests consider the relative importance and placement of missions and research based on this strategic plan. During the budget formulation process, missions provide life cycle cost data and must show relevance to annual and long term goals. The NASA new budget process, Planning, Program, Budget and Execution will ensure that prioritizations and strategic goals tie directly to the funding decision.

Evidence: The National Research Council Decadal Survey was delivered in 2003 and is available at www.nap.edu . The community roadmaps are available at http://science.hq.nasa.gov/strategy/comm.html#roadmaps. Summaries of program decisions taken, presented to subordinate bodies of the NASA Advisory Council, are available at http://science.hq.nasa.gov/strategy/subcomm.html. Additionally, NASA is implementing Planning, Program, Budget, and Execution, a budget process that NASA will use to implement and guide the FY2008 budget decision. This process allows NASA to focus additional attention on the translation of strategy into actionable programs, allows for trade-offs, and ensures that NASA, the Science Mission Directorate, and Planetary Science priorities and strategic outcomes are aligned.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The program collects relevant technical and programmatic performance data on a monthly basis. This information is used to assess monthly progress and annual progress toward meeting long-range outcomes, and can be used to develop risk mitigation strategies, adjust priorities, or make additional resource allocation or take other appropriate management actions.

Evidence: The Science Mission Directorate reviews performance data monthly. Programs over a certain dollar value must exercise a contractor-owned, Agency-approved earned value system and are reported on a monthly basis at all level within NASA (from the center to Mission Directorate levels). Recently, the Science Mission Directorate reviewed risk mitigation and cost reduction strategies to determine whether and where to make strategic changes. It was decided to add additional funding to the Mars Phoenix mission to retire technical risk associated with Entry Descent and Landing. By allowing long-pole technology to mature before incorporating it into a project, risk is reduced and cost growth avoided. Monthly performance data have been collected on the DAWN mission since its confirmation began to show severe cost and schedule erosion beginning in early 2005. By the Fall of 2005, Science Mission Directorate management perceived, based on these data and consequent interactions with the Jet Propulsion Laboratory, that the situation was uncontained. The Science Mission Directorate directed a stand-down of mission development and a thorough analysis of technical, cost and management aspects of the program. As a result, the decision was made by the Science Mission Directorate Associate Administrator in March 2006 to terminate the project. A subsequent appeal to the NASA Administrator reversed this decision. All along the way, timely and credible performance information has served as the basis for assessment and decision-making.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: Federal managers and program partners are held accountable for cost, schedule and performance results through a series of formal procedures and requirements. Federal managers who fail to demonstrate the required level of performance are subject to a variety of disciplinary actions, including reassignment or termination. Partners who likewise fail to demonstrate the required level of performance may find their level of participation in the program either diminished or terminated.

Evidence: Every manager is required to develop a formal personal performance plan with his or her supervisor. This plan consists entirely of critical elements, at least one of which must be linked to the Agency's Strategic Plan or the organization's operating plan or goals. Although the program's performance may be evaluated on a more frequent basis, the program manager's performance is formally evaluated twice yearly. Bonuses and promotions are dependent upon the manager making positive progress toward meeting the goals of the program. Should he or she fail to do so, corrective actions ranging from counseling, reassignment or, in extreme cases, termination may result. Partners who fail to perform as required may likewise find their participation reduced or terminated. Implementing Centers are held accountable for the successful implementation of their programs. Contractors and subcontractors are held accountable for the timely delivery and quality of products. NASA uses award fees to incentivize performance where appropriate. Partners who fail to perform as required may find their participation reduced or terminated.

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: Annual NASA funds are available for obligation for two years and are fully obligated by the end of that period. NASA's new Program/Project work breakdown structure has been implemented in FY06 and allows more visibility into the expenditure of funds, allowing NASA to verify that funds are being spent for the intended purpose. Operating plans for the program year are submitted to Congress and revised as needed over the two year time period if changes in program expenditures need to be made. Internally, obligation and cost plans are developed, compared to actual spending, and reviewed monthly by all levels of the program. Contractor and government accounting systems are audited periodically to ensure compliance with government standards.

Evidence: Through September 30, 2005, 91% of FY05 program funds had been obligated. Individual program obligations range from 55.3% to 99.1%, with most programs in the mid to upper 90% range. The remainder of FY05 funds will be expended during FY06. Grants programs typically maintain a larger uncosted and/or unobligated carryover into the next year in order to guard against the likelihood of a continuing resolution. Specific reports that record and track the obligation and expenditure of program funds are as follows: contractor monthly and quarterly 533 reports, SF133 reports on budget execution and budgetary resources, FMS2108 year-end closing statement, and the annual Performance and Accountability Report.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The program has adopted effective management procedures to ensure that programs are executed in a cost effective and efficient manner.

Evidence: The program has policies to incentivize competitive outsourcing, best value procurement practices, and employee performance and productivity improvements. Information Technology is used extensively. Efficiency/effectiveness are generally measured in the timely delivery of scientific products that address the Strategic Plan and are consistent with Level 1 requirements and agreements. The implementation of full cost accounting has given program managers a more detailed view of all the costs associated with program activity. During the budget formulation process, Centers provide program managers with the estimated costs of doing business at that Center. Program managers then choose the most cost effective Center for their program. Contractors (including the Jet Propulsion Laboratory) are motivated to achieve cost effectiveness and efficiency via fee review. A projected cost growth of 15%+ triggers automatic review by senior management. Outcomes of past reviews have been program delay, redirection, or cancellation. All programs must meet uniform efficiency measures: each development project must complete its current phase within 110% of total life-cycle cost and schedule; each research project must allocate 80% of funding competitively; at least 90% of scheduled operating hours for operating spacecraft must be delivered; and the time within which 80% of NRA research grants are awarded must be reduced. Examples of operating mission efficiency include consistent and timely delivery of science data for the Mars Exploration Rovers, Mars Global Surveyor, and Mars Odyssey while maintaining and decreasing spacecraft operating costs. The Mars Odyssey mission has achieved an approximately 20% reduction in operating cost while maintaining science delivery. Cassini fuel efficiency is higher than planned, with spacecraft and all instruments operating nominally. The Radioisotope Power System program attempts to improve cost effectiveness by comparing costs for different alternatives and implementing the most cost effective approach; the project is currently developing advanced thermoelectric systems with higher efficiencies. In this program, it was determined that it would be cost effective to conduct basic materials research at universities, rather than at the Jet Propulsion Laboratory. Therefore, basic materials research activities are being conducted through small grants/contracts with multiple universities.

Does the program collaborate and coordinate effectively with related programs?

Explanation: The program collaborates and cooperates, where reasonable and practicable, with other NASA programs and/or Federal agencies where shared or similar goals and objectives might permit a more efficient use of resources while increasing the scientific and/or technological return. In addition, NASA maintains a willingness to collaborate with other nations in exploring the solar system where there is evidence of a genuine intersection of interests.

Evidence: The program coordinates and collaborates with NASA's Aeronautics programs to facilitate enabling and enhancing technology maturation and infusion and continues to work closely with the Space Operations Mission Directorate to ensure the availability of launch services. The program also maintains an ongoing collaborative relationship with various international partners at the program and project level for planning and coordination. The Mars Exploration Program maintains an ongoing collaborative relationship with its international partners at the program and project level for planning and coordination. The Science Mission Directorate/Solar System Exploration program is working with the Exploration Systems Mission Directorate to identify Lunar Science enabled by the Lunar Reconnaissance Orbiter, and to ensure that the mission's data are properly archived and available to the scientific community through NASA's Planetary Data System. The Science Mission Directorate also expects to support scientific data analysis of these data sets. The Science Mission Directorate and Exploration Systems Mission Directorate are both planning to ensure that scientific opportunities for science that arise within future Robotic Lunar Exploration Program missions and also the human phase of lunar exploration are exploited. Conversely, when the Science Mission Directorate selects lunar missions or lunar missions of opportunity such as the recently confirmed Discovery Moon Mineralogy Mapper, the Exploration Systems Mission Directorate is working with the Science Mission Directorate to obtain potential "opportunity" measurements that support exploration goals. As is always our approach, the Science Mission Directorate will work closely with the scientific community.

Does the program use strong financial management practices?

Explanation: The most recent NASA independent auditor report identified four material weaknesses for the agency, all of which are repeats, as well as noncompliance with the Federal Financial Management Improvement Act.

Evidence: NASA's FY 2004 Performance and Accountability Report (available at www.nasa.gov/about/budget/index.html) includes the communication from the NASA Inspector General and the report of the independent auditor. In addition, the GAO has published numerous reports identifying shortcoming in NASA's new financial management system as well as its financial management processes (example is GAO-04-754T released on May 19, 2004).

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The Science Mission Directorate, which manages the solar system exploration program, has a well-structured process in place to conduct monthly and annual performance reviews. Each program is required to present management, technical, and financial status information at monthly reviews held at HQ. Further, each program is evaluated and management deficiencies are noted through budget formulation process. The program director has frequent contact with directors of implementing organizations for program projects to discuss and mitigate any management deficiencies. In addition, there is a long tradition of inviting independent bodies to review projects for various deficiencies, including management, and propose solutions to any problems. Lessons-learned workshops are conducted to alert management to the kinds of mistakes that have been made under similar circumstances in the past so as to avoid repeating them in the future.

Evidence: Phoenix, Dawn, and the Moon-Mineralogy Mapper are example of missions that have recently undergone corrective management. Corrective management deficiencies include: Retiring the Phoenix mission risk associated with Entry Descent and Landing in order to achieve mission success and preventing cost overrun; changing the Dawn project manager to ensure an achievable and viable science mission within the rebaselined schedule; and only confirming the Moon Mineralogy Mapper after management deficiencies (cost, schedule, contingency plan, and data delivery) were resolved to the Science Mission Directorate Program Management Council's satisfaction.

Is the program managed by maintaining clearly defined deliverables, capability/performance characteristics, and appropriate, credible cost and schedule goals?

Explanation: Each solar system exploration program project is defined by a plan, which contains a clearly defined list of deliverables, along with the required performance characteristics, costs and schedule goals. Progress is measured by traditional methods such as earned value, schedule accomplishment, independent assessments, etc., in order to determine whether the limited window for launch can be met and whether the cost is exceeding predetermined limits. The results of these assessments and reviews impact program management decisions.

Evidence: A typical program project often has very limited launch opportunities. In order to meet those launch windows, a clearly defined list of hardware and software deliverables, along with required performance characteristics and costs and schedule must be developed, documented, maintained and managed. Documentation includes the Program Commitment Agreement, the program plans, and the project plans. The program manages carefully to the information contained within these documents, because allowing requirements creep and schedule slips might prove disastrous to the project's ability to launch on schedule. There is always a hardware descope plan in case the mission is projected to be over budget or has been losing schedule and must take an action to get back on schedule and budget. Any mission that projects costs in excess of 15% of total life cycle cost estimates will be subject to a termination review.

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: The program awards the great majority of its Research and Analysis (R&A) grants (> 99%) based on clear competitive processes that include a qualified assessment of merit. This assessment of merit is based on the following primary evaluation factors: 1) relevance to NASA objectives, 2) intrinsic scientific and/or technical merit, and 3) value and costs. However, approximately five to ten grants per year are awarded non-competitively as a result of congressional direction; these are carried in the Solar System Exploration Research budget.

Evidence: The great majority of grants selected for funding by the Science Mission Directorate, including the solar system exploration program, are broadly competed through the NASA Research Announcement process. Grant proposals must relate directly to both Agency and Mission Directorate goals and objectives. All proposals are peer-reviewed by a mix of scientific disciplines and are selected on merit. NASA also utilizes an electronic mailing list as part of its outreach efforts.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: NASA has an oversight practice that provides sufficient insight into and knowledge of grantees' activities.

Evidence: Discipline scientists take the results of the grant peer reviews and select investigation proposals for funding according to their scientific merit, relevance to NASA missions and programs, and availability of funds. These scientists then monitor the progress of the grant toward meeting its stated goals for the duration. Formal annual reports are provided by the grantee, and expenditures are tracked at a cumulative level. This gives the discipline scientists who work with the grantee sufficient insight into the performance to understand what the grantees do with the resources that are allocated to them. The formal annual reports are the primary method through which oversight and management control are exerted on the grantees. All grantees are required to submit annual progress report before the next increment of funding is released to them.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: NASA collects grantee performance data and makes it available to the public in a manner that is both useful and meaningful.

Evidence: Formal progress reports, which are a required output of each research and analysis activity, are submitted on an annual basis. The NASA lead scientist, together with appropriate discipline scientists review the progress reports before recommending continuation of the research activity. The results of grants-based research are broadly disseminated to the public through the use of science forums, publications, NASA press releases and news conferences, museum displays, educational materials, and NASA's web site. Due to high-priority requirements on NASA's web-based systems to accommodate e-Gov initiatives, including Grants.gov, NASA has currently suspended work on a publicly accessible database to post grantees' annual reports. NASA's future system will be responsive to the "Research Progress Report" standards established by the Research Business Models Subcommittee of the NSTC Committee on Science.

For R&D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain program quality?

Explanation: The program selects missions using a broadly competitive process and will only sole-source projects on the basis of a demonstrated unique expertise or capability and to maintain technical competencies at NASA field centers. In this way, the program can ensure that its funds are spent on only the highest quality missions. Funds are allocated to each mission as directed by Congress, and the financial status is tracked closely and reported regularly to program management. The program has the flexibility to re-allocate funds as needed to maintain program quality, and reports these changes to Congress in the Operating Plan. In addition, the peer review process is used extensivelyand is essential for a high quality, relevant program. The use of external peer review enhances the quality of NASA's investigations and activities because it brings the best and most critical national and international experts to the evaluation process. External peer review ensures that fresh viewpoints, alternative perspectives and state-of-the-art understanding are included in the evaluation process.

Evidence: The solar system exploration program maintains three programs through which most flight projects are selected: Mars, Discovery, and New Frontiers. Each program has a budget that must support ongoing missions as well as future missions yet to be selected. Regular financial and management status reports allow program directors to make decisions regarding the balance between ongoing and future missions as appropriate. Furthermore, most new missions include extended Phase A and B stages in order to retire technical risk and ensure program quality before going into full development. During development, review boards comprising contractor and NASA personnel conduct Preliminary Design Review, Critical Design Review, and Design Certification Review. This verifies that the "design-to" baseline is established and meets requirements, the detailed design is suitable and the "build-to" baseline is established, and each "as-built" system satisfies the final performance requirements.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: With the reorganization of the Science Mission Directorate (which took place since the last PART assessment), the Planetary Science theme (comprised of the old Solar System Exploration and Mars Exploration Themes) has created new long term performance goals that reflect the goals of the new organization. There is much carryover from the long term goals of the previous Themes which show that significant progress has been made.

Evidence: NASA's FY05 Performance and Accountability Report indicates that the Planetary Science Theme achieved 81% of its GPRA annual performance goals. Since the long-term performance goals reflect the cumulative effect of annual activities, and the degree to which long-term performance measures are being achieved is determined by the degree to which annual performance goals are being met, PSE can be said to have demonstrated significant progress toward achieving its long-term performance goals.

Does the program (including program partners) achieve its annual performance goals?

Explanation: Yes. The program exceeded its targets for delivering scheduled operating hours, competing its research funding, and reducing time to fund awarded grants. The agency did not meet its annual cost and schedule goals.

Evidence: Planetary Science (PS) Theme has achieved its annual performance goals to a large extent. NASA's FY05 Performance and Accountability Report indicates that the Planetary Science theme achieved 81% of its GPRA annual performance goals.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The program does demonstrate, to a large extent, improved efficiencies or cost effectiveness in achieving program goals each year. All key elements are in place for future missions to continue returning science data with increased efficiencies and cost effectiveness.

Evidence: The program tries to restrain unwarranted growth in cost and schedule (thereby increasing efficiency) by additional testing, extending initial Phase A and B development, and maturing critical technology off-line until it reaches a mission-acceptable level. This will cost more up front, but it saves the expenditure of larger amounts later on in the development cycle. Specifically, examples of operating mission efficiency include consistent and timely delivery of science data for the Mars Exploration Rovers, Mars Global Surveyor, and Mars Odyssey while maintaining and decreasing spacecraft operating costs. Consistent with NASA uniform efficiency measures (projects will complete current phase within 10% of total life-cycle cost; projects will allocate 75% of funding competitively; and all missions will be completed within 10% of baseline schedule), the program has continued to demonstrate improved efficiencies and cost effectiveness in achieving its program goal. The program has and will continue to demonstrate better mechanisms for cost estimation, continuous cost monitoring, control, and risk mitigation strategies.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: There are no other programs, either government or private, with similar purpose and goals and of similar size and scope, with which to compare the solar system exploration program.

Evidence:

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: Independent evaluations of the SSE program conducted by the National Research Council and the NASA Advisory Council confirm that the program is being managed effectively and is achieving anticipated results.

Evidence: The National Research Council's decadal survey reviewed the status of NASA's Solar System Exploration program, evaluated progress to date, and helped NASA prioritize missions and science objectives for the next ten years. The resulting report, "New Frontiers in the Solar System" (www.nap.edu/catalog/10432.html), reported substantial achievements in this program, such as the discovery of the diversity of bodies in the Kuiper Belt and evidence that Mars might have been hospitable to life in its past. The NASA Advisory Council conducts independent reviews of the program's annual performance (www.nasa.gov/pdf/139618main_FY_2005_PAR_text-only.pdf) and reported for FY 2005 that this program met most of it annual performance goals, including final integration and testing of the New Horizons/Pluto mission, which was successfully launched in January 2006. The National Research Council's "An Assessment of Balance in NASA's Science Programs" (2006), however, cautions that the NASA Science program "is neither sustainable nor capable of making adequate progress toward the goals recommended in...the decadal surveys." For the future, the Congress has mandated that each SMD division be assessed by the NRC every five years. The first assessment for the Planetary Science Division is anticipated to take place in 2007-08.

Were program goals achieved within budgeted costs and established schedules?

Explanation: The program goals were to a large extent achieved within budgeted costs and established schedules.

Evidence: With regard to cost, excluding Dawn, there was about 12% increase to the established life cycle cost for developing missions (Mars Reconaissance Orbiter, Deep Impact, and New Horizons).