Detailed Information on the
NASA Earth Science Assessment

Report for major missions on: estimated mission life cycle cost upon entering development; key schedule milestones associated with each mission phase for those missions formally approved for formulation; mission cost and schedule progress achieved in each phase before entering the next; and any plans to re-baseline life-cycle cost and schedule.

Post grantee performance data on a publicly available website.

Continue to seek efficiencies in Earth Science programs as demonstrated by developing at least one additional efficiency measure covering a greater proportion of the Earth Science budget.

Benchmark the approach and implementation of the Applied Sciences Program relative to other programs with similar purposes and objectives.

Measure: Number of distinct users of Earth Science data and services.

Explanation:In 2008, NASA changed the methodology for measuring distinct users. This metric will be retired after FY 2009. It will be replaced with a new metric, "Distribution of Earth Science data to the broad user community," which will more accurately measure Earth science data distribution.

Measure: Level of customer satisfaction with Earth Science information systems, as measured by a baselined index obtained through the use of externally administered annual surveys.

Explanation:Exceed the most recently available Federal government average rating of the Customer Satisfaction Index.

Measure: Number of reports issued with partnering organizations that validate that using NASA research capabilities (e.g., observations and/or forecast products) could improve their operational decision support systems.

Explanation:

Measure: Accomplishment of key development activities in support of advanced observing capabilities to help resolve key Earth science questions.

Explanation:FY08: Complete Orbiting Carbon Observatory (OCO) Operational Readiness Review; complete Global Precipitation Mission (GPM) Preliminary Design Review; launch OSTM; complete Glory mission Operational Readiness Review; complete Aquarius mission instrument Pre-Ship Review. FY09: Complete OCO Launch Readiness Review; complete Glory Launch Readiness Review; complete Aquarius instrument integration and testing; complete CLARREO mission advanced concept study; complete GPM Confirmation Review; complete Landsat Data Continuity Mission (LDCM) Critical Design Review (CDR); complete DESDynI mission advanced concept study; complete SMAP mission advanced concept study; complete ICESat II mission advanced concept study. FY10: TBD

Measure: Accomplishment of key technology activities in support of advanced observing capabilities to help resolve key Earth science questions.

Explanation:Advance at least two technologies to the point (Technology Readiness Level 6 or greater) that they could be demonstrated in space or in an operational environment, and advance 25% of all funded technology developments at least one Technology Readiness Level.

Measure: Percentage of scheduled operating hours delivered for all on-orbit instruments.

Explanation:

Measure: Cumulative percentage baseline cost overrun on spacecraft under development.

Explanation:Applies to missions in development with LCC >$100M; calculations consider growth in direct costs from baseline established at Mission Confirmation Review. *2006 and 2007 figures include Heliophysics missions under "Earth-Sun System" theme.

Measure: Cumulative percentage schedule slip on spacecraft under development.

Explanation:Applies to missions in development with LCC >$100M; calculations consider growth in schedule from baseline established at Mission Confirmation Review. *2006 and 2007 figures include Heliophysics missions under "Earth-Sun System" theme.

Measure: Percentage of budget for research projects allocated through open, peer-reviewed competition.

Explanation:Applies to Earth Science Research & Analysis (R&A) budget.

Measure: Cost per Earth Science data product (in the form of computer files) distributed to users.

Explanation:This metric measures the efficiency of EOSDIS in distributing data products. It is calculated each fiscal year by dividing the total budget of EOSDIS by the total number of products (computer files) distributed to users. NASA has several satellites orbiting the Earth to make frequent global observations. These satellites were launched during the period 1997 through 2004, and have instruments on board that provide measurements of many properties of the land, oceans and atmospheres. These measurements are converted into usable digital information products by NASA-funded scientists. NASA has a data and information system called EOSDIS that processes, stores and distributes the information products. These products, in the form of computer files, are distributed to a large number of users for purposes including science research, applications and education. Recently some changes have been made to EOSDIS to improve the mechanisms by which data are distributed to users. These include the storage of more data holdings on online disk systems rather than near-line robotic tape systems, on-demand generation of large volume data products to save storage space, and the provision of more on-line services to enable data volume reduction at the data centers (rather than transferring large volumes of data to users over networks). NASA has recently begun to see the benefits of these improvements.

Measure: Average number of days for completion of research award selections.

Explanation:Reduction of time within which eighty percent of NASA Research Announcement (NRA) research grants are selected, from proposal due date to selection notification. Targets represent a decrease of five percent peryear, with a goal of 180 days, and are based on constant (inflation-adjusted) peer-review contract funding and program personnel. *FY06 and FY07 actuals include Heliophysics as part of the "Earth-Sun System" theme. Heliophysics achieved faster processing times; standalone Earth Science figures were higher (280 in FY06 and 517 in FY07). **FY08 target represents a five percent decrease from the Earth Science Program's best performance in the past four years (251 days in FY04).

Measure: Progress in understanding and improving predictive capability for changes in the ozone layer, climate forcing, and air quality associated with changes in atmospheric composition (based on measurements from presently orbiting NASA and non-NASA assets). Progress is evaluated by external expert review.

Explanation:Advances are measured against the previously existing body of knowledge, which is well known by the panel of external experts performing the evaluation and is based on the existing body of peer-reviewed scientific literature. The evaluation criteria call for the panel to assess whether expectations for the research program have been met in the context of resources invested.

Measure: Progress in enabling improved predictive capability for weather and extreme weather events. Progress is evaluated by external expert review.

Explanation:Advances are measured against the previously existing body of knowledge, which is well known by the panel of external experts performing the evaluation and is based on the existing body of peer-reviewed scientific literature. The evaluation criteria call for the panel to assess whether expectations for the research program have been met in the context of resources invested.

Measure: Progress in quantifying global land cover change and terrestrial and marine productivity, and in improving carbon cycle and ecosystem models. Progress is evaluated by external expert review.

Explanation:Advances are measured against the previously existing body of knowledge, which is well known by the panel of external experts performing the evaluation and is based on the existing body of peer-reviewed scientific literature. The evaluation criteria call for the panel to assess whether expectations for the research program have been met in the context of resources invested.

Measure: Progress in quantifying the key reservoirs and fluxes in the global water cycle and in improving models of water cycle change and fresh water availability. Progress is evaluated by external expert review.

Explanation:Advances are measured against the previously existing body of knowledge, which is well known by the panel of external experts performing the evaluation and is based on the existing body of peer-reviewed scientific literature. The evaluation criteria call for the panel to assess whether expectations for the research program have been met in the context of resources invested.

Measure: Progress in understanding the role of oceans, atmosphere, and ice in the climate system and in improving predictive capability for its future evolution. Progress is evaluated by external expert review.

Explanation:Advances are measured against the previously existing body of knowledge, which is well known by the panel of external experts performing the evaluation and is based on the existing body of peer-reviewed scientific literature. The evaluation criteria call for the panel to assess whether expectations for the research program have been met in the context of resources invested.

Measure: Progress in characterizing and understanding Earth surface changes and variability of Earth's gravitational and magnetic fields. Progress is evaluated by external expert review.

Explanation:Advances are measured against the previously existing body of knowledge, which is well known by the panel of external experts performing the evaluation and is based on the existing body of peer-reviewed scientific literature. The evaluation criteria call for the panel to assess whether expectations for the research program have been met in the context of resources invested.

Is the program purpose clear?

Explanation: The Earth Science Program is designed to pursue strategic goal 3A of NASA's 2006 Strategic Plan: "Study Earth from space to advance scientific understanding and meet societal needs." As articulated in the 2007 Science Mission Directorate Science Plan, the foundation for pursuing this goal is built on the fundamental question: "How is the Earth changing and what are the consequences for life on Earth?" To achieve its mission, NASA pursues Earth science questions for which NASA technology and space-based observations make a defining contribution. NASA works with the science and engineering communities, other Federal agencies, and international programs to develop and demonstrate new observing capabilities from space, pioneer the use of these data to further science goals, and transition mature capabilities and results to operational users. NASA's authorizing legislation (Space Act of 1958), its amendments, and the Clean Air Act mandate that NASA conduct space activities to increase knowledge of the Earth, monitor the Earth's upper atmosphere, and report to Congress on stratospheric ozone depletion. NASA's Earth Science Program fulfills its overarching objective by posing a suite of scientific questions, which are broken down into tractable components. A systematic approach is used for each scientific question that includes enhancing observing capability and providing information and technology for the realization of societal and economic benefits.

Evidence: NASA's Science Mission Directorate (SMD) Science Plan (http://www.science.hq.nasa.gov/strategy/Science Plan 07.pdf) and FY 2009 Integrated Budget and Performance Document (IBPD) (http://www.nasa.gov/news/budget/index.html) articulate the rationale for the program. National Aeronautics and Space Act, as amended (http://www.nasa.gov/offices/ogc/about/space_act1.html); Clean Air Act, as amended (http://www.epa.gov/air/caa/).

Does the program address a specific and existing problem, interest, or need?

Explanation: The program addresses specific, identified national needs in several areas, including: the causes and consequences of climate change; improvements in the reliability and extension of weather forecasts; and the monitoring and eventual prediction of natural hazards such as floods, volcanic eruptions, and earthquakes. These national needs are identified in the strategic plans for the Climate Change Science Program (CCSP) and the U.S. Integrated Earth Observation System Strategic Plan. The 2005 NASA Authorization Act and the 2006 National Space Policy assign NASA a leading role in advancing fundamental scientific knowledge of the global integrated Earth system. NASA's role is to develop and make first use of new observing and research capabilities to understand the underlying processes, provide objective scientific information to researchers and decision-makers, and transition mature capabilities and results to operational users. NASA's Earth Science Program is essential to the imple??mentation of three major Presidential initiatives: Climate Change Research (June 2001), Global Earth Observation (July 2003), and the U.S. Ocean Action Plan (December 2004).

Evidence: National needs in the areas of climate, weather, and natural hazards, along with the need for NASA participation, are summarized in the following documents: Strategic Plan for the U.S. Climate Change Science Program (CCSP) (http://www.climatescience.gov); U.S. Weather Research Program (USWRP) Vision Document (http://box.mmm.ucar.edu/uswrp/program_overview/overview.html); Reducing Disaster Vulnerability Through Science & Technology; report of the Subcommittee on Disaster Reduction of the National Science and Technology Council (http://www.sdr.gov/SDR_Report_ReducingDisasterVulnerability2003.pdf), Reducing Loss of Life and Property from Disasters: A Societal Benefit Area of the Strategic Plan for U.S. Integrated Earth Observation System (IEOS) (http://www.sdr.gov/Reducing_loss_of_life_and_property_from_disasters.pdf).

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: The Earth Science Program is designed to uniquely address Earth science questions through space-based observations and does not significantly duplicate other programs or efforts. The Program expends considerable effort to insure that it is complementary to other Federal programs, although in some cases it fills critical roles for the community, like developing Landsat operational measurements, when these roles are not filled by other potentially more appropriate agencies. NASA, as directed by Congress in NASA's authorizing legislation, has the responsibility to conduct activities in space to expand Earth system science. Its role is unique and strongly complements the mission and role of other U.S. agencies (such as NOAA, NSF, USGS, EPA) by continually advancing Earth system science from space, creating new remote sensing capabilities, and enhancing the operational capabilities of other agencies and collaborating with them to advance Earth science goals. The NASA Earth Science Program is the only program to develop and fly new experimental instruments in space to resolve scientific questions about the Earth that cannot be answered by existing data, ground-based measurements, or measurements from operational instruments. Through interagency and international working groups, NASA conducts joint planning to ensure that observing capabilities are well leveraged. The largest component of the Earth Science Program is coordinated via the Congressionally mandated U.S. Global Change Research Program (USGCRP) and the Climate Change Science Program (CCSP). Earth Science Program research and observations are an integral part of the interagency CCSP strategic plan released in the summer of 2003. As a specific example, NASA is working with other agencies to implement the U.S. Carbon Cycle Science plan. NASA also ensures coordination with other agencies' efforts through the U.S. Weather Research Program (USWRP), and the National Disaster Reduction Committee, each of which actively seeks to avoid duplication among contributing agencies' programs. NASA is a key participant in the recent efforts to coordinate Earth observation both nationally (via USGEO) and internationally, contributing to GEOSS, while supporting the Ocean Action Plan and its research priorities. Memoranda of Agreement are executed with partners to ensure that duplication of effort does not occur. Partnering organizations include federal agencies (FAA, USDA, USGS, DHS, FEMA, EPA, CDC, NIH, DOE, DOD, DOI), state organizations (NSGIC, ASA, AAGS) and other national and international organizations. The Earth Science Program is the Nation's only program designed to partner with operational agencies to systematically benchmark uses of NASA's remote sensing data and research results (observations and model predictive capabilities) in their decision support systems.

Evidence: Memoranda of Agreement with program partners are kept on file by the Science Mission Directorate Interagency Agreements Manager. U.S. Carbon Cycle Science Plan (http://www.carboncyclescience.gov/documents/cc_sp_1999.pdf); Carbon Cycle Interagency Working Group (CCIWG) interagency coordination and comparison of participating agencies' missions (http://www.carboncyclescience.gov/partners.php); Climate Change Science Program Strategic Plan (http://www.climatescience.gov/Library/stratplan2003/default.htm).

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The Earth Science Program design is free of major flaws that would limit its ability to effectively and efficiently achieve its goals. The design is continually assessed, adjusted, and optimized by incorporating the advice and counsel of a very broad community of experts. The program design is built on a systematic, end-to-end approach that seeks to develop new technologies and data sets that contribute to improved predictive capability in research and operational forecast models. This encompasses interdisciplinary research, advanced technology development, development and deployment of global observing capabilities, scientific information product generation, and data assimilation and modeling. The use of open, competitive programs provides an effective approach for obtaining the best available talent to address our nation's scientific needs. An example of program design review is The National Research Council's recently released first-ever Earth Science Decadal Survey, entitled Earth Science and Applications from Space: National Imperatives for the Next Decade and Beyond." The Earth Science Program has already reflected the report's priorities in its FY 2009 budget request, and continues to develop plans to follow the report's recommendations, including the implementation of Venture-class missions to restore more frequent launch opportunities and facilitate the demonstration of innovative ideas and higher-risk technologies.

Evidence: Decadal Survey: http://www.nap.edu/catalog.php?record_id=11820. Earth Science Subcommittee Evaluations: SMD Science Plan: (http://www.science.hq.nasa.gov/strategy/Science Plan 07.pdf)

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: The program is designed to target key Earth Science Program questions, and to provide data sets and model results to researchers, operational users and decision-makers. The rigor with which the Earth Science Program is designed, structured, managed and funded ensures that resources reach only the intended beneficiaries and address the program's purpose directly. The Science Mission Directorate (SMD) Science Plan and supporting documents guide the activities of the Earth Science Program and provide the context through which specific research objectives are formulated, science investigations are solicited, and missions that address them are planned. Missions are prioritized on the basis of Decadal Survey science priorities, available budget resources, technological maturity, and partnering opportunities. The Earth Science program has a free and open data policy for dissemination of observations and research results. Data sets from NASA Earth Science Program missions are made available in standard formats to researchers and other users. The Earth Science research and analysis (R&A) program funds researchers on a competitive basis to use those data in their scientific research. The peer reviewed scientific publications serve as a vetting process for those results, which are available to policymakers world-wide. Earth Science Program tasks are broken down into discrete work breakdown structure (WBS) activities, and funds are issued at that level. These funds may not be spent on anything other than the purpose for which they were issued. Federal law prohibits the redirection of resources issued for one program to another program without express Congressional approval.

Evidence: The scientific purpose of each mission is well documented (see the Strategic Plan and the FY 2009 IBPD at http://www.nasa.gov/news/budget/index.html) and is linked to specific Earth Science Program and Agency goals and objectives. (See the SMD Science Plan at http://www.science.hq.nasa.gov/strategy/Science Plan 07.pdf). Data sets from NASA Earth Science Program missions are made available in standard formats to researchers and other users at gcmd.gsfc.nasa.gov/. Selected higher level data products are posted at www.esipfed.org/. Decadal Survey: http://www.nap.edu/catalog.php?record_id=11820.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The Earth Science Program's long-term measures focus on outcomes and meaningfully reflect the program's purpose. The measures reflect the program's goal of developing and demonstrating new space-based observing capabilities and providing global Earth observations and information to answer priority science questions and improve understanding and prediction of the Earth system. The Earth Science Program has seven long-term performance measures, six of which address scientific outcomes that embody the purpose of the program's science focus areas. The seventh measure addresses the extension of Earth science research and data to decision support tools and applications of societal benefit. Both the long-term and annual measures require that a panel of external experts review the growth of knowledge resulting from NASA and NASA-funded efforts. This external review details the advances being made toward each of the long-term science outcomes in the Strategic Plan.

Evidence: The long-term performance measures flow directly from the 2006 NASA Strategic Plan (http://www.nasa.gov/news/budget/index.html) multi-year outcomes and the 2007 Science Plan (http://www.science.hq.nasa.gov/strategy/Science Plan 07.pdf) research objectives. The performance measures are included in the Performance Measures section of the PART, as well as NASA's GPRA Performance Plan, which is part of the Agency's Integrated Budget and Performance Document (http://www.nasa.gov/news/budget/index.html).

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The Earth Science Program is ambitious by nature, pursuing an improved understanding of the complex nature of interaction among Earth's systems, requiring space based measurements and missions that have never been attempted before. The program seeks to advance the state of scientific knowledge in each of six interdisciplinary science focus areas: atmospheric composition, carbon cycle and ecosystems, water and energy cycle, climate variability and change, weather and Earth surface and interior. The nature of these objectives and the requirement for comprehensive and sustained sets of observations are reflected in the decadal timeframe for the program's long-term measures. Progress against these goals is measured by annual targets calling for a panel of experts external to the Agency to assess the growth in knowledge in each of the science focus areas, or outcomes. Advances are measured against the previously existing body of knowledge, which is well known by the panel of external experts performing the evaluation and is based on the existing body of peer-reviewed scientific literature. The material reviewed and produced by the panel details and evaluates the advances being made each year.

Evidence: The NASA 2006 Strategic Plan (http://www.nasa.gov/news/budget/index.html) and the Science Mission Directorate Science Plan (http://www.science.hq.nasa.gov/strategy/Science_Plan_07.pdf) contain an extremely ambitious set of goals, which are supported by the long-term performance measures in the PART. The measures can also be found in NASA's Performance Plan, which is part of the Agency's FY 2009 Integrated Budget and Performance Document (http://www.nasa.gov/news/budget/index.html). The external expert review was performed in FY 2007 by the Earth Science Subcommittee of the NASA Advisory Council. Annual assessment documentation and minutes of the Subcommittee meetings during which the evaluation process is completed are kept on file by the Earth Science Subcommittee Executive Secretary.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: Each long-term measure has associated annual measures to evaluate annual progress toward accomplishing each long-term science goal. Related annual measures track the degree to which programmatic milestones are reached in the development of observatories that will be used to acquire the data necessary for further scientific progress. In addition to measures directly addressing scientific progress and mission development, the Earth Science program is committed to several other annual measures that are important indicators of the program's ability to meet its long-term goals. These measures address the program's success in satisfying customer information needs, taking steps to transition assets from research to operations purposes, operating its spacecraft without interruption, competing research awards to ensure value for the dollar, and adhering to baseline cost, baseline schedule, and a competitive awards regime. The majority (ten) of the program's annual measures include objectively verifiable baselines and targets. Documentation of performance data is maintained in the Agency's PARWeb database. The six measures designed to assess progress toward the program's scientific outcomes call for a panel of external experts to review the growth of knowledge resulting from NASA and NASA-funded efforts. This external review details the advances being made toward each of the long-term science outcomes in the Strategic Plan. Advances are measured against the previously existing body of knowledge, which is well known by the panel of external experts performing the evaluation and is based on the existing body of peer-reviewed scientific literature. The external expert review was performed in FY 2007 by the Earth Science Subcommittee of the NASA Advisory Council.

Evidence: The performance measures are included in the Performance Measures section of the PART, as well as NASA's Performance Plan, which is part of the Agency's Integrated Budget and Performance Document (http://www.nasa.gov/news/budget/index.html). Annual assessment documentation and minutes of the Earth Science Subcommittee meetings during which the evaluation process is completed are kept on file by the Subcommittee Executive Secretary.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The Earth Science Program has baselines and ambitious targets for its annual performance measures. As an example, the annual performance goal related to cost and schedule growth establishes a 10% threshold, which is quite stringent when applied to programs charged with building one-of-a-kind spacecraft. Each mission involves a unique configuration of hardware, and, very often, new inventions of hardware, which makes cost and schedule estimation and performance very challenging. Targets for the development and technology measures stem from the baseline schedules and milestones established for the projects when they are confirmed to proceed. The majority of targets are quantifiable. However, given the program's primary focus on basic research, the performance plan includes measures with qualitative targets; these call for advances in the understanding of various facets of Earth science knowledge to result from NASA and NASA-funded efforts. Advances are measured against the previously existing body of knowledge, which is well known by the panel of external experts performing the evaluation and is based on the existing body of peer-reviewed scientific literature. The evaluation criteria call for the panel to assess whether expectations for the research program have been met in the context of resources invested. This approach is consistent with the finding of the NRC's Committee on Metrics for Global Change Research that annual quantitative measures have proven to be difficult to apply to Federal research programs, and that higher-order measures (e.g., outcome and impact) are more amenable to qualitative assessments based on stakeholder analysis/expert opinion.

Evidence: The performance measures are included in the Performance Measures section of the PART, as well as NASA's Performance Plan, which is part of the Agency's Integrated Budget and Performance Document (http://www.nasa.gov/news/budget/index.html). The measures include information regarding baselines and targets. Thinking Strategically: The Appropriate Use of Metrics for the Climate Change Science Program, National Research Council (http://www.nap.edu/openbook.php?isbn=0309096596).

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: Many Earth Science Program partners are involved in planning and establishing the program's goals and objectives. Consequently, they fully support and are committed to the achievement of both the annual and the long-term goals of the program. The Earth Science Program engages partners through the solicitation process and through documented agreements that make these goals clear. Missions must document their goals, objectives, and technical deliverables in program plans and commitments, signed agreements between NASA HQ, program offices, and the lead NASA center. The program also uses various instruments available to government agencies to enter into agreements with other entities. The purpose of such instruments is to obtain commitments to working toward and reporting on progress in achieving the annual and/or long-term goals of the program. Letters of Agreement, Memoranda of Understanding, and Technical Assistance Agreements have been signed with major international partners and other Federal agencies. NASA recently documented an agreement with the Japanese space agency JAXA to provide the launch of the Global Precipitation Mission (GPM). JAXA will supply the dual frequency radar and the launch and NASA will provide the microwave radiometer and the spacecraft. In June 2008, NASA will launch the Ocean Surface Topography Mission (OSTM) in cooperation with the French space agency CNES. This mission is a 50-50 split mission, with CNES providing the spacecraft and altimeter and NASA providing a radiometer and the launch. The agreement was documented in a Memorandum of Understanding in April 2006. Contracts and grants with industry and universities have been signed and task-level agreements between the Earth Science Program and the Jet Propulsion Laboratory and other NASA centers have also been reached. Earned Value Management (EVM) reporting has been established as a requirement for all contracts valued over $25M. As a general policy, NASA uses cost plus award fee contracts to pursue its high risk, technically challenging space systems developments. The program conducts award fee reviews (normally every 6 months) in addition to regularly scheduled and ad hoc reviews to verify partner commitment and performance. Independent contract and programmatic reviews are conducted routinely. Partners who fail to exhibit proper support can have their award fees limited or withheld, and in appropriate cases can be terminated from the program.

Evidence: The NASA Office of the Chief Engineer (OCE) maintains Program Commitment Agreements (PCAs). The Office of External Relations (OER) maintains Letters of Agreement and Memoranda of Understanding established with international and interagency partners. These documents are also kept on file by the SMD Interagency Agreements Manager.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: The Earth Science Program's effectiveness and relevance are subjected to regular reviews and evaluation by the National Research Council (NRC), the NASA Advisory Council (NAC) Science Committee and, in particular, its Earth Science Subcommittee. Annual performance toward achieving stated outcomes is determined and validated by annual external reviews. The NRC reviews meet the quality, scope and independence requirements of the PART and are used to assess many of the Federal Government's R&D programs. NASA contributions to international programs are often assessed, usually by science steering groups or the NRC, for their performance, as well as the quality and appropriateness of NASA contributions to the program. In addition, the first Earth Science Decadal Survey was delivered by the NAS to NASA in January 2007. The Survey was commissioned to inform NASA's planning for the next decade in order to ensure program performance and relevance. Scientific and programmatic progress and performance for the Earth Science Program are presented to the Earth Science Subcommittee (ESS), which reports to the NASA Administrator via the FACA-chartered NASA Advisory Committee (NAC) on an annual basis, and by the NRC on a periodic basis. Results of these reviews are the basis for revisions and updates of all Earth Science Program activities and plans. The NAC Earth Science Subcommittee (ESS) meets three to four times per year to conduct reviews of science and program implementation strategies. The NRC provides advice on the strategy and evolution of the whole CCSP and carries out independent analyses and studies via three Committees: i) Committee on Strategic Advice on the Climate Change Science Program, ii) Ad Hoc Committee on Analysis of Global Change Assessments, and iii) Climate Research Committee. These are included in the annual activities report prepared by the NRC. For discrete missions, independent reviews are held that include but are not limited to: non-advocate reviews, flight readiness and launch readiness reviews, and operational readiness reviews. The Agency has implemented quarterly performance reviews, entitled the "Baseline Performance Review" process, to independently evaluate programs and projects. On a monthly basis, an independent assessment team (consisting of PA&E, OCE, and OSMA) provides the Agency with a performance assessment and rating for all key programs and projects. The results of the independent assessment are reported at the Agency Program Management Council (PMC) and inform decisions at the Agency and Mission Directorate levels on the direction of programs and projects. For operating missions, external peer-review panels weigh the scientific merits of continued operation of missions that have fulfilled their designed lifetimes.

Evidence: Minutes and findings of ESS meetings: http://science.hq.nasa.gov/strategy/NAC_sci_subcom/index.html. Minutes and findings of NAC meetings: http://www.hq.nasa.gov/office/oer/nac/minutes.htm. The ESS assessment of progress toward annual performance goals is included in the Agency's FY 2007 Performance and Accountability Report (http://www.nasa.gov/news/budget/index.html). CCSP reports released by the NRC in 2007: 1) Evaluating Progress of the U.S. Climate Change Science Program: Methods and Preliminary Results (Sept. 2007, http://books.nap.edu/catalog.php?record_id=11934); 2) Analysis of Global Change Assessments: Lessons Learned (Feb 2007, http://books.nap.edu/catalog.php?record_id=11868). Studies conducted by the Space Studies Board Committee on Earth Science: http://www.nationalacademies.org/ssb. NASA Procedural Requirements (NPR) 7120.5D: http://nodis3.gsfc.nasa.gov/displayDir.cfm?Internal_ID=N_PR_7120_005D. PMC minutes are maintained by the NASA OCE.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The Earth Science Program budget request is the outcome of a rigorous bottom-up formulation process whereby the requirements are aligned to the science focus areas. Budget requests for each mission and project are dependent upon the successful completion of the current year's planned activities and estimates of resources required to complete the mission. NASA's full cost budgeting aligns the institutional and program resource needs and reflects a direct relationship between resources and Program strategic plans. The life-cycle cost requirements for each mission, in full cost, are included in NASA's Integrated Budget and Performance Document (IBPD). The IBPD shows the relationship between annual and long-term performance targets and resources, clearly indicating the effects of funding changes on expected performance.

Evidence: NASA's FY 2009 IBPDs (http://www.nasa.gov/news/budget/index.html) display important status data for each mission, list budget requirements for life cycle cost, and identify the specific long-term outcomes and annual performance goals enabled by the mission. Performance against each annual and long-term measure is reported to Congress each year in the Performance and Accountability Report (http://www.nasa.gov/news/budget/index.html). NASA is one of the few agencies to receive a "green" rating from OMB for budget and performance integration.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The strategic documents (e.g. Science Mission Directorate Science Plans, science questions, and discipline research plans) that guide the program are vetted by various external advisory groups. Both NASA's science advisory groups and the National Research Council (NRC) have performed these reviews. In addition to the Committee on Earth Science, the Space Studies Board invokes all other major NRC boards (including the Ocean Studies board, Polar Research Board, and the Board on Atmospheric Sciences and Climate) as needed in reviewing strategic planning products. NASA engages the NRC to develop decadal surveys with recommended priorities for science questions and measurements. The Earth Science Program has lacked an effective mechanism to jointly plan research to operations transitions with the National Oceanographic and Atmospheric Administration (NOAA), a partner agency. The program has moved to correct this by establishing regular, frequent interactions between the Earth Science Program Director and NOAA counterparts. These successful interactions have resulted in the identification of the first in a series of transitions, ocean surface altimetry (JASON III). The Earth Science Program has a Steering Committee comprising its senior leadership team that meets on a weekly basis to address strategic planning implementation issues.

Evidence: NAC/ESS: http://science.hq.nasa.gov/strategy/NAC_sci_subcom/index.html. National Research Council (NRC) Space Studies Board: http://www7.nationalacademies.org/ssb/. Committee on Earth Science: http://www7.nationalacademies.org/ssb/ces1.html.

Has the agency/program conducted a recent, meaningful, credible analysis of alternatives that includes trade-offs between cost, schedule, risk, and performance goals, and used the results to guide the resulting activity?

Explanation: The analysis of alternatives to implement NASA's goals in Earth science begins with the development and broad vetting of science goals, questions, and focus areas. Concepts and alternatives for achieving the goals of these focus areas are analyzed and summarized in program planning documents which identify the measurement and modeling capabilities needed, and provide a framework for assessing technical requirements and readiness, budget profile options and tradeoffs, internal management controls, operations concepts, and other implementation considerations (e.g. launch vehicle requirements, implied cumulative data rate). The NRC Decadal Survey has analyzed NASA's overall Earth Science Program and agreed that the current partitioning between mission and mission supporting activities represents an appropriate balance for achieving an effective Earth information system. The broad science and aerospace community is typically invited to propose mission alternatives to meet the program's goals through open and competitive solicitations. Selections themselves are based on analyses of trade-offs between cost, schedule, risk, and performance goals. After selection, programs and projects are subjected to independent reviews throughout their life-cycle to evaluate their ability to meet commitments. All mission proposals are analyzed for scientific merit, cost, schedule, technical and management acceptability by scientific and technical experts from inside and outside NASA. Implementation follows a strict process, including independent review defined by a NASA Procedural Requirements document (NPR 7120.5D). The Earth Science Program oversees all mission development and conducts independent reviews as appropriate. The program recently completed mission concept studies for each of the Decadal Survey missions. These studies reviewed implementation alternatives and developed cost, schedule, risk, and performance goals and trade spaces. The results were then used to identify the missions to be initiated first. Also during the past year, the program conducted its first Senior Review, an external peer review to weigh the scientific merits of continued operation of missions that have fulfilled their designed lifetimes.

Evidence: NPR 7120.5D: http://nodis3.gsfc.nasa.gov/displayDir.cfm?Internal_ID=N_PR_7120_005D The Decadal Survey Mission Concept Studies and Senior Review evaluations are kept on file by the Associate Director for Flight Programs.

If applicable, does the program assess and compare the potential benefits of efforts within the program and (if relevant) to other efforts in other programs that have similar goals?

Explanation: The Earth Science Program continually evaluates its efforts with respect to the relative potential benefits of alternatives, in part to determine if progress can be achieved using results produced outside of NASA. In some cases, NASA's Earth Science Program has unique goals that no other agency has or is capable of accomplishing. In other cases, coordination with others is appropriate and beneficial, as when missions are selected to work with in situ networks to contribute to a global observations strategy, as called for under the global Earth Observation Summit. Examples of NASA coordination with other programs include the interagency Committee on Earth Observation Satellites (CEOS), and the interagency working groups' coordination of U.S. government research in each of the focus areas of the CCSP. Documents that state requirements of satellite remote sensing in an integrated surface/in situ/satellite context include those found in the global climate observation system and the Integrated Global Observing Strategy.

Evidence: CEOS: http://www.ceos.org/pages/overview.html. CCSP: www.usgcrp.gov/usgcrp/ProgramElements/default.htm. Global Climate Observation System: http://www.wmo.ch/pages/prog/gcos/index.php?name=about Integrated Global Observing Strategy: http://www.igospartners.org/Part.htm Assessment documentation is maintained by the cognizant Program managers/executives.

Does the program use a prioritization process to guide budget requests and funding decisions?

Explanation: Independent outside organizations and the science community help set scientific priorities in line with Agency and Program goals and objectives. Within each science focus area are specific scientific advances to be pursued in the near and mid-term. These questions form the framework for identification of specific missions, technology tasks, and science NASA Research Announcements. The Earth Science Program uses scientific priorities, cost estimates, technological maturity, partnering opportunities, and the availability of funding to guide budget requests and funding decisions. Experience has shown that an effective program requires roughly equal investments in missions and mission supporting activities (including research, data analysis, modeling and non-space-based measurements). The National Research Council (NRC) recently reviewed NASA's Earth Science Program and related program elements of NOAA and USGS. The NRC delivered, after two years' discussion among the science community, the first ever Decadal Survey, titled "Earth Science and Applications from Space - National Imperatives for the Next Decade and Beyond," to help NASA prioritize missions and science objectives for the next ten years. The NRC agreed with the current partitioning between NASA missions and mission support and recommended that NASA undertake fifteen space-based missions over the next ten years while maintaining a balanced program. The Earth Science Program carried out community workshops and mission concept studies to develop detailed plans and budget requests in order to implement the Decadal Survey priorities. The Earth Science Program plans are included in the Science Mission Directorate (SMD) Science Plan, which shows the program's research priorities and mission priorities, together with the relevant rationale. These priorities are clearly reflected in the program's Integrated Budget and Performance Document.

Evidence: Decadal Survey: http://www.nap.edu/catalog.php?record_id=11820. SMD Science Plan: http://www.science.hq.nasa.gov/strategy/Science_Plan_07.pdf) FY 2009 IBPD: (http://www.nasa.gov/news/budget/index.html). The Decadal Survey Mission Concept Studies are kept on file by the Associate Director for Flight Programs.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The Earth Science Program conducts monthly and quarterly reviews to gather and analyze performance data from all participating organizations and compare against expected baseline performance data. Each flight mission and data management program activity is reviewed monthly for cost versus plan, schedule movement, reserves, mass and power margins, and estimate to complete assessments, often with the aid of earned value assessment. More detailed quarterly reviews are conducted for all Earth Science Program flight projects. The weekly, monthly, and quarterly reports are archived and available online to management. This information is used to assess progress toward meeting long-term outcomes and goals, and is used to develop risk mitigation strategies, adjust priorities, and/or make resource allocations. Key data includes the tracking of lifecycle costs (LCC) for missions in formulation, development, and operations. The Earth Science Program has descoped, restructured, or cancelled programs and projects based on the evaluation of programmatic performance data showing negative or non-recoverable trends. Independent science and program technical/management groups review progress toward achieving long-term performance outcomes and efficiencies at major program milestones in development and bi-annually during implementation. As part of the standard performance monitoring process, the NPP Project technical team recently reviewed the technical plans of NASA's NPOESS IPO partner, contractor NGST/Raytheon, for the VIIRS instrument development. These plans were found to be inadequate, and the NASA team worked with the IPO team to rework the entire development plan forward. This new plan was then reviewed and agreed to by the NPOESS contract team. While this activity resulted in a schedule slip, the quality of the final product can now meet NASA's technical standards.

Evidence: Weekly, monthly, and quarterly reports are retained in SMD archives, available online to management and other program personnel.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: Federal managers and partners are required to meet annual performance standards, including cost and schedule goals, and are rated through a number of formal mechanisms and requirements. Depending on their performance in meeting these standards, partners and managers are subject to awards, such as award fees or additional work, or penalties, such as diminished program participation or funding cuts. A program or project manager is clearly designated for each mission per NASA Procedural Requirements (NPR) 7120.5D, Program/Project Management. All employee performance plans for Federal managers include elements tied to the program's strategic plans. Program managers are held accountable for cost, schedule, and performance results, and their performance is rated in annual appraisals. Managers who fail to demonstrate the required level of performance are subject to a variety of disciplinary actions, including reassignment, or, in extreme cases, termination. Contractors and subcontractors are held accountable for the timely delivery and quality of products. Award fee reviews are performed on contracts and past performance evaluations are integral in Announcement of Opportunity (AO) criteria. All grants and cooperative agreements are subject to deliverables and milestones that must be met in order to receive funding renewal. Partners who fail to perform as required face having their participation reduced or terminated. Cooperative Agreements, for example, have defined milestones that must be met by the Principal Investigators (PI) in order for the PIs to receive payment. These milestones are measured against performance metrics that are part of the Cooperative Agreements and approved by the NASA Program Managers in order to release further funding. International and Federal government partners work in accordance with applicable Memoranda of Understanding (MoUs) and agreements, which detail schedule and performance commitments. In late 2005, when it became apparent that the NPOESS contractor was not going to deliver the VIIRS instrument to NPP in a timely manner, NASA called for an independent review. This led to a series of reviews and the identification of a Nunn-McCurdy breach in the NPOESS program. NASA was an integral part the Nunn-McCurdy process with the Air Force and NOAA that led to a restructuring and recertification of the NPOESS Program in the summer of 2006. In another example, the OCO instrument contract was originally awarded to an industry partner (Hamilton Sundstrand). When they failed to perform, NASA stepped in and began to selectively remove work from the partner and transfer it JPL. The final result was the complete withdrawal of work from the contractor with only a few key technical people from Hamilton assisting JPL in the development of the instrument. The OCO instrument is currently scheduled for delivery to the spacecraft before the end of March 2008.

Evidence: NPR 7120.5D: http://nodis3.gsfc.nasa.gov/displayDir.cfm?Internal_ID=N_PR_7120_005D MoUs are maintained by The Office of External Relations (OER) and also kept on file by the SMD Interagency Agreements Manager. NPOESS performance data documented in Agency Program Management Council (PMC) minutes and Baseline Performance Reviews, maintained by NASA OCE. OCO performance data documented in SMD archives, available online to management and other program personnel.

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: Annual NASA funds are available for obligation for two years and are fully obligated by the end of that period. NASA's program/project work breakdown structure implemented in FY 2006 provides greater visibility into the expenditure of funds, allowing NASA to verify that funds are being spent for the intended purpose. Operating plans for the program year are submitted to Congress and revised as needed over the two-year time period if changes in program expenditures are required. Internally, obligation and cost plans are developed, compared to actual spending, and reviewed monthly by all levels of the program. Contractor and government accounting systems are audited periodically to ensure compliance with government standards. As of September 30th, 2007, the Earth Science Program had obligated approximately 82% of its authorized FY 2007 two-year budget. The program had fully obligated its FY 2007 funding by the end of March 2008. This meets NASA's new OCFO obligation standards, which are designed to ensure that funds are obligated no later than eighteen months into the two-year obligation period. Currently, the Earth Science Program is initiating several actions to optimize its FY 2008 funding and to improve its overall funding execution. ESD issues several bi-weekly funding reports, maintains comprehensive planning and funding execution data, and initiates ongoing reviews with project staff to ensure that funding is being obligated appropriately and effectively. The Science Mission Directorate has instituted a detailed review of unobligated funds at the end of the fiscal year third quarter to ensure high obligation rates and the optimal use of funds, with the goal of having 92% of current year funding obligated by the end of the first year of the two-year obligation period.

Evidence: Reports that record and track the obligation and expenditure of program funds are as follows: contractor monthly and quarterly 533 reports (retained by the Contracting Officer's Technical Representative/COTR), SF133 reports on budget execution and budgetary resources, FMS2108 year-end closing statement (both retained by Center and/or Agency OCFO), and the FY 2007 Agency Financial Report (http://www.nasa.gov/news/budget/index.html).

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: All activity carried out in the program is either openly competed or selected after comprehensive peer review, and performance is monitored on a monthly basis (grants on an annual basis). Competition and regular contract review cycles assure that vendors selected perform in a cost-effective manner. The most effective method by which contractors are motivated to achieve cost effectiveness and efficiencies is through the fee review process. Several times a year, depending upon contract stipulations, a panel reviews the contractor's progress and assigns it a rating which determines how much fee the contractor will earn for that review period. In FY 2007, the Earth Science Program conducted its first Senior Review of all missions in extended operations to assess their scientific relevance and evaluate the cost versus benefit of continued operation. The program will continue to conduct these reviews biannually, using the resulting data to determine mission extensions and to guide decisions regarding funding levels. EOSDIS Evolution, the first phase of upgrading the 15-year old Earth Observing System Data and Information System (EOSDIS), which distributes nearly 4 terabytes of satellite data per day to a worldwide community of users, is being completed in 2008. This concerted and integrated implementation was based on a year-long user benefit study, with accompanying system analysis and trade studies, involving the system's representative community users, NASA Headquarters Program Executive, and GSFC implementation team. The Evolution effort provides improved management of archive volume growth, enhanced data access, and increased data distribution closer to the science knowledge cycle, while sharply reducing recurring costs of operation and sustaining engineering. The recurring cost savings from the EOSDIS Evolution effort to EOSDIS operations and sustaining engineering is over 17% per year. A new efficiency measure addressing data processing costs has been added to the SMD Performance Plan. SMD continues to examine its various program areas for further opportunities to achieve more efficient and cost-effective performance.

Evidence: NASA Advisory Council Earth Science Subcommittee (ESS) EOSDIS Evolution Presentation, January 2008. An Earth Science Program efficiency measure is included in the Performance Measures section of the PART.

Does the program collaborate and coordinate effectively with related programs?

Explanation: NASA plays a leadership role in conducting Earth science research through space-based observations and actively supports transition of select observations to operational use. To promote the utilization of these observations in accomplishing shared goals, the Program actively participates in joint planning and implementation efforts, joint initiatives, and jointly funded center and grant announcements with many other NASA programs and Federal agencies. NASA also pursues collaborations with other nations where there is evidence of an intersection of research interests. Currently, the Program is in discussions with international partners and other Federal agencies to evaluate collaborative opportunities for future Decadal Survey missions. To promote transition of select observations to operational use, NASA works closely with NOAA, USGS, and others. The research to operations transition has proven, in general, to be difficult due to unclear agency responsibilities and budgetary constraints, but many successes are evident. Along these lines, NASA recently completed an agreement with NOAA to re-manifest the (OMPS) Limb and CERES instruments on the NPP mission and is working with NOAA to identify a flight of opportunity for TSIS. Numerous joint projects, such as GOES, POES, and NPOESS, support both the research and operational communities. NOAA and the Air Force Weather Agency use Earth Science mission data products to provide daily space weather predictions for operational and national defense applications. Joint planning activities are also addressed by the NASA/NOAA Roundtable. During the Jan. 2008 meeting, the Roundtable chartered a working group for research to operations transition in such areas as ocean altimetry. The first annual "Section 306 Report" detailing coordination was delivered to Congress in July 2007; work is underway on plans for the following year. NASA program officials serve as CCSP principals and participate in working groups tasked with coordinating efforts and developing joint interagency plans, milestones and deliverables (e.g., synthesis and assessment reports). NASA participates in the U. S. Weather Research Program, developing common goals and coordinating efforts, and in the Natl. Space Weather Program coordinated by the Office of the Federal Coordinator for Meteorology. Close international coordination is carried out both bilaterally (e.g., Ocean Topography with France, Global Precipitation Mission with Japan) and through the Committee on Earth Observation Satellites. The Research and Applied Science Programs are an integral component of Administration and interagency collaborations, including the CCSP, the Climate Change Tech. Program, Commercial Remote Sensing Space Policy, the CENR Subcommittee on Earth Observation, the CENR Subcommittee on Disaster Reduction, and the Group on Earth Observations (GEO). In its participation with GEO, NASA has led the development of the SERVIR system, which is the Regional Visualization and Monitoring System for environmental management for the eight countries of Mesoamerica. SERVIR has grown into a multi-agency effort, with significant funding from outside NASA. SERVIR provides open access to a variety of satellite and vector data, online maps, environmental visualizations, and decision-support products in the context of the Societal Benefit Areas outlined for GEO. The partnership's effectiveness is demonstrated by the recent NASA/USAID decision to fund an expansion of SERVIR capabilities to Africa in FY 2008. NASA Applied Science has also demonstrated effective collaboration with the U.S. Forest Service, FEMA, and the State of California during the recent southern CA wildfires. NASA contributed new technology in unpiloted airborne systems (UAS), on-board data processing and real-time data transmission. The results were recognized as a significant contribution to the wildfire response and post-fire assessment and recovery effort.

Evidence: For major interagency collaborative programs such as CCSP, Earth Science engages the NRC to assess its program. For international programs, SMD conducts periodic bilateral meetings to assess current collaborations and future opportunities. "Implementing Climate and Global Change Research," NRC's review of the final U.S. Climate Change Science Program Strategic Plan: http://www.nap.edu/catalog.php?record_id=10635#toc. Memoranda for the Record created, maintained by Office of External Relations for bilateral meetings. CEOS: www.ceos.org/pages/overview.html. SERVIR: http://servir.net "NASA Flies Ikhana UAV to Help California Firefighters:" http://www.aviation.com/technology/071024-nasa-ikhana-california-wildfires.html

Does the program use strong financial management practices?

Explanation: SMD currently has extensive financial review and analysis processes in place to ensure approved funding is managed appropriately. The Earth Science Program utilizes the SMD processes, as well as additional procedures to deal with unique aspects of the program. However, the most recent NASA independent auditor report identified two material weaknesses for the agency, both of which are repeat items, as well as noncompliance with the Federal Financial Management Improvement Act.

Evidence: NASA's FY 2007 Annual Performance Report (http://www.nasa.gov/about/budget/index.html) includes the communication from the NASA Inspector General and the report of the independent auditor. In addition, the GAO has published numerous reports identifying s in NASA's new financial management system, as well as its financial management processes (GAO-04-754T released on May 19, 2004).

Has the program taken meaningful steps to address its management deficiencies?

Explanation: NASA and the Science Mission Directorate have developed a well-structured process to conduct monthly and quarterly performance reviews. (See Question 3.1.) This system of evaluating program management identifies deficiencies and provides for corrective action to be taken in a timely manner. Earth Science Program management also has frequent contact with directors of implementing organizations for Earth Science projects to discuss and mitigate any management deficiencies. Finally, independent reviews of programs and projects are conducted at major milestones and periodically during implementation to provide an unbiased quality check of the engineering and management efforts. Corrective action plans are prepared, reviewed, and implemented for any deficiencies identified. The Earth Science Program conducts program management in accordance with the NASA Program and Project Management Procedural Requirements (NPR 7120.5D). A hierarchy of Program Management Councils (PMCs) exists to ensure appropriate levels of management oversight for programs. PMCs are established at the Agency level, Directorate level, at the assigned NASA Centers, and at lower levels within each Center as required. These councils evaluate monthly and quarterly the cost, schedule, and technical performance of Earth Science programs to ensure commitments are being met. When deficiencies are identified, corrective actions are assigned and tracked to completion. As a result of this management system, the following deficiencies were identified, with corrective actions taken as detailed below: For example, the OCO instrument contract was originally awarded to an industry partner (Hamilton Sundstrand). When they failed to perform, NASA stepped in and began to selectively remove work from the partner and transfer it to the Jet Propulsion Laboratory (JPL). The final result of this fundamental change in the management approach to the OCO instrument was the complete withdrawal of work from the contractor, with only a few key technical personnel from Hamilton assisting JPL in the development of the instrument. The OCO instrument is currently scheduled for delivery to the spacecraft before the end of March 2008. Another example is the Aquarius mission, a partnership with CONAE, the Argentine space agency, providing the spacecraft and NASA providing the primary instrument and launch. After detailed review and collaboration with CONAE, it became clear to NASA fairly early in the development that CONAE and their contractor would not be able to meet the originally agreed-upon launch date. NASA sent a team to Argentina to meet and review plans and schedules with CONAE and their industry contractor. This effort resulted in the development of a new consensus launch date of May 2010, and the program has continued to monitor CONAE progress closely. This led to a change in approach, under which NASA is currently assisting CONAE with the implementation of their power subsystem and with the procurement of electronic parts in order to protect their delivery schedule. Efforts on the Glory mission have been intense, but less effective. The industry partner responsible for the Aerosol Polarimeter Sensor (APS) instrument has performed very poorly. NASA management at all levels from the Goddard Space Flight Center, the Earth Science Program, the Science Mission Directorate, and the Administrator's Office have met and negotiated with Raytheon senior management in efforts to control cost and schedule growth. As a result, Raytheon Project management personnel have been changed and implementation has been reorganized. In addition, NASA has placed personnel in-house at Raytheon to improve communication and performance.

Evidence: NPR 7120.5D: http://nodis3.gsfc.nasa.gov/displayDir.cfm?Internal_ID=N_PR_7120_005D. Weekly, monthly, and quarterly reports, which include details of management actions taken, are retained in SMD archives, available online to management and other program personnel.

Is the program managed by maintaining clearly defined deliverables, capability/performance characteristics, and appropriate, credible cost and schedule goals?

Explanation: The program clearly defines and documents the capabilities or characteristics that are expected from projects, including specific milestones to demonstrate progress towards completion, and clearly identifies who is responsible and authorized to make management decisions. Progress is measured by traditional methods such as earned value, schedule accomplishment, independent assessments, etc., in order to verify that the project's performance and schedule requirements will be met for the anticipated cost. Every flight mission is required to develop Level 1 Requirements as part of the Program Commitment Agreement (PCA), which is a key component of formal program implementation and documents the scientific, technical, schedule, and cost commitments for the program or project. In addition, flight projects are required to develop Mission Success Criteria which define the minimum capabilities that must be achieved in order to declare mission success. The Level 1 Requirements and the Mission Success Criteria are established and controlled by the responsible Program Management Council. The recently issued NASA Procedural Requirements NPR 7120.5D defines the deliverables and performance expectations and defines the requirements for key management documentation that include deliverables, cost, schedule, and technical performance characteristics. The program manages carefully to the requirements contained within these documents, using an array of tools and measures at Key Decision Point reviews to ensure that the project is being executed at a reasonable level of risk. Technical capability on orbit is the driving force; modifications to schedule and/or cost are considered before the program compromises on technical capability required to meet Level 1 requirements. Any mission that projects cost increases in excess of 15% of total lifecycle cost established at confirmation is subject to a termination review. The NASA Authorization Act of 2005 mandates Congressional notification if projected costs exceed this threshold. Should the projection exceed 30%, funding is halted by Congress and NASA must request reinstatement, with credible plans for keeping cost under control. Recent changes have been made in an effort to improve cost estimating, the most significant of which is the requirement for independent cost estimating to the 70% confidence level for missions with a lifecycle cost exceeding $250M. The Agency has also developed a "Baseline Performance Review" process to independently evaluate programs. On a monthly basis, an independent assessment team (consisting of PA&E, OCE, and OSMA) provides the Agency with an assessment. The results of the independent assessment (technical, schedule, cost, and programmatic) are reported at the Agency PMC and require directorate/program response.

Evidence: The key management documents required to plan and control programs and projects are defined in NPR 7120.5D. (http://nodis3.gsfc.nasa.gov/displayDir.cfm?Internal_ID=N_PR_7120_005C_&page_name=main).

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: The Earth Science Program awards a great majority of its Research and Analysis (R&A) grant funding (> 95%) based on clear competitive processes that include a qualified assessment of merit. This assessment of merit is based on the following primary evaluation factors: 1) relevance to NASA objectives, 2) intrinsic scientific and/or technical merit, and 3) value and costs. However, in most years, the program includes grants awarded non-competitively as a result of Congressional direction. Such grants are still subjected to a rigorous merit assessment, per the NASA Grant Information Circular, prior to being funded. Solicitation vehicles provide a "level playing field" for all entrants, new and previously funded. When the specified period of performance concludes, the grantee must again go through the same competitive process as other applicants.

Evidence: The great majority of grants are selected for funding in response to full and open competition through Announcements of Opportunity, NASA Research Announcements, and Cooperative Agreement Notices (http://research.hq.nasa.gov/Formats.cfm). Criteria for selection can be found in Appendix C of the Guidebook for Proposers (http://www.hq.nasa.gov/office/procurement/nraguidebook/proposer2008.doc). Grant Information Circular 08-02: http://ec.msfc.nasa.gov/hq/gic/gic08-02.html

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: The Earth Science Program maintains effective oversight of grantee activities and performance. Grant renewals require annual reports and demonstrated performance levels consistent with the milestones and quality of performance established by the peer reviewed proposals. The agency has improved its oversight of grantees with a revised Grants Information Circular that addresses unsolicited proposals and Congressional earmarks. Earth Science Program Managers utilize the results of the grant peer reviews to make recommendations for selections. Recipients of awards are finalized by the selecting official as indicated in the Research Opportunities in Space and Earth Science (ROSES) research solicitation. Managers monitor the progress of the selected grants toward meeting their stated goals. Formal annual reports are provided by the grantee, and cumulative expenditures are closely tracked. This gives the discipline scientist who works with the grantee sufficient insight into his/her performance to understand what the grantees are accomplishing with the resources that are allocated to them. The formal annual reports are the primary method through which oversight and management control are exerted on the grantees. Release of further funding is contingent on the manager's approval of this annual progress report. Close contact with funding recipients is maintained through site visits, Principle Investigator meetings, and discipline-oriented meetings. In accordance with the Agency's Grant and Co-operative Agreement section 1260.22, an annual progress report describing the accomplishments during the reporting period is due 60 days before the anniversary date of the grant. A final report, including a comprehensive summary of significant accomplishments, is due within 90 days after the expiration date of the grant.

Evidence: Requirements for awards can be found in Appendix D of the Guidebook for Proposers (http://www.hq.nasa.gov/office/procurement/nraguidebook/proposer2008.doc).

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: NASA collects grantee performance data and makes it available to the public in a manner that is both useful and meaningful. The agency has federally mandated external reporting requirements to the public on grants and contracts. The results of grants-based research are broadly disseminated to the public through the use of science forums, publications, NASA news releases and news conferences, museum displays, educational materials, and NASA's web site. The NASA website currently indicates when grantees' annual progress reports have been reviewed and approved, which is required prior to release of the next year's funding increment. The Agency is investigating database systems for posting grant results; further action will be taken consistent with the pending OSTP requirements regarding standardization of progress reports. The database is scheduled to become available to the public by mid-FY 2009. The system will provide for each report to be available for public view, but also provide for a non-public portion of the report to restrict access to proprietary content.

Evidence: NASA Science Mission Directorate web site: http://science.hq.nasa.gov/. News releases: http://www.nasa.gov/news/index.html. Information for educators, including teaching materials: http://www.nasa.gov/audience/foreducators/index.html. The processing status for grantees' annual progress reports can be viewed at https://www.nssc.nasa.gov/portal/site/customerservice/menuitem.7957303076be7df8282261f24dd72749/.

For R&D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain program quality?

Explanation: Nearly all Earth Science Program activity is selected in response to fully open competition through Announcements of Opportunity, NASA Research Announcements, and Cooperative Agreement Notices. The peer review process is used extensively and is essential for a high quality, relevant program. The use of external peer review enhances the quality of NASA's investigations and activities because it brings the best and most critical national and international experts to the evaluation process. External peer review ensures that fresh viewpoints, alternative perspectives and state-of-the-art understanding are included in this process. Mail and/or panel review is used for solicited as well as the small fraction of unsolicited research. The Earth Science Program is managed per NPR 7120.5D, NASA Program and Project Management Processes and Requirements. Science investigations (science instruments and data analyses) are 100% competed and peer-reviewed prior to the start of Phase A. Typical evaluation criteria include scientific merit, technical merit, and feasibility of the proposed investigation implementation (including cost risk). Once these projects are awarded and begin the definition process, they are subject to a risk mitigation phase in order to retire technical risk and ensure program quality before going into full development. Included in the development process are a series of reviews which serve to demonstrate that the baseline requirements are properly established and met. Review boards comprising contractor and NASA personnel conduct the Preliminary Design Review (PDR), the Critical Design Review (CDR), and the Design Certification Review (DCR). This certifies that the "design-to" baseline is established and meets requirements. NASA oversees all projects and holds the projects to uniform standards for management (NPR 7120.5D), including NASA-initiated independent reviews. Verification methods include test, analysis, independent verification and validation, demonstration and inspection.

Evidence: Announcements of Opportunity, NASA Research Announcements, and Cooperative Agreement Notices (http://research.hq.nasa.gov/Formats.cfm). NPR 7120.5D. (http://nodis3.gsfc.nasa.gov/displayDir.cfm?Internal_ID=N_PR_7120_005C_&page_name=main).

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The Earth Science Program has six long-term science goals, and both NASA and NASA-funded efforts have produced significant scientific advances in each of the program's science focus areas. The program has noted concern regarding progress toward Outcome 3A.5, understanding the role of oceans, atmosphere, and ice in the climate system and in improving predictive capability for its future evolution. This concern is due to uncertainties in climate data continuity and delays and technical issues related to the NPOESS Preparatory Project (NPP), a joint mission between NASA and the Integrated Program Office (IPO) of NPOESS. The IPO is developing and will deliver to NASA the three main instruments. NPP has been experiencing significant delays and technical issues. NASA continues to work with partner agencies to utilize the assessment information developed by the NPP project and science team in developing a joint mitigation strategy and implementation plan. However, NASA's current Earth Science missions have continued to operate much longer than planned; due to their continued provision of necessary measurements, the NPP delays have had only a marginal impact thusfar.

Evidence: NASA's progress in pursuit of its long-term Earth Science goals is reflected in the annual evaluations performed by panels of experts external to the Agency, in particular the Earth Science Subcommittee (ESS) of the NAC Science Committee. In FY 2007, the ESS determined that expectations for the research program had been met across the six long-term performance goals, in the context of resources invested. As a representative example of the long-term scientific progress, innovative analysis techniques of GRACE observations led to estimating changes of the Greenland and Antarctic ice sheets with unprecedented spatial detail. The estimated net change in mass of 100 gigatons per year is equivalent of 0.3 mm/yr of sea level rise, as derived from ocean altimetry measurements and ICESat measurements since the mid 1990s. (See http://www.nasa.gov/news/budget/index.html for the FY 2007 Performance and Accountability Report.) The external expert review was performed by the Earth Science Subcommittee of the NASA Advisory Council. Annual assessment documentation and minutes of the Subcommittee meetings during which the evaluation process is completed are kept on file by the Earth Science Subcommittee Executive Secretary.

Does the program (including program partners) achieve its annual performance goals?

Explanation: The Earth Science Program has achieved its annual performance goals to a large extent. The program's success in delivering scientific results is reflected in its achievement of all six annual performance measures related to science progress. The program also exceeded its goals for delivering scheduled instrument operating hours, competing research funding, and attaining high levels of customer satisfaction, among others. Some cost and schedule goals were not met. In particular, goals were missed due to delays on the OCO and Glory missions, and the program failed to meet its overarching cost and schedule metric. In response to cost and schedule performance issues, the Science Mission Directorate has recently adopted a new management approach which features control of cost through scope changes and other methods that allow missions to be managed within cost caps while maintaining risk at acceptable levels.

Evidence: Data on annual performance goals is provided in the Performance Measures section and can also be found in NASA's FY 2007 Performance and Accountability Report (http://www.nasa.gov/news/budget/index.html). The external expert review was performed in FY 2007 by the Earth Science Subcommittee of the NASA Advisory Council. Annual assessment documentation and minutes of the Subcommittee meetings during which the evaluation process is completed are kept on file by the Earth Science Subcommittee Executive Secretary.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The Earth Science Program demonstrates, to a large extent, improved efficiencies and cost effectiveness in achieving program goals each year. A key area of activity with regard to efficiencies has been in the area of mission operations. NASA's re-engineering, consolidation, and automation of flight operations have led to documented cost savings. This is especially important because many Earth Science missions are in extended operations. There is also a strong emphasis on reducing the validation period needed to make Earth Science data available to the wider community. This has directly improved access to science and applications data at reduced costs. In the area of Mission Operations, NASA has invested in multi-satellite control center operations and development of robust automation procedures that allow missions to share personnel and to operate autonomously on weekends. At Goddard Space Flight Center, the Aura and Aqua flagship missions share both control center and a unified Flight Operations Team; at the University of Colorado's Laboratory for Atmospheric and Space Physics (LASP), three Earth Science missions (SORCE, ICESat, QuikSCAT) are operated along with one Heliophysics mission (AIM). The LASP model has proved to be extremely effective: in addition to sharing a limited cadre of professional flight operations engineers among 4 missions and utilizing automation effectively during weekends, LASP operates a student program where professionals train, certify and supervise university students as members of the flight team. The students are highly motivated and provide exceptional value for supporting efficient and low risk flight operations. Other recent investments in autonomous procedures that have resulted in greater efficiencies: (1) within the Terra mission operations, an advanced solid state recorder scheduling tool has reduced the task of planning recorder buffer playbacks from 20 hrs to 1 hr ; and (2) the science value of EO-1 observations has been increased through use of an autonomous dynamic re-tasking technology to incorporate cloud model results so that EO-1 imagers can be scheduled to target areas most likely to be cloud-free, increasing overall useful images by 12%. EOSDIS Evolution, the first phase of upgrading the 15-year old Earth Observing System Data and Information System (EOSDIS), which distributes nearly 4 terabytes of satellite data per day to a worldwide community of users, is being completed in 2008. This concerted and integrated implementation was based on a year-long user benefit study, with accompanying system analysis and trade studies, involving the system's representative community users, NASA Headquarters Program Executive, and GSFC implementation team. The Evolution effort provides improved management of archive volume growth, enhanced data access, and increased data distribution closer to the science knowledge cycle, while sharply reducing recurring costs of operation and sustaining engineering. The recurring cost savings from the EOSDIS Evolution effort to EOSDIS operations and sustaining engineering is over 17% per year. The Program also seeks to restrain unwarranted growth in cost and schedule (thereby increasing efficiency) through additional testing, extending initial Phase A and B development, and maturing critical technology off-line until it reaches a mission-acceptable level. Maturing technologies prior to development, as recommended by Government Accountability Office reports, costs more up front, but is expected to save the expenditure of larger amounts later in the development cycle. The Science Mission Directorate has introduced a new efficiency measure addressing operating mission data processing costs. Baseline data and ambitious, yet achievable, targets have been established for this measure.

Evidence: NASA Advisory Council Earth Science Subcommittee (ESS) EOSDIS Evolution Presentation, January 2008. The efficiency measure is included in the Performance Measures section.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: There are no other programs, either government or private, with similar purpose and goals, with which a direct comparison can be made to NASA's Earth Science Program (ESP). No other agency or organization has the end-to-end capability to formulate new one-of-a-kind experiments; develop the associated instruments, space systems and ground-based calibration and validation systems and the needed retrieval algorithms; collect, process, archive and distribute the associated data; execute a peer reviewed research and analysis (R&A) program to improve understanding of the Earth as a system; and use that improved understanding to formulate the follow-on experiments. The Program's scientific research associated with R&A is similar to that of such organizations as the NSF, although the focus on new research satellites differentiates NASA's efforts from those of other Federal agencies. The Program follows the same basic strategy as the NSF by relying on the peer review process for competition and selection of its researchers, as well as the scientific literature peer review and publication process for the vetting of research results. A comparison can be made with the space agencies of our international partners such as CNES and JAXA, although most of them do not maintain their own research programs, which instead are run by other entities within the sponsoring nation or multinational coordinating mechanism. While these programs have made critical contributions, NASA provides these services on a scale that other countries cannot match. [The Earth Observing System Data and Information System distributes nearly 4 terabytes of satellite data per day to a worldwide community of users. The Earth Science Program is also a cornerstone of the international climate research effort, serving as the largest contributor to the U.S. Climate Change Science Program.] Comparison of the Earth Science Program's space system development to those of military or intelligence systems is difficult, as those developments are most often operational, rather than research, in nature. In addition, the acquisition approach used by these organizations is very different due to the operational nature of the effort and the fact that they do not have the cadre of civil service engineers and scientists that populate NASA's ten centers, which play a large role in NASA's acquisition process. However, while NASA's Earth Science Program is unique in its integration of science and space-borne remote sensing, some comparisons along subprogram lines can be made with other programs. In all cases, the ESP compares favorably with the related programs. In terms of overall scientific performance, the NRC Decadal Survey judged that NASA, along with NOAA and USGS, had made extensive scientific contributions with their satellite observing capabilities. For ESP, scientific progress is made through its over 1700 competitive grants and contracts. The scope and size of the ESP research effort is roughly equal to the NSF Geosciences Directorate (approx 1500 current grants) and achieves comparable numbers of literature reports (approx 2000 publications/year for NASA supported work). On the flight side, NASA's Earth Science Program maintains a fleet of fourteen technologically advanced research satellites and is developing seven new missions on a flight budget that is 30% less than that of NOAA NESDIS. The NOAA program currently operates sixteen satellites and is developing four new missions. NPOESS, the largest of NOAA's missions under development is designed to transition highly successful NASA EOS research sensors, such as MODIS and AIRS, to operational use. However, as noted by the NRC Decadal Study, the replacement sensors on NPOESS, when they exist, are generally less capable than their EOS counterparts. Furthermore, NASA's performance in implementing EOS is proving to well surpass NOAA's performance on NPOESS in terms of meeting cost and schedule commitments.

Evidence: Announcements of Opportunity, NASA Research Announcements, and Cooperative Agreement Notices (http://research.hq.nasa.gov/Formats.cfm). NASA Advisory Council Earth Science Subcommittee (ESS) EOSDIS Evolution Presentation, January 2008. NSF Geosciences Directorate grants (http://www.nsf.gov/awardsearch/).

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: Independent evaluations of relevance, quality, and performance have indicated that the Earth Science Program is effective and achieving results. Peer review of proposals submitted in response to Announcements of Opportunity (AOs) and NASA Research Announcements (NRAs) serve as very effective quality reviews for the Science Mission Directorate. Retrospective quality evaluations of the overall ESD program conducted by the NAC Earth Science Subcommittee (ESS) have shown significant progress in each of the program's six science focus areas. Scientific and programmatic progress and performance for the Earth Science Program are presented to the Earth Science Subcommittee (ESS), which reports to the NASA Administrator via the FACA-chartered NASA Advisory Committee (NAC) on an annual basis, and by the NRC on a periodic basis. Results of these reviews are the basis for revisions and updates of all Earth Science Program activities and plans. The NAC Earth Science Subcommittee (ESS) meets three to four times per year to conduct reviews of science and program implementation strategies. The first Earth Science Decadal Survey was delivered by the National Research Council (NRC) to NASA in January 2007. The Survey was commissioned to inform NASA's planning for the next decade in order to ensure program performance and relevance. The Decadal Survey, while noting that the U.S. foundation of global observations is at great risk due in part to budgetary reasons, recognized the "extensive" scientific and societal contributions of the U.S. satellite observing capabilities. In addition, the NRC annually reviews aspects of the program, in particular the climate change related program. These reviews are conducted in connection with the interagency Climate Change Science Program (CCSP). NASA's Earth Science Program constitutes more than 70% of the whole program as reported to Congress in the Our Changing Planet annual report. The most recent NRC reviews of the CCSP attest to the effectiveness of the program in achieving great progress regarding the global physical climate system, credited to a large extent to NASA's global climate observations and research. However, the studies identified weakness and lack of progress towards developing regional climate understanding, which is of value to decision support. Given that NASA's role is primarily targeted toward the global climate system, the NRC concerns are more applicable to the work of other agencies. In the future, NASA will begin to address regional science issues, especially as the suite of observational capability providing higher spatial resolution observation, and that of the models that utilize and assimilate them, increase. For discrete missions, independent reviews are held that include but are not limited to: non-advocate reviews, flight readiness and launch readiness reviews, and operational readiness reviews. The Agency has implemented quarterly performance reviews, entitled the "Baseline Performance Review" process, to independently evaluate the performance of programs and projects. On a monthly basis, an independent assessment team (consisting of PA&E, OCE, and OSMA) provides the Agency with a performance assessment and rating for all key programs and projects. The results of the independent assessment are reported at the Agency Program Management Council (PMC) and inform decisions at the Agency and Mission Directorate levels on the direction of programs and projects. For operating missions, external peer-review panels weigh the scientific merits of continued operation of missions that have fulfilled their designed lifetimes.

Evidence: National Academy reports relevant to Climate Change Science Program released in 2007 were: 1) Evaluating Progress of the U.S. Climate Change Science Program: Methods and Preliminary Results (Sept. 2007, http://books.nap.edu/catalog.php?record_id=11934); 2) Analysis of Global Change Assessments: Lessons Learned (Feb 2007, http://books.nap.edu/catalog.php?record_id=11868). Space Studies Board Committee on Earth Science: http://www7.nationalacademies.org/ssb/ces1.html. USGCRP Our Changing Planet: http://www.usgcrp.gov/usgcrp/default.php.] The results of the ESS FY 2007 science evaluation are contained in the FY 2007 Performance and Accountability Report (http://www.nasa.gov/news/budget/index.html). Peer review records are maintained by Program Discipline Scientists.

Were program goals achieved within budgeted costs and established schedules?

Explanation: Earth Science has performed its work well within cost and schedule in the areas of research, data systems development and operations, technology, applied science and mission operations. These areas represent 66% of the Earth Science FY 2008 budget. Expectations have been exceeded in several areas, including the EOSDIS Evolution, which has increased data information systems capabilities while decreasing costs, and in the area of mission development risk. Cost and schedule performance for Earth Science missions in development has been mixed. Numerous missions have experienced cost and/or schedule increases. In some cases, these problems have been due to technical difficulties experienced by NASA's domestic and international government partners, but other facts have also affected this performance to the overall detriment of the Earth Science program at NASA. The OSTM mission (a partnership with CNES, the French space agency) is scheduled to be launched in June 2008, consistent with the development cost and schedule commitments set at its Mission Confirmation Review (MCR). OCO is scheduled for a December 2008 launch and has experienced an 18% cost growth due to technical difficulties with the instrument development. These difficulties led to NASA bringing the instrument development in-house at JPL from the contractor. The Glory mission has experienced cost and schedule growth due to instrument development issues at Raytheon. NASA has a full time presence at Raytheon and has established monthly management reviews with senior level SMD and Earth Science Program personnel. Both NPP and Aquarius have experienced launch date delays and associated cost growth due to partner's (NPOESS and CONAE, the Argentine space agency) delayed delivery of mission hardware. NPOESS has delayed the delivery of the VIIRS instrument, and CONAE the SAC-D spacecraft. NASA has provided significant technical assistance to the NPOESS partners and is working with CONAE to resolve long-lead parts and power system issues. The Earth Science Program has been able to make trades within the program to cover the cost growth and has been able to minimize the impacts to other portions of Earth Science. While these partner delays are difficult and result in unplanned costs, maintaining and building partnerships in Earth Science is critical to the long-term strategy of cost sharing/reduction and data sharing, validation and acceptance.

Evidence: The FY 2009 IBPD contains specific cost and schedule information for missions in development (http://www.nasa.gov/news/budget/index.html), as do internal cost and schedule documents.