Detailed Information on the
National Institute of Standards and Technology Laboratories Assessment

Investing in research in key technologies to support new measurement requirements, important for the U.S. economic and scientific infrastructure, in such areas as nanotechnology, enabling the hydrogen economy, and quantum science in FY 2008.

Invest in facility improvements at NIST labs to sustain the program's strong research capability in FY 2008.

Measure: Annual output of peer-reviewed technical publications.

Explanation:This measure is the number of peer-reviewed technical publications approved by the NIST Editorial Review Board. It indicates the quality and demand for NIST research results and standards services.

Measure: Standard reference materials sold

Explanation:This measure is a count of the number of standard reference material (SRM) units sold to customers in industry, academia, and other government agencies. It is used to indicate the dissemination and use of NIST measurement standards.

Measure: Web access to / downloads of NIST-maintained databases

Explanation:This measure is the number of downloads of NIST-maintained scientific and technical databases. It is used to indicate the dissemination and use of NIST measurements and standards tools, data, and information.

Measure: Number of calibration tests performed

Explanation:This measure indicates the quantity of physical measurement services provided by NIST for its customers, including calibration services, special tests, and Measurement Assurance Programs for claibrating entire measrement systems.

Measure: Technical peer review of quality, relevance, adn technical merit of NIST Laboratories program by the National Research Council.

Explanation:The measure indicates whether the laboratory-wide external technical assessments of quality and performance are conducted. Typically four NIST laboratories are assessed each year by the NRC.

Measure: Number of economic Impact Studies (Net benefit to cost ratio; social rate of return) initiated

Explanation:This measure is the number of microeconomic impact studies initiated to estimate and evaluate the outcomes that derive from NIST research programs.

Measure: Citation impact of NIST-authored publications

Explanation:This measure represents NIST's "relative citation impact" - that is, the average citation rate per NIST publication relative to the Institute for Scientific Information (ISI) baseline citation reate number for all scientific and technical organizations. The relative citation impact is a 5-year running average ending in the year below, that considers all NIST-authorized publications during the period, and how many citations they received versus the ISI baseline during the period.

Is the program purpose clear?

Explanation: The NIST laboratory program mission provides a clear, well-defined focus on technology infrastructure: the mission is "to develop and promote measurement, standards, and technology to enhance productivity, facilitate trade, and improve the quality of life".

Evidence: The mission and purpose of the NIST laboratory program is stated in the DOC Annual Performance Plans, budget justifications, and associated documents. See 15 USC 272 for NIST's statutory authority.

Does the program address a specific and existing problem, interest, or need?

Explanation: The measurements and standards capabilities provided by the NIST laboratories have been and remain a critical component of the nation's scientific, technical, and economic infrastructure. The measurements and standards mission of the laboratories is inherently public: Common and broadly accessible measurement tools and methods, physical standards, and documentary standards are essential to the nation's R&D-intensive organizations, providing R&D and manufacturing productivity enhancements, transactional efficiency in the marketplace, and a more level playing field for international trade.

Evidence: The inherently governmental power to "fix the standards of weights and measures" is written in the U.S. Constitution (article I, section 8). The current measurement and standards role of the NIST Laboratories is rooted in the mission of the National Bureau of Standards, which was established in 1901. See 15 USC 271, in which Congress declares, among other findings, that "The Federal Government should maintain a national science, engineering, and technology laboratory which provides measurement methods, standards, and associated technologies and which aids United States companies in using new technologies to improve products and manufacturing processes."

Is the program designed so that it is not redundant or duplicative of any Federal, state, local or private effort?

Explanation: There is no other federal agency with the same mission as the NIST laboratories. By design, the nation's measurement and standards system requires a peak-level national measurement institute, which is the core role of the NIST laboratories. This function cannot be provided by state or other governmental organizations or by the private sector.

Evidence: NIST's distinctive mission and role is reflected in numerous federal and international agreements that designate NIST as the national measurement institute (NMI) for the United States. For instance, international trade rules that mandate measurement traceability to an NMI designate NIST in that capacity; this role is similarly recognized in mutal recognition agreements (MRAs) with other NMIs as well as in federal laws that prescribe NIST roles with respect to other agencies (in areas such as computer security, construction investigations, and others).

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: Overall, the current structure appears to be the most efficient and effective means for achieving the mission of the NIST laboratories. NIST has designed an impact logic model to identify the path from resource inputs to potential outcomes and has established evaluation processes for validating progress at various stages along the path. It uses this model and evaluations to assess organizational structure, management processes, and balance of its research portfolio. OMB has encouraged NIST to explore opportunities to obtain research services through grants or contracts where appropriate to obtain increased efficiencies. However, OMB recognizes that in many cases NIST's researchers also serve on various standards boards or provide other services and functions that could create a conflict of interest if the individual were from a private firm; these are important considerations in competitive sourcing decisions.

Evidence: Note that the impact logic model for the NIST laboratories is provided in the NIST budget justification and outlines the basic linkages between the activities of the NIST laboratories, the use of NIST's measurement and standards products and services by customers, and the ultimate impacts on innovation, productivity, and improved public health and safety. This logic model has been tested via approximately 30 microeconomic impact studies over the last decade, which have measured the net public benefits of specific outputs from the NIST laboratories. Commerce's Management Scorecard includes recommendations for identifying positions that could be obtained through commercial sources.

Is the program effectively targeted, so program resources reach intended beneficiaries and/or otherwise address the program's purpose directly?

Explanation: Resources appropriated to the NIST laboratories are used to fund mission-specific intramural research. The measurement research, methods, tools, and services provided by the laboratories are planned with extensive input from NIST's customers and are broadly distributed and accessible to intended beneficiaries in the private sector, universities, and other public organizations. NIST's end impacts, evaluated in over 30 economic impact studies, attest to the broad dissemination and use of NIST's outputs.

Evidence: NIST plans and conducts its work with extensive input from intended beneficiaries in industry, universities, and other research-intensive organizations. At an operational level, the laboratories continuously search for ways to improve the efficiency and reach of NIST measurement and standards services. Two well-documented examples include the Josephson voltage standard (in which NIST devised an improved technical method for disseminating the measure of the volt) and Digital Encryption Standard Program (in which NIST enabled successive generations of a standard cryptographic algorithm). These cases illustrate processes NIST uses to plan and deliver measurements and standards capabilities to intended beneficiaries. For details and impact estimates, see NIST Planning Report 01-1 ( on JVS, available at www.nist.gov/director/prog-ofc/report01-1.pdf) and Planning Report 01-2 (on DES, available at www.nist.gov/director/prog-ofc/report01-2.pdf)

Does the program effectively articulate potential public benefits?

Explanation:  

Evidence:  

If an industry-related problem, can the program explain how the market fails to motivate private investment?

Explanation:  

Evidence:  

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: NIST has two primary long-term goals, which are identified in its strategic plan. These goals represent complementary elements of the laboratories' statutory mission: conducting R&D to improve physical measurements and associated standards--e.g. developing a measurement regime for next-generation technologies; and providing measurement and standards products and services to existing customers. These long-term goals are inherently difficult to quantifiably measure due to such factors as the breadth of NIST's research porfolio, the long time horizons associated with research programs, and the difficulty of evaluating specific NIST contributions to end outcomes that have complex causal structures. However, NIST has established a number of processes, including external reviews and economic impact analyses, to ensure that it is making progress in supporting these goals.

Evidence: Goal 1: Provide technical leadership for the Nation's measurement and standards infrastructure. Goal 2: Assure the availability and efficient transfer of measurement and standards capabilities essential to established industries. Sources: NIST 2010 Strategic Plan (available at www.nist.gov); and the DOC FY 2004 Annual Performance Plan (available at www.osec.doc.gov/bmi/budget/FY2004APP.htm). The NRC conducts annual review in which it assesses NIST's laboratory program for technical merit, relevance and effectiveness, and impact of resources. Economic impact studies estimate microeconomic impacts of NIST activities on various industry sectors. Links to recent evaluations and studies are cited in question 2.2. NIST has successfully worked with OMB to develop clearer long-term measures and targets, and further improvements may be possible over time.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: As discussed in the 2.1, it is difficult to establish meaningful, quantifiable targets for NIST's long-term strategic goals. NIST assesses progress toward its three long-term goals through extensive annual expert review conducted independently by the National Research Council. NIST further evaluates its long-term impacts through microeconomic impact studies. In terms of management practices, NIST's strategic management is overseen by the Visiting Committee on Advanced Technology (VCAT), which provides an additional check on the Institute's strategic direction and a method for identifying opportunities for continuous improvement.

Evidence: NIST's annual performanc goals and its performance evaluation system are provided in the DOC FY 2004 Annual Performance Plan. For an annual evaluation of the laboratories' quality, relevance, and performance, see the annual report of the NRC Board on Assessment of NIST Programs (current and prior years are available at www7.nationalacademies.org/NIST/NIST_reports.html). For economic impact studies that estimate long-term outcomes from the use of NIST's products and services, see www.nist.gov/director/planning/strategicplanning.htm#recent. See also the annual report of the VCAT (available at www.nist.gov/director/vcat/index.htm).

Does the program have a limited number of specific annual performance measures that demonstrate progress toward achieving the program's long-term measures?

Explanation: As with many intramural R&D programs, NIST's net performance in any given year cannot be represented by a limited number of goals with quantifiable performance measures. NIST does set annual performance goals for key outputs; collectively, these output goals represent a proxy for progress. In addition, NIST's annual external peer review system provides NIST with an additional measure of progress and effectiveness. While the current measures and processes are adequate, further improvements could be made. NIST has agreed to introduce new measures in FY 05 that better represent customer use of NIST's outputs, including a citation metric and improvements in the indicators of measurement transfer. NIST also has responded to OMB's request for a more systematic process for tracking how it implements recommendations that are generated by external reviews.

Evidence: NIST's annual performance goals are provided in the DOC FY 2004 Annual Performance Plan. For an annual evaluation of the laboratories' quality, relevance, and performance, see the annual report of the NRC Board on Assessment of NIST Programs.

Does the program have baselines and ambitious targets and timeframes for its annual measures?

Explanation: NIST's annual output measures establish specific targets for assessing performance. These targets are set carefully, and reflect NIST's attempt to balance quantity and quality. It is undesirable to view absolute increases in outputs independent of quality; for example, a large number of low-quality research publications is less desirable than a smaller number of high quality publications (for this reason, NIST includes aggregate citation data with its annual publication metric). In addition, it is difficult to directly associate annual output metrics with prior year budget inputs, as many outputs involve time lags or derive from work conducted over more than one fiscal year.

Evidence: As in prior years, NIST's annual output targets are included in the FY 2005 Annual Performance Plan; this year, the performance plan is integrated with the FY 2005 budget justification.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, etc.) commit to and work toward the annual and/or long-term goals of the program?

Explanation: Most of the appropriated resources for the NIST Laboratories are spent on intramural research and related activities conducted by the NIST laboratories.

Evidence: In general terms NIST does not fund program partners in a manner implied by this question. The NIST laboratories do, however, collaborate with a large number of organizations in order to develop their plans and conduct their work. Prominent long-term collaborations include JILA in Boulder and the Center for Advanced Research on Biotechnology within the University of Maryland. With regard to contracting practices, all NIST contracts are performance-based.

Are independent and quality evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: Each year since 1959 the National Research Council (NRC) Board on Assessment of NIST Programs has evaluated the quality, relevance, and performance of the NIST laboratories. This independent peer review process is systematic and comprehensive, covering all of the NIST laboratories. It provides the NIST Director and NIST's laboratory managers with a thorough evaluation of each laboratory's effectiveness and opportunities for improvement. This peer assessment is conducted in the context of NIST's strategic plans, which are derived with extensive input from industry and other customers. NIST also evaluates its long-term impacts through economic impact assessments that are conducted by independent external contractors.

Evidence: The NRC Board on Assessment of NIST Programs controls the selection of external experts who form panels suitable for reviewing the specialized scientific and technical programs within each of NIST's seven research laboratories. The results of each annual review cycle are published by the NRC and are available to the general public. NIST also uses highly qualified independent contractors to perform both retrospective and prospective economic studies.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The laboratory component of NIST's budget request is represented in the Scientific and Technical Research and Services (STRS) appropriation. NIST's annual STRS budget request reflects the Institute's long-term strategic plan and annual performance plan. Requests for new increases clearly tie to the focus areas identified in NIST's long-term strategic plan, and NIST uses formal reviews of OU Operating Plans as well as senior management peformance plans to assure that base funding is prioritized to achieve long-term goals. Further improvements in strategic planning would help to ensure that laboratories are effectively applying resources in support of priorities identified in the long-term strategic plan.

Evidence: Key documents: The NIST 2010 Strategic Plan and FY 2004 Annual Performance Plan contain NIST's long-term programmatic goals. The FY 2003 Congressional budget request reflects the structure of NIST's budget line items and sub-activities. With respect to opportunities for improvement, the NRC's FY 2002 report identifies strategic planning by laboratories as an area where improvement is needed.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: NIST has developed a new Institute-wide long-term strategic planning process that resulted in a limited number of long-term goals and strategic focus areas for the NIST Laboratories. The planning process includes new mechanisms for aligning Operating Unit plans with the NIST-wide plan, and includes external review with regard to the content of the plan as well as the planning process itself. Room for improvement still exists, but NIST's leadership has demonstrated a commitment to continued focus on strategic planning efforts.

Evidence: NIST 2010 Strategic Plan. In addition to stakeholder and customer comments on the plan itself, NIST's external advisory bodies routinely observe and comment on NIST's strategic planning processes generally. The NIST Visiting Committee on Advanced Technology (VCAT) provides this function for the Institute as a whole; see the Annual Report of the VCAT.

If applicable, does the program assess and compare the potential benefits of efforts within the program to other efforts that have similar goals?

Explanation: As discussed in section 1, there is not another entity within the U.S. with the same mission. NIST has attempted to collect data on other national measurement institutes, but little comparable information is available to provide meaningul comparisons.

Evidence:  

Does the program use a prioritization process to guide budget requests and funding decisions?

Explanation: In terms of programmatic priorities, the NIST 2010 Strategic Plan provides a long-term view of anticipated changes in demand for NIST's measurement and standards functions and competencies. The laboratory goals set out priority measurement and standards needs in four strategic focus areas: health care quality assurance; information / knowledge management; nanoscale measurements and data; and homeland security. NIST's recent budget requests reflect these long-term priority areas. NIST also responds to Administration and Congressional priorities expressed in any given year; in FY 03 these included several items related to NIST's strategic focus areas as well as other priorities. At the Operating Unit level, the NIST laboratories use a general set of criteria for prioritizing programs; these priorities are reviewed on an annual basis by the NIST Director.

Evidence: NIST 2010 Strategic Plan; NIST FY 03 Congressional budget request. NIST's long-term priorities have been reviewed by an independent advisory body, the VCAT; in its quarterly meetings the VCAT continues to monitor NIST's strategic direction and progress to plan. NIST also has made its long-term strategic plan available to other stakeholders and customers for comment. The annual report of the Visiting Committee on Advanced Technology is available at: www.nist.gov/director/vcat/index.htm. Guiding principles for laboratory programs and investment decisions were reviewed and amended by the Director and NIST Senior Management Board in April 2003. Annual laboratory operating unit plans are reviewed each year by the Director.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: On an annual basis, NIST collects high quality, comprehensive external evaluations of the laboratory programs from the National Research Council. This evaluation data is used at the division, lab, and Director's Office level to assess the quality, relevance, and performance of the laboratories' technical programs and make appropriate adjustments. NIST also makes numerous adjustments on an ongoing basis to its measurements and standards services, in response to external review and feedback from customers. Individual laboratories also have specific customer feedback channels that are used to evaluate and improve performance; examples include feedback from the Council for Optical Radiation Measurements in the Physics Laboratory, feedback from the Semiconductor Industry Association in the Electronics and Electrical Engineering Laboratory, etc. A number of economic impact studies also provide mid-stream analyses of research programs that NIST management uses to improve program performance.

Evidence: NRC feedback sessions and interactions during the external review process frequently include suggestions for improvements that are implemented at the division or laboratory level. NIST responses to these suggestions typically are reviewed in subsequent evaluation cycles; descriptions of this process and relevant examples can be found in the laboratory chapters of the annual NRC evaluation report. NIST also has implemented improvements in response to customer feedback, such as recent improvements in calibration service delivery processes; in general, such opportunities for continuous improvements are pursued on an on-going basis within the laboratories. Examples of mid-stream economic studies that provided evaluation data to program management include the Standard for the Exchange of Product Model Data (STEP) and Role-Based Access Control (RBAC) programs; for details, see appropriate studies at: www.nist.gov/director/planning/strategicplanning.htm#recent

Are Federal managers and program partners (grantees, subgrantees, contractors, cost-sharing partners, etc.) held accountable for cost, schedule and performance results?

Explanation: In addition to external review processes described above, NIST assures accountability for program results through an internal strategic management process that centers on Operating Unit alignment with NIST-wide long-term goals and strategic focus areas. The NIST Director reviews annual OU operating plans to evaluate operational alignment with the Institute's strategic plan, ensure that mechanisms are in place for coordinating strategic interdisciplinary work, and to review and evaluate OU-level resource allocation choices.

Evidence: The NIST Director reviews and evaluates the performance of each laboratory Operating Unit Director; the performance plans of each OU Director include elements related to NIST-wide programmatic and managerial goals.

Are all funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: NIST manages its resources carefully and typically has a limited amount of unobligated funds at year end. NIST's strong budget and accounting systems include rigorous internal reviews and external audits to ensure that funds are expended as intended.

Evidence: SF-132 (apportionment schedule) and SF-133 (report on budget execution). Internal processes include rigorous quarterly financial reviews. See the TA/NIST chapter in the audited Commerce FY 2002 Performance and Accountability Report (available at www.osec.doc.gov/bmi/budget/02commercereport.htm).

Does the program have procedures (e.g., competitive sourcing/cost comparisons, IT improvements, approporaite incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: Currently, NIST's performance plans do not contain measures or targets that could be used to demonstrate gained efficiencies or cost effectiveness. However, NIST does have a fairly flat organizational structure with a minimal number of managerial layers, and overhead costs have declined slightly over the years. NIST has also pursued competitive sourcing practices in keeping with the President's Management Agenda, but efforts have been delayed due to Congressional reporting requirements. NIST recently took two new steps to improve operational efficiency and effectiveness: it established a Business Council to develop improved managerial policies and processes for Institute overhead expenditures; and it established a Chief Information Office to help improve productivity and effectiveness throughout the laboratories and the Institute as a whole.

Evidence: NIST's competitive sourcing plans have been made available to the Department of Commerce and OMB. With regard to overhead efficiency, see answer to question 4.3. The office of the CIO was implemented too recently to attribute specific IT efficiency gains at this point in time. Once competitive sourcing studies have progressed and the office of CIO is more firmly established, NIST will have additional information that will be useful in demonstrating progress in this area.

Does the program collaborate and coordinate effectively with related programs?

Explanation: While the role of the NIST laboratories is distinct at the federal level, the laboratories accomplish their work through extensive collaboration with other National Measurement Institutes (NMIs), state offices, and diverse private organizations that play a role in the nation's measurement and standards system.

Evidence: Evidence includes Mutual Recognition Agreements with other NMIs; agreements with state weights and measures organizations; agreements with members of the National Conference of Standards Laboratories and NCSL-International; collaborations with ANSI and diverse standards development organizations; etc. The NIST laboratories also conduct a wide variety of work for other federal agencies on the basis of performance-based contracts. In some instances NIST's collaboration with other agencies is reflected in inter-agency MOUs, such as that recently signed by the DOC Technology Administration and the Department of Homeland Security.

Does the program use strong financial management practices?

Explanation: NIST has a long history of unqualified financial audits; in fact, NIST provides accounting services for several other DOC bureaus.

Evidence: See the TA/NIST chapter in the audited Commerce FY 2002 Performance and Accountability Report.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: In April 2002 NIST established a Senior Management Board and a set of supporting Councils with the overarching purpose of improving managerial efficiency and effectiveness. NIST uses these mechanisms for continuous improvement in NIST-wide managerial policies. Regular program oversight is obtained through the NIST Visiting Committee on Advanced Technology (VCAT), internal NIST program and planning reviews. In addition, the annual performance evaluations conducted by the National Research Council often identify management issues.

Evidence: The NIST Senior Management Board and supporting Councils have been used to improve policies that affect employee safety, performance-based management practices, leadership and management development, and other managerial areas. NIST also responds directly to program management recommendations produced by its external review mechanisms; for a description of the response process and for relevant examples, see the annual reports of the VCAT and NRC.

Does the program allocate funds through a competitive, merit-based process, or, if not, does it justify funding methods and document how quality is maintained?

Explanation: For base funding, resource choices generally are allocated on the basis of competitive merit and customer needs as assessed at the Operating Unit level and reviewed internally by the NIST Director (during annual planning and program reviews) and by external reviews of technical merit, relevance, and performance (conducted annually by the NRC). NIST-wide resource allocation processes to advance core metrology competencies are made annually on the basis of technical merit, and incorporate extensive internal and external review of each proposal. Program quality is further reinforced through rigorous technical peer review of research outputs; for instance, all technical publications must clear the Washington or Boulder Editorial Review Boards, and technical measurement capabilities and results are continually tested through international measurement comparisons.

Evidence: NIST's annual appropriation includes a very small quantity of earmarked funds. Descriptions of OU-level resource allocation processes and choices can be made available upon request; processes vary by lab given different research portfolios and customer requirements. NIST-wide resource allocation choices for "Competence" projects can be made available upon request. The results of external reviews are available from the NRC. Descriptions of quality management practices, such as editorial review boards, can be made available upon request.

Does competition encourage the participation of new/first-time performers through a fair and open application process?

Explanation:  

Evidence:  

Does the program adequately define appropriate termination points and other decision points?

Explanation:  

Evidence:  

If the program includes technology development or construction or operation of a facility, does the program clearly define deliverables and required capability/performance characteristics and appropriate, credible cost and schedule goals?

Explanation:  

Evidence:  

Has the program demonstrated adequate progress in achieving its long-term outcome performance goals?

Explanation: The NIST laboratories' long-term goals are inherently difficult to quantifiably measure due to such factors as the breadth of NIST's research porfolio, the long time horizons associated with research programs, and the difficulty of evaluating specific NIST contributions to end outcomes that have complex causal structures. NIST's annual peformance measures, external reviews and economic studies show continued progress, and the Institute's scientific and technical quality is widely recognized within the S&T community.

Evidence: External reviews by the National Research Council and numerous economic impact studies have identified and analyzed many of the economic and social impacts of NIST's laboratory research. Data are presented in DOC's budget justification and annual DOC performance plans and reports. Available data include detailed qualitative and quantitative findings from microeconomic impact assessments that, individually and collectively, confirm NIST's impact logic model and provide strong evidence of desired programmatic outcomes.

Does the program (including program partners) achieve its annual performance goals?

Explanation: NIST routinely meets or exceeds its annual quantitative performance targets and successfully passes rigorous annual qualitative reviews by the NRC. These targets and reviews collectively represent progress toward NIST's long term goals.

Evidence: See the FY 2004 Annual Performance Plan and the TA/NIST chapter in the audited Commerce FY 2002 Performance and Accountability Report. For annual evaluation of the laboratories' quality, relevance, and performance, see the annual report of the NRC Board on Assessment of NIST Programs.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program performance goals each year?

Explanation: It is inherently difficult to measure the overall productivity and efficiency of R&D programs, so there are limited measures available to quantifiably demonstrate improved efficiencies or cost effectiveness in meeting performance goals. External peer reviews provide a mechanism for identifying inefficiencies in laboratory programs. In terms of administrative efficiency, the NIST laboratories have a structural incentive to maximize efficiency as gains in administrative efficiency are realized as resources for research. NIST management has taken additional measures to maximize efficiency and effectiveness, as described in question 3.4.

Evidence: Some information is available to demonstrate efficiencies in select parts of the program. For example, between 1992 and 2002, laboratory overhead has been reduced from 13% to 10% of all laboratory obligations (e.g. appropriated resources, other agency contracts, and reimbursables). NIST also tracks other specific measures of efficiency, such as turn-around time for calibration services. NIST does not have an Institute-wide measure of research efficiency; however, OMB recognizes that R&D-performing organizations typically cannot provide unit cost measures of efficiency do to the long time frame for research, multivariate inputs, and diverse sets of outputs that derive from R&D activities.

Does the performance of this program compare favorably to other programs, including government, private, etc., that have similar purpose and goals?

Explanation: As discussed in section 1, there is not another entity within the U.S. with the same mission. NIST has attempted to collect data on other national measurement institutes, but little comparable information is available to provide meaningul comparisons.

Evidence:  

Do independent and quality evaluations of this program indicate that the program is effective and achieving results?

Explanation: External advisory groups and formal assessments by the National Research Council (NRC) Board on Assessment have found the NIST laboratories to be effective. The scope and nature of NIST's end outcomes are documented in a set of approximately 30 economic impact studies that have been conducted over the last ten years; these studies collectively validate NIST's impact logic model and provide additional independent evaluations and specific outcome-oriented measures of the programmatic effectiveness of the NIST laboratories.

Evidence: See the NRC Board of Assessment annual report and the annual report of the NIST VCAT. For NIST's economic impact studies, see: www.nist.gov/director/planning/strategicplanning.htm#recent

If the program includes construction of a facility, were program goals achieved within budgeted costs and established schedules?

Explanation:  

Evidence: