NSF - OLPA - Testimony of David A. Ucko, Ph.D., Head, Informal Science Education, Division of Elementary, Secondary, and Informal Education, Before the Senate Committee on Homeland Security and Governmental Affairs Subcommittee on Federal Financial Management, Government Information, and International Security, April 5, 2006

Chairman Coburn, Co-Chairman Carper, and distinguished members of the Subcommittee, thank you for the opportunity to describe the merit review process by which the National Science Foundation (NSF) makes available grant funds for museums. These institutions may compete for funds from programs throughout the agency. The Informal Science Education (ISE) program within NSF’s Directorate for Education and Human Resources (EHR), which I represent, is most closely aligned with the educational mission of science museums¹ and will be the focus for this testimony.

In addition, ISE investments have had a significant impact on the field of informal STEM education.

The National Science Foundation is funded at $5.58 billion for FY 2006. The NSF appropriation does not receive earmarks for museums or other institutions. At $63 million, the ISE program represents approximately 1 percent of the total NSF budget. Grants to museums make up approximately 40 percent of the total ISE dollars awarded each year. They typically fund development and implementation of innovative permanent and traveling exhibitions, after-school programs and other educational activities, Web sites, planetarium shows, and ancillary materials for classroom use. Museums and other types of organizations submit proposals in response to the ISE program solicitation (NSF 06-520), which emphasizes strategic impact, innovation, and collaboration. The ISE program invests in specific projects that develop and implement exhibitions and other educational deliverables. Awards are typically made for a three-to-five year duration, with a maximum total investment of $3 million in any project. About 40 percent of a given year's ISE funding goes towards 25 to 30 major new project grant awards based on the merit review process. The remaining program funds provide continuing grant increments for earlier multi-year awards, as well as a small number of competitive grants for planning, conferences, and educational outreach by NSF researchers.

The total dollar amounts awarded by the Foundation to museums each year since FY 2000 are summarized below.

Attachment A shows the sources of funds within NSF for museums. The Informal Science Education program provides 61% of the NSF funds awarded to museums; organizational units throughout the agency fund the remainder. Awards made by other programs within NSF’s Directorate for Education and Human Resource represent 13% of funds going to museums in support of formal K-12 education. For example, the Center for Learning and Teaching program funded a Center for Informal Learning and Schools based at San Francisco's Exploratorium that advances research and develops leaders at the interface between formal and informal learning. The Foundation’s research directorates represent about 26% of awards to museums. The Directorate for Biological Sciences, for instance, funds scientific research carried out by curators and scientists at natural history museums and botanical gardens.

Attachment B shows which museums have received grants in each of these years. In some cases, the grantee is a foundation associated with the museum or a university that serves as the parent organization. It should be noted that this table shows only the primary grantee institution. Since most projects involve multiple partner organizations, additional museums also receive funds through many of these awards, as well as through grants to other types of institutions as participating collaborators. As an example, the Queens Borough Public Library in New York received an award in partnership with three museums to embed science exhibits within a children's library. Principal museum awardees also typically disseminate or travel the results of the award to other museums across the nation. The Nanoscale Informal Science Education (NISE) Network out of the Museum of Science in Boston, for example, will create exhibits and other educational products on nanotechnology for some 100 museum sites.

In the ISE program, museums may apply for small amounts of additional funding as supplements to original awards in amounts up to 20 percent or $200,000, whichever is less. These supplements are intended either to protect the initial investment by ensuring completion of the original project scope if changes in conditions have occurred after the original award was made, or to take advantage of opportunities to extend further the project impact. Cost overruns are not grounds for awarding a supplement. In FY 2005, ISE awarded three such supplements to museums, with average funding of $164,163.

The ISE program does not fund operating support or capital expenses. Museums, like other NSF grantees, are eligible to recover approved indirect costs necessary to the general operation of the organization in support of the proposed project, such as certain administrative expenses. Organizations are required to support proposed indirect cost rates shown on their grant proposal budget by submission of a current indirect cost rate agreement negotiated with a Federal agency.

The ISE program holds competitions and corresponding review panels two times each year. All ISE awards are made competitively through the NSF merit review process and are based on the National Science Board criteria of intellectual merit and broader impacts. The following criteria are considered by reviewers in assessing ISE proposals:

Within intellectual merit, reviewers assess:

Deliverables. Does this project creatively "push the envelope" in enhancing informal science learning? Have the deliverables been selected and integrated to achieve the greatest project impacts? Are front-end and formative evaluation efforts adequate for their development? Are the scope and depth of STEM content appropriate to the target audience?

Project Design. Are the deliverables, project design, and timeline well developed and integrated to produce the specified impacts? Does the project design build on informal learning research and on lessons learned from prior efforts? Is the proposed budget reasonable and adequate? Does the proposal present meaningful strategies for managing potential risks?

Project Team. Is the team qualified to carry out the project? Do external advisors provide the expertise necessary to conduct the project, including relevant expertise based in informal science learning, STEM content, any media used, and evaluation?

Partnerships. Does the project fully take advantage of partnerships to enhance project impacts? Is there a credible strategy and plan for fostering or strengthening collaboration among the partners?

Within broader impacts, reviewers assess:

Audience. Is the primary target audience, as well as any secondary audience, clearly identified and segmented into subgroups as appropriate? Does the project demonstrate knowledge about the target audiences, their needs, and their interests?

Public Audiences. Will the project likely achieve a significant impact on the target audience of informal learners? Does the project maximize reach to audiences nationally, regionally, or community-wide? Does the proposal offer effective ways to reach nontraditional audiences and underrepresented groups?

Professional Audiences. Will the project likely achieve a significant impact on professionals in the field of informal science learning?

Impact Evaluation. Are there clear, appropriate measures and criteria for defining project success? Is there an appropriate summative evaluation plan for assessing impact? Is there an effective plan for broadly sharing project outcomes and findings?

Strategic Impact. Is the project likely to advance the field of informal science education in a significant way?

Results achieved through any prior NSF funding are considered in light of these review criteria. In each proposal, the principal investigator (PI), who serves as project director, is required to describe outcomes from previous NSF awards. New projects from prior grantees or resubmissions of previously declined proposals are subject to the same competitive review process as all other proposals.

Program officers facilitate merit review by forming diverse panels of experts with relevant knowledge and experience in informal science education, STEM content, evaluation, and areas specific to the type of proposal, such as exhibition design and production. All reviewers and panelists serve as volunteers. Selected panelists for each proposal submit written reviews with ratings (excellent, very good, good, fair, or poor) in advance of the panel meeting. After discussion at the panel meeting, reviewers then rank each proposal as high, medium, or low as a priority for funding, and a panel summary is written by an assigned panelist. After funding decisions are finalized, anonymous copies of the reviews and panel summaries are made available to principal investigators to indicate the panel's assessment of their projects and offer suggestions for improvement.

After the panel meetings, program officers meet as a group to determine which of the most highly rated proposals will be recommended for funding. Their goal is to create a diverse program portfolio, based on such factors as type of deliverable (e.g., exhibit, TV series, after-school program, Web) and target audience, in order to achieve the greatest national impact for the dollars invested.

The program officer recommendations and their rationale must then be approved by the Division Director. The official awards are then made by the NSF Division of Grants and Agreements, following review of the budget and financial capability of the awardee. The ISE program is highly competitive; the funding rate in FY 2005 was 17 percent.

Costs for running merit review panels, covered by program funds, are modest. The ISE program provides reimbursement for non-local panelists at a flat rate of $480 for each meeting day (typically three days for ISE panels) and $280 for each of two travel days. Reviewers residing in the local metropolitan area receive compensation of $280 for each full day of the meeting. These amounts are intended to cover all expenses, including lodging (as appropriate), taxis, parking, meals, and incidental expenses, except travel, which is paid directly. The following table shows the costs associated with holding ISE panels. These figures cover two sets of museum panels per year and currently represent about 1% of the funds awarded by ISE to museums annually. (The 2005 figure reflects greater use of external reviewers for preliminary proposal panels in response to an increased number of submitted proposals.)

As throughout NSF, each ISE program officer is responsible for ongoing post-award management for an assigned portfolio of awards. Significant changes in the original scope, project management, and budget must be submitted for review and approval by the cognizant program officer, and when required, the Division of Grants and Agreements.

Every grantee is required to submit an annual report describing progress of the project each year; award of the next annual continuing grant increment requires approval of the report by the cognizant program officer. Site visits also may be conducted by program officers to monitor progress and by the Division of Grants and Agreements to monitor financial aspects of awards.

Upon completion of the project, a final report is required that describes the outcomes of the award, including a summative evaluation of project impact carried out and written by an independent, third-party evaluator. The principal investigator for each ISE award is required to post these summative evaluations on the www.informalscience.org Web site so that others can learn from their experiences.

Every three to four years, each NSF program is reviewed by a Committee of Visitors (COV) consisting of outside independent experts. The ISE program recently underwent such a review. The complete report, entitled "Informal Science Education Program COV," and the "Response to ISE COV Report" by staff are posted for public access at http://www.nsf.gov/od/oia/activities/cov/covs.jsp#ehr under the heading "EHR 2005."

Regarding merit review, the COV reported that ISE's mechanisms were extremely well suited to the tasks; that the process was efficient; and as orchestrated by program officers, the process seemed strictly and appropriately (if not impressively) implemented to meet all ISE goals. It noted that panelists were appropriately balanced between scientists with content expertise and experienced practitioners within the informal science education field, and that program officers were very careful in meeting desired characteristics of geographic, institutional, gender, and ethnic diversity. In addition, the COV stated that individual reviews were, at most times, comprehensive and written in such a way as to provide helpful comments to assist proposers. In general, the COV stated that the overall quality of the reviews was strong and, in almost every case, addressed both merit review criteria.

The ISE program continually seeks ways to improve its efficiency and effectiveness. It has started to use Web conferencing as a cost-effective means to orient reviewers and panelists in advance, as well as to help prospective proposers become more familiar with the program and develop competitive proposals, especially from those states with fewest awardees. The ISE program is developing an online database to enable program officers to monitor awards and analyze the impact of the portfolio. The current solicitation calls for proposals to establish an ISE Resource Center, which will support continued professionalization and build capacity across the field; provide assistance to current and prospective project directors; and help assess the program.

The ISE program supports the development of a well-informed citizenry and a diverse, well-prepared workforce of scientists, engineers, mathematicians, and technicians, an effort that complements the President's recently announced American Competitiveness Initiative. This outcome is especially important to our nation today when science and technology play ever-increasing roles in our everyday lives, in local and national policy, and in a competitive global marketplace.

ATTACHMENT A: NSF MUSEUM FUNDING, FISCAL YEARS 2000-2005

(Dollars in Millions)

ATTACHMENT B: NSF MUSEUM AWARDS, FISCAL YEARS 2000-2005

(Dollars in Millions)

Museums receiving awards are listed below by year. Click on a number in the table above to see the recipients for that year or scroll through the lists below.
Note: lists include only the primary grantee institutions.

¹ The term "science museum" is used broadly here to describe awards to science-technology centers, natural history museums, children's museums, aquariums, zoos, planetariums, arboreta, nature centers, and similar types of institutions.
Return to testimony.