The Office of High-Energy and Nuclear Physics of the Office of Science (SC), U.S. Department of Energy (DOE), hereby announces its interest in receiving proposals for the department's Scientific Discovery through Advanced Computing Program (SciDAC). The goal of this program is to enable the use of terascale computers to dramatically extend our exploration of the fundamental processes of nature as well as to advance our ability to predict the behavior of a broad range of complex natural and engineered systems. This goal is to be achieved through the creation of scientific simulation codes that achieve high performance on a single node, scale to hundreds of nodes and thousands of processors, and have the potential to adapt over time and to be ported to future generations of high performance computers. Projects should address a problem of national scientific or engineering significance clearly related to the mission of DOE. They are expected to have high visibility and to present a long-term vision of how their work will fundamentally impact scientific discovery in specific areas of High-Energy Physics or Nuclear Physics research.

Background: Scientific Discovery through Advanced Computing

Advanced scientific computing will be a key contributor to scientific research in the 21st Century. Within the Office of Science (SC), scientific computing programs and facilities are already essential to progress in many areas of research critical to the nation. Major scientific challenges exist in all SC research programs that can best be addressed through advances in scientific supercomputing, e.g., designing materials with selected properties, elucidating the structure and function of proteins, understanding and controlling plasma turbulence, and designing new particle accelerators. To help ensure its missions are met, SC is bringing together advanced scientific computing and scientific research in an integrated program entitled "Scientific Discovery through Advanced Computing."

The Opportunity and the Challenge

Extraordinary advances in computing technology in the past decade have set the stage for a major advance in scientific computing. Within the next five to ten years, computers 1,000 times faster than today’s computers will become available. These advances herald a new era in scientific computing. Using such computers, it will be possible to dramatically extend our exploration of the fundamental processes of nature (e.g., the structure of matter from the most elementary particles to the building blocks of life) as well as advance our ability to predict the behavior of a broad range of complex natural and engineered systems (e.g., the earth’s climate or an automobile engine).

To exploit this opportunity, these computing advances must be translated into corresponding increases in the performance of the scientific codes used to model physical, chemical, and biological systems. This is a daunting problem. Current advances in computing technology are being driven by market forces in the commercial sector, not by scientific computing. Harnessing commercial computing technology for scientific research poses problems unlike those encountered in previous supercomputers, in magnitude as well as in kind. As noted in the 1998 report (See Footnote Number 1) from the NSF/DOE "National Workshop on Advanced Scientific Computing" and the 1999 report (See Footnote Number 2) from the President’s Information Technology Advisory Committee, this problem will only be solved by increased investments in computer software—in research and development of scientific simulation codes as well as on the mathematical and computing systems software that underlie these codes.

Investment Plan of the Office of Science

To meet the challenge posed by the new generation of terascale computers, SC will fund a set of coordinated investments as outlined in its long-range plan for scientific computing, Scientific Discovery through Advanced Computing (See Footnote Number 3), submitted to Congress on March 30, 2000. First, it will create a Scientific Computing Software Infrastructure that bridges the gap between the advanced computing technologies being developed by the computer industry and the scientific research programs sponsored by the Office of Science. Specifically, the SC effort proposes to:

• Create a new generation of Scientific Simulation Codes that take full advantage of the extraordinary computing capabilities of terascale computers.
• Create the Mathematical and Computing Systems Software to enable the Scientific Simulation Codes to effectively and efficiently use terascale computers.
• Create a Collaboratory Software Environment to enable geographically separated scientists to effectively work together as a team and to facilitate remote access to both facilities and data.

These activities are supported by a Scientific Computing Hardware Infrastructure that will be tailored to meet the needs of SC’s research programs. The Hardware Infrastructure is robust, to provide the stable computing resources needed by the scientific applications; agile, to respond to innovative advances in computer technology that impact scientific computing; and flexible, to allow the most appropriate and economical resources to be used to solve each class of problems.

Specifically, the SC proposes to support:

• A Flagship Computing Facility, the National Energy Research Scientific Computing Center (NERSC), to provide the robust, high-end computing resources needed by a broad range of scientific research programs.
• Topical Computing Facilities to provide computing resources tailored for specific scientific applications and to serve as the focal point for an application community as it strives to optimize its use of terascale computers.
• Experimental Computing Facilities to assess the promise of new computing technologies being developed by the computer industry for scientific applications.

Both sets of investments will create exciting opportunities for teams of researchers from laboratories and universities to create new revolutionary computing capabilities for scientific discovery.

The Scientific Computing Software Infrastructure, along with the upgrades to the hardware infrastructure, will enable laboratory and university researchers to solve the most challenging scientific problems faced by the Office of Science at a level of accuracy and detail never before achieved. These developments will have significant benefits to all of the government agencies that rely on high-performance scientific computing to achieve their mission goals as well as to the U.S. high-performance computing industry.

The Office of High Energy and Nuclear Physics supports a program of research into the fundamental nature of matter and energy. In carrying out this mission it

• Builds and operates large, world class charged-particle accelerator facilities for the nation and for the international scientific research community;
• Builds detectors and instruments designed to answer fundamental questions about the nature of matter and energy; and
• Carries out a program of scientific research based on experimental data, theoretical studies, and scientific simulation.

This Announcement is to solicit proposals to accelerate progress through the use of scientific simulation codes.

Computational modeling and simulation are among the most significant developments in the practice of scientific inquiry in the 20^th century. The coming advances in computing performance, if they can be realized for scientific problems, herald a new era in scientific computing. If computers capable of 100 teraflops or more become available in the next few years, it will be possible to dramatically extend our exploration of the fundamental processes of nature. It will also be possible to predict the behavior of a broad range of complex systems, such as charged-particle accelerator components, and eventually entire accelerators.

However, it is clear that the development of scientific codes that are capable of utilizing terascale computers efficiently and are adaptable, portable and re-usable is a massive undertaking that could take as long as 8-10 years to achieve its most ambitious scientific goals. This may require efforts of hundreds of person years of work.

It is also apparent that the most appropriate, cost-effective computing resources for scientific simulations vary significantly from application to application. Therefore much work is needed to understand the optimal configuration of computing hardware for each task and to design operating environments best able to foster significant scientific discoveries.

This solicitation is for proposals that articulate the long-term vision and potential for scientific progress through simulation, whilst laying out a concrete step-wise program of work and scientific research for the next 3 to 5 years.

The scope and complexity of the proposed projects will require close collaboration among researchers from computational and theoretical physics, computer science, and applied mathematics disciplines. Accordingly, this solicitation calls for the creation of scientific simulation teams, or collaborations, as the organizational basis for a successful application. A scientific simulation team is a multi-institutional, multi-disciplinary group of people who will :

• create scientific simulation codes that take full advantage of terascale computers,
• work closely with other SciDAC teams and centers to ensure that the best available mathematical algorithms and computer science methods are employed, and
• manage the work of the team in a way that will foster good communication and decision making (see section on Collaboration and Coordination below).

Partnerships among universities, national laboratories, and industry are encouraged.

Proposals are being sought in the broad topical areas listed below.

provide a major quantitative tool for simulations of quantum chromodynamics (QCD) on a lattice, which will :

provide crucial information in support of the experimental programs in high energy and nuclear physics

make accurate determinations of a number of fundamental quantities, such as the coupling constant that determines the strength of quark-gluon interactions, and the underlying masses of the quarks

explore the limitations, if any, of the "Standard Model" of particle interactions

explore how quarks and gluons provide the binding and spin of the nucleon

develop theoretical models of complex systems under extreme conditions, such as :

exploration of theoretical models of supernovae and comparison of the predictions with experimental results

study of the behavior of supersymmetric and other quantum field theories

Particular areas of interest include, but are not limited to:

Collaboratories link geographically dispersed researchers, data and tools, via high performance networks, to enable remote access to facilities, access to large datasets and shared environments. They enable geographically separated scientists to effectively work together as a team and facilitate remote access to both computing facilities and data.

As the size and complexity of high energy and nuclear physics experiments has increased so has the number and geographical dispersion of the researchers and the amount of data that must be collected, simulated and analyzed. Thus future experiments critically depend on the existence of such distributed hardware and software environments for their success. The scientific simulation applications that are the focus of this solicitation will also consist of geographically dispersed researchers, and will require high performance networks, to enable remote access to computing facilities, and multi-terabyte datasets.

Proposals for testbeds and collaborations across organizations that include network researchers, middleware developers and high energy and nuclear physicists are encouraged. They should be submitted in response to another Announcement, LAB 01-06 of the Office of Advanced Scientific Computing Research (ASCR). Copies should also be submitted to the Office of HENP.

Collaboration and Coordination

Preproposals

Potential proposers are strongly encouraged, but not required, to submit a brief preproposal consisting of two or three pages of narrative describing the research objectives, technical approaches and management plan. Each preproposal should include a cover sheet with the title of the project, project principal investigator, institutions involved, and their principal investigators and senior personnel. The name, telephone number, and e-mail address of each principal investigator should also be provided. In addition, brief, one-page curriculum vitae should be submitted for the principal investigators and other senior personnel involved. Preproposals will be evaluated to assess their programmatic relevance, and a response will be provided to the principal investigator within 14 days of receipt. However, notification of a successful preproposal is not an indication that an award will be made in response to a formal proposal.

Program Funding

It is anticipated that up to $5,500,000 of Fiscal Year 2001 funding will be available for project funding in FY 2001. Additional funding for the proposed project may be available through the Office of Advanced Scientific Computing Research for closely related research in computer science and/or applied mathematics. Proposals may request support for up to three years, with out-year support contingent on the availability of funds and satisfactory progress. To support multi-disciplinary, multi-institutional efforts, funding levels of up to $3.0 million per project may be requested, under this announcement, for the first year of the project. Requests for increased funding levels in future years will be entertained subject to availability of funds, progress of the funded activity, and programmatic needs.

The requested funding for the proposed work in computer science and applied mathematics should be included with the other project costs on the Budget Page. However, proposers are also requested to list the proposed computer science and applied mathematics costs separately in an appendix, as the Office of Advanced Scientific Computing Research may support this part of the work (up to 20-25% of the total project cost). The Office of High Energy and Nuclear Physics expects to fund three or four successful projects, depending on the size of the awards.

DATES: Preproposals referencing Program Announcement LAB 01-11 must be received by 4:30 P.M. EST, February 7, 2001. A response encouraging or discouraging the submission of a formal proposal will be communicated by e-mail within 14 days.

Formal proposals in response to this announcement should be received by 4:30 P.M., EST, March 15, 2001 to be accepted for merit review and funding in FY 2001.

Formal proposals referencing Program Announcement LAB 01-11, should be sent to: Dr. S. Peter Rosen, U.S. Department of Energy, Office of Science, Office of High Energy and Nuclear Physics, 19901 Germantown Road, Germantown, MD 20874-1290, ATTN: Program Announcement LAB 01-11. This address must be used when submitting proposals by U.S. Postal Service Express Mail or any commercial mail delivery service, or when hand-carried by the proposer. An original and seven copies of the proposal must be submitted. In addition, electronic copies in pdf file format of all proposal material are encouraged.

FOR FURTHER INFORMATION CONTACT: Dr. S. Peter Rosen, U.S. Department of Energy, Office of Science SC-20, 19901 Germantown Road, Germantown, MD 20874-1290. 

The instructions and format described below should be followed. Reference Program Announcement LAB 01-11 on all submissions and inquiries about the program.

Proposals from National Laboratories submitted to the Office of Science (SC) as a result of this program announcement will follow the Department of Energy Field Work Proposal process with additional information requested to allow for scientific/technical merit review. The following guidelines for content and format are intended to facilitate an understanding of the requirements necessary for SC to conduct a merit review of a proposal. Please follow the guidelines carefully, as deviations could be cause for declination of a proposal without merit review.

Proposals

Proposals will be subjected to scientific merit review (peer review) and will be evaluated against the following criteria listed in descending order of importance:

The evaluation of proposals under item 1, Scientific and Technical Merit, will pay particular attention to:

1. the potential of the proposed project to achieve a major advance in high energy and/or nuclear physics;
2. the potential of the proposed project to advance the state-of-the-art in computational modeling and simulation in areas pertinent to high-energy and nuclear physics research; 
3. the need for extraordinary computing resources to address problems of critical scientific importance to the high energy physics or nuclear physics program and the demonstrated abilities of the applicants to exploit terascale computers;
4. knowledge of and coupling to previous efforts in scientific simulation;
5. the extent to which the project incorporates broad community (industry/academia/other federal programs) interaction;
6. the extent to which the results of the project are likely to be extensible to other program or discipline areas; and
7. the importance of the proposed project to the mission of the Office of High Energy and Nuclear Physics and its impact on overall DOE goals.

The evaluation under item 2, Appropriateness of the Proposed Method or Approach, will also consider the following elements related to appropriateness of the proposed Scientific Computing Hardware Infrastructure to be used and of the quality of planning:

1. viability of the plan with respect to the scale and nature of current and future Computing Hardware Infrastructure needed;
2. clarity of the plan in detailing areas of work to be addressed by discipline scientists, computational scientists, applied mathematicians, computer scientists and computer programmers;
3. quality of the plan for effective collaboration among participants;
4. quality of the plan for ensuring communication with other advanced computation and simulation efforts;
5. viability of the plan for deploying the software and for assuring long-term maintenance, support, and re-use of the scientific codes and software infrastructure developed;
6. viability of the plan for verifying and validating the models developed, including verification using experiment results; and
7. quality and clarity of the proposed work schedule and project deliverables.

The evaluation will include program policy factors such as the relevance of the proposed research to the terms of the announcement and the agency's programmatic needs.

Note, that external peer reviewers are selected with regard to both their scientific expertise and the absence of conflict-of-interest issues. Non-federal reviewers may be used, and submission of a proposal constitutes agreement that this is acceptable to the investigator(s) and the submitting institution.

In addition, for this announcement, project descriptions must be 25 pages or less, including tables and figures, but excluding attachments. The proposal must also contain an abstract or project summary, letters of intent from all non-funded collaborators, and short curriculum vitae of all senior personnel.

Field Work Proposal (FWP) Format (Reference DOE Order 5700.7C)
Proposal Cover Page
Table of Contents
Abstract
Narrative
Literature Cited
Budget and Budget Explanation
Other support of investigators
Biographical Sketches
Description of facilities and resources
Appendix

An original and seven copies of the formal proposal/FWP must be submitted.

Proposals must be readily legible, when photocopied, and must conform to the following three requirements: the height of the letters must be no smaller than 10 point with at least 2 points of spacing between lines (leading); the type density must average no more than 17 characters per inch; the margins must be at least one-half inch on all sides. Figures, charts, tables, figure legends, etc., may include type smaller than these requirements so long as they are still fully legible.

3.1 Field Work Proposal Format (Reference DOE Order 5700.7C)

The Field Work Proposal (FWP) is to be prepared and submitted consistent with policies of the investigator's laboratory and the local DOE Operations Office. Additional information is also requested to allow for scientific/technical merit review. Laboratories may submit proposals directly to the SC Program office listed above. A copy should also be provided to the appropriate DOE operations office.

The following proposal cover page information may be placed on plain paper. No form is required.

Title of proposed project
SC Program announcement title
Name of laboratory
Name of principal investigator (PI)
Position title of PI
Mailing address of PI
Telephone of PI
Fax number of PI
Electronic mail address of PI
Name of official signing for laboratory*
Title of official
Fax number of official
Telephone of official
Electronic mail address of official
Requested funding for each year; total request
Use of human subjects in proposed project:

Provide the initial page number for each of the sections of the proposal. Number pages consecutively at the bottom of each page throughout the proposal. Start each major section at the top of a new page. Do not use unnumbered pages and do not use suffices, such as 5a, 5b.

Provide an abstract of no more than 250 words. Give the broad, long-term objectives and what the specific research proposed is intended to accomplish. State the hypotheses to be tested. Indicate how the proposed research addresses the SC scientific/technical area specifically described in this announcement.

The narrative comprises the research plan for the project and is limited to 25 pages. It should contain the following subsections:

Background and Significance: Briefly sketch the background leading to the present proposal, critically evaluate existing knowledge, and specifically identify the gaps which the project is intended to fill. State concisely the importance of the research described in the proposal. Explain the relevance of the project to the research needs identified by the Office of Science. Include references to relevant published literature, both to work of the investigators and to work done by other researchers.

Preliminary Studies: Use this section to provide an account of any preliminary studies that may be pertinent to the proposal. Include any other information that will help to establish the experience and competence of the investigators to pursue the proposed project. References to appropriate publications and manuscripts submitted or accepted for publication may be included.

Research Design and Methods: Describe the research design and the procedures to be used to accomplish the specific aims of the project. Describe new techniques and methodologies and explain the advantages over existing techniques and methodologies. As part of this section, provide a tentative sequence or timetable for the project.

Subcontract or Consortium Arrangements: If any portion of the project described under "Research Design and Methods" is to be done in collaboration with another institution, provide information on the institution and why it is to do the specific component of the project. Further information on any such arrangements is to be given in the sections "Budget and Budget Explanation", "Biographical Sketches", and "Description of Facilities and Resources".

List all references cited in the narrative. Limit citations to current literature relevant to the proposed research. Information about each reference should be sufficient for it to be located by a reviewer of the proposal.

A detailed budget is required for the entire project period, which normally will be three years, and for each fiscal year. It is preferred that DOE's budget page, Form 4620.1 be used for providing budget information*. Modifications of categories are permissible to comply with institutional practices, for example with regard to overhead costs.

A written justification of each budget item is to follow the budget pages. For personnel this should take the form of a one-sentence statement of the role of the person in the project. Provide a detailed justification of the need for each item of permanent equipment. Explain each of the other direct costs in sufficient detail for reviewers to be able to judge the appropriateness of the amount requested.

This information is required for senior personnel at the laboratory submitting the proposal and at all subcontracting institutions. The biographical sketch is limited to a maximum of two pages for each investigator

Describe briefly the facilities to be used for the conduct of the proposed research. Indicate the performance sites and describe pertinent capabilities, including support facilities (such as machine shops) that will be used during the project. List the most important equipment items already available for the project and their pertinent capabilities. Include this information for each subcontracting institution, if any.

Include collated sets of all appendix materials with each copy of the proposal. Do not use the appendix to circumvent the page limitations of the proposal. Information should be included that may not be easily accessible to a reviewer.

 Reviewers are not required to consider information in the Appendix, only that in the body of the proposal. Reviewers may not have time to read extensive appendix materials with the same care as they will read the proposal proper. 

The appendix may contain the following items: up to five publications, manuscripts (accepted for publication), abstracts, patents, or other printed materials directly relevant to this project, but not generally available to the scientific community; and letters from investigators at other institutions stating their agreement to participate in the project (do not include letters of endorsement of the project).

4. Detailed Instructions for the Budget
DOE Form 4620.1 "Budget Page" may be used with one Budget Page for each year of requested funding with one Budget Page for each year of requested funding. The requested funding for the proposed work in computer science and applied mathematics should be included with the other project costs on the Budget Page (DOE Form 4620.1). However, applicants are also requested to list the proposed computer science and applied mathematics costs separately in an appendix.

List the names of the principal investigator and other key personnel and the estimated number of person-months for which DOE funding is requested. Proposers should list the number of postdoctoral associates and other professional positions included in the proposal and indicate the number of full-time-equivalent (FTE) person-months and rate of pay (hourly, monthly or annually). For graduate and undergraduate students and all other personnel categories such as secretarial, clerical, technical, etc., show the total number of people needed in each job title and total salaries needed. Salaries requested must be consistent with the institution's regular practices. The budget explanation should define concisely the role of each position in the overall project.

DOE defines equipment as "an item of tangible personal property that has a useful life of more than two years and an acquisition cost of $5000 or more." Special purpose equipment means equipment which is used only for research, scientific or other technical activities. Items of needed equipment should be individually listed by description and estimated cost, including tax, and adequately justified. Allowable items ordinarily will be limited to scientific equipment that is not already available for the conduct of the work. General purpose office equipment normally will not be considered eligible for support.

The type and extent of travel and its relation to the research should be specified. Funds may be requested for attendance at meetings and conferences, other travel associated with the work and subsistence. In order to qualify for support, attendance at meetings or conferences must enhance the investigator's capability to perform the research, plan extensions of it, or disseminate its results. Consultant's travel costs also may be requested.

Foreign travel is any travel outside Canada and the United States and its territories and possessions. Foreign travel may be approved only if it is directly related to project objectives.

The budget should itemize other anticipated direct costs not included under the headings above, including materials and supplies, publication costs, computer services, and consultant services (which are discussed below). Other examples are: aircraft rental, space rental at research establishments away from the institution, minor building alterations, service charges, and fabrication of equipment or systems not available off-the-shelf. Reference books and periodicals may be charged to the project only if they are specifically related to the research.

The budget should indicate in general terms the type of required expendable materials and supplies with their estimated costs. The breakdown should be more detailed when the cost is substantial.

The budget may request funds for the costs of preparing and publishing the results of research, including costs of reports, reprints page charges, or other journal costs (except costs for prior or early publication), and necessary illustrations.

Anticipated consultant services should be justified and information furnished on each individual's expertise, primary organizational affiliation, daily compensation rate and number of days expected service. Consultant's travel costs should be listed separately under travel in the budget.

The cost of computer services, including computer-based retrieval of scientific and technical information, may be requested. A justification based on the established computer service rates should be included.

Subcontracts should be listed so that they can be properly evaluated. There should be an anticipated cost and an explanation of that cost for each subcontract. The total amount of each subcontract should also appear as a budget item.

Explain the basis for each overhead and indirect cost. Include the current rates.

_________________________

Posted at the Office of Science Grants and Contracts Web Site January 22, 2001.