Final Information Quality Bulletin for Peer Review

Word Version (269 KB)

HIGHLY INFLUENTIAL SCIENTIFIC DISSEMINATIONS

INFLUENTIAL SCIENTIFIC DISSEMINATIONS

October 5, 2005

TABLE OF CONTENTS

1. BACKGROUND AND PURPOSE OF THESE PROCEDURES

2. SCOPE AND APPLICABILITY OF PROCEDURES TO DOT ORGANIZATIONS

3. KEY PEER REVIEW PRINCIPLES

4. PROCEDURES FOR DETERMINING WHETHER INFORMATION IS COVERED BY THE OMB BULLETIN

5. PROCEDURES FOR MANAGING PEER REVIEWS OF PLANNED SCIENTIFIC INFORMATION DISSEMINATIONS

6. PROCEDURES FOR ESTABLISHING AND MAINTAINING AN ADMINISTRATIVE RECORD

7. PROCEDURES FOR CONDUCTING PEER REVIEWS

8. PROCEDURES FOR DEVELOPING PEER REVIEW PLANS

9. PROCEDURES FOR PUBLIC PARTICIPATION IN PEER REVIEWS

10. PROCEDURES FOR AVOIDING CONFLICTS OF INTEREST

11. PROCEDURES FOR ANNUAL REPORTING

12. PROCEDURES FOR PEER REVIEWS SUPPORTING RULEMAKING

13. BUDGETING FOR PEER REVIEW EXPENSES

14. DEFERRALS AND TO THE PEER REVIEW PROCESS

15. EXEMPTIONS TO THE PEER REVIEW PROCESS

APPENDIX A: KEY DEFINITIONS

APPENDIX B: LIST OF ACRONYMS AND ABBREVIATIONS

APPENDIX C: QUESTIONS AND ANSWERS REGARDING COMPLIANCE WITH OMB BULLETIN

APPENDIX D: SUMMARY TRACKING MATRIX USING FHWA SUBMISSIONS TO ACHIEVE COMPLIANCE TEMPLATE

APPENDIX E: SUBMISSION FORM AND PEER REVIEW PLAN

APPENDIX F: SAMPLE CONFLICT OF INTEREST FORM

APPENDIX G: PEER REVIEW CHECKLIST

1.     BACKGROUND AND PURPOSE OF THESE PROCEDURES

The following procedures are designed to facilitate the Department of Transportation's (DOT) implementation of the Office of Management and Budget's (OMB) Bulletin entitled, "Final Information Quality Bulletin for Peer Review." This Bulletin establishes government-wide guidance aimed at enhancing the practice of peer review of government science documents. Peer review is intended to increase the quality and credibility of the scientific information generated across the Federal government.

OMB recognizes that different types of peer review are appropriate for different types of scientific information. Under the Bulletin, Federal agencies, including DOT, are granted broad discretion to weigh the benefits and costs of using a particular peer review mechanism for a specific information product. The selection of an appropriate peer review mechanism for scientific information is left to agency discretion. Various types of information are exempted from the requirements of the Bulletin, including time-sensitive health and safety determinations, in order that peer review not unduly delay the release of urgent findings.

OMB's Bulletin is issued under the Information Quality Act (IQA) and OMB's general authorities to oversee the quality of Operating Administration (OA) information, analyses, and regulatory actions. In the Information Quality Act, Congress directed OMB to issue guidelines to "provide policy and procedural guidance to Federal agencies for ensuring and maximizing the quality, objectivity, utility, and integrity of information, (including statistical information) disseminated by Federal agencies. Pub. Law. No. 106-554-515(a). The Information Quality Act was developed as a supplement to the Paperwork Reduction Act, 44 U.S.C.& 3501 et seq., which requires OMB, among other things, to "develop and oversee the implementation of policies, principles, standards and guidelines to...apply to Federal OA dissemination of public information."

The DOT has designated the OA Chief Information Officer (CIO) as the senior official responsible for compliance with Section 515 of the Treasury and General Government Appropriations Act for Fiscal Year 2001 (PL 106-554). Under this Act, Federal agencies subject to the Paperwork Reduction Act (44 U.S.C. Chapter 35) implemented written guidelines to ensure and maximize the quality, utility, objectivity, and integrity of the information that they disseminate. DOT's IQA guidelines apply to a wide variety of substantive information dissemination activities in order to meet basic information quality standards set forth by Section 515.

The DOT acknowledges that ensuring the quality of information is an important management objective along with other Departmental objectives, such as ensuring the success of OA missions, observing the budget and resource priorities and restraints, and providing useful information to the public. The implementation of scientific peer review completes the DOT's organizational excellence objective by further ensuring quality scientific information.

Back to Table of Contents

2.  SCOPE AND APPLICABILITY OF PROCEDURES TO DOT ORGANIZATIONS

The procedures provided in this document apply to all operating administrations and organizations that are planning to disseminate to the public scientific information after June 16, 2005. Scientific information[1] means factual inputs, data, models, analyses, technical information, or scientific assessments related to such disciplines as the behavioral and social sciences, public health and medical sciences, life and earth sciences, engineering or physical sciences. This includes any communication or representation of knowledge such as facts or data, in any medium or form, including textual, numerical, graphic, cartographic, narrative, or audiovisual forms. There are two categories of scientific information that are covered by these procedures:

2.1.   Influential scientific information (ISI)

Influential scientific information (ISI) means scientific information that DOT can reasonably determine "will have or does have a clear and substantial impact on important public policies or private sector decisions."[2]

2.2.   Highly influential scientific assessments(HISAs)

Highly influential scientific assessments (HISAs) are those "that could have a potential impact of more than $500 million in any one year on either the public or private sector or that the dissemination is novel, controversial, precedent-setting, or has significant interagency interest."[3] A scientific assessment is an "evaluation of a body a scientific or technical information that typically synthesizes multiple factual inputs, data, models, assumptions, and/or applies best professional judgment to bridge uncertainties in the available information." Other criteria that can be used to determine whether a scientific work product is highly influential could include:

2.2.1.      Information focused on significant emerging scientific, technical, or regulatory issues.

2.2.2.      Information that involves the application of an innovative approach for a previously defined problem/process/methodology.

2.2.3.      Information that satisfies a statutory or other legal mandate for peer review.

2.2.4.      Guidance for conducting economic or financial analyses.

2.2.5.      New financial or economic methodologies that serve as the primary method or protocol for conducting economic analyses within a program or operating administration.

2.2.6.      Unique, novel, or previously un-validated applications of economic or financial assessment methodologies.

2.2.7.      Broad scale assessments of regulatory programs, such as those required by Congressional-mandates.

2.2.8.      Surveys of customer preferences.

2.2.9.      New national surveys of costs and expenditures for environmental protection.

2.2.10.  Economic research plans developed to assess and advance the state-of-the-art science in economic theory, methodologies, or modeling.

2.2.11.  New meta-analyses that re-analyze existing published literature and supporting data.

2.2.12.  New algorithms that will provide the basis of modeling and predicting traffic behavior.

2.2.13.  Large-scale data sources that have not gone through an OMB-approved peer review process in the past.

Examples of highly influential scientific assessments can be found at http://www.dot.gov or in Appendices D & E. Examples of influential scientific information will be available in the Docket Management System (DMS)[4] by October, 2005.

Back to Table of Contents

3.     KEY PEER REVIEW PRINCIPLES

There are various ways to conduct peer reviews of scientific information disseminations. Some may be internal and/or informal while others may be the formalized and rigorous reviews reserved for highly influential scientific assessments. The following principles, which are organized into four categories, are designed to promote the consistency and adequacy of peer reviews conducted for highly influential scientific assessments and influential scientific information:

3.1.   Accountability

3.1.1.      Each organization within DOT will have discretion for determining the type of peer review appropriate for influential scientific information and adapting procedures for the selection of peer reviewers.

3.1.2.      Scientific peer reviews will be coordinated by a supervisor at least one level removed from that at which the scientific information was produced.

3.1.3.      Selection of peer reviewers is the responsibility of the individual managing the peer review.

3.1.4.      Peer review processes and activities should be reviewed on an annual basis by each operating administration's chief information office.

3.1.5.      Public participation will be encouraged as a key component of the peer review process.

3.2.   Scope

3.2.1.      All influential and highly influential scientific information shall be peer reviewed prior to dissemination to the public.

3.2.2.      Peer review planning should take into account both benefit and costs.[5]

3.3.   Integrity

3.3.1.      Peer reviewers must be selected on the basis of three criteria: expertise, balance,[6] and independence.

3.3.2.      OAs shall ensure that their peer review practices are characterized by both scientific integrity and process integrity.[7]

3.3.3.      Reviews shall be conducted by scientific peers as demonstrated by their scientific achievements.

3.3.4.      Peer reviewers shall be independent and free of conflicts of interest.

3.3.5.      Peer reviewers should not be employed by OA or office producing the document.

3.4.   Transparency

3.4.1.      Responses to public comments on peer review plans and highly influential and influential scientific information will be provided in writing and provided to the DMS for posting.

3.4.2.      The comments and statements obtained in a review will be made available to the investigator or author, and responses by the investigator or author about how each comment was addressed are to be maintained in project and administrative files for at least three years after the completion of a project or the release of products, or such other period of time as is judged to be reasonable.

3.4.3.      Disclosure and attribution for peer reviews is encouraged within legally-specified protections.[8] Specifically, to the extent to which information about a reviewer (name, credential, affiliation) will be disclosed along with his/her comments or analysis, the agency must comply with the requirements of the Privacy Act, 5 U.S.C. 552a, as amended, and OMB Circular A-130, Appendix I, 61 Fed. Reg. 6428 (February 20, 1996) to establish appropriate routine uses in a published Systems of Records Notice.

Back to Table of Contents

4.     PROCEDURES FOR DETERMINING WHETHER INFORMATION IS COVERED BY THE OMB BULLETIN

Effective compliance with OMB Bulletin depends on each OA developing mechanisms that enable the comprehensive identification of scientific information disseminations that are either of a highly influential or influential nature.

4.1.

DOT OAs shall establish a formal process for reviewing research and program activities, projects, and programs to determine if they will result in the dissemination of HISAs or ISIs in the "foreseeable" future.

4.2.

DOT OAs shall institute guidance for classifying planned information disseminations that includes real world examples of what types of information and activities would be properly classified as HISAs or ISI.

4.3.

OA Guidance shall include examples of what types of information provided by the OA would be considered exempted or deferred under OMB requirements and the rationale for exempting or deferring them. These examples may come from prior experience or the experiences of other departments and agencies across government.

4.4.

OA Guidance shall include information on how an administrative record will be maintained within the operating administration.

4.5.

Each OA shall establish a committee to periodically assess whether the scientific information review process is working in a manner consistent with the intent of the OMB Guidance and DOT implementation procedures.

Back to Table of Contents

5.     PROCEDURES FOR MANAGING PEER REVIEWS OF PLANNED SCIENTIFIC INFORMATION DISSEMINATIONS

Administrators are responsible for ensuring proper implementation of Departmental guidelines. The Information Quality Peer Review Bulletin falls under the scope of Section 515/IQA giving the CIO additional responsibility to post highly influential assessments and influential scientific information, along with Peer Review Plans, on the Department of Transportation website: http://www.dot.gov. The process for identifying, reviewing and submitting scientific information is the responsibility of each OA. The minimum requirements for this process are as follows:

5.1.

Each OA will designate a Peer Review Coordinator (PRC) to coordinate and monitor peer review activities across the OA. The PRC is the main contact for each OA on peer review matters. The responsibilities of the PRC include:

5.1.1.      Overseeing the tracking of submissions (See Appendix D for template).

5.1.2.      Reporting peer review activities to OA leadership.

5.1.3.      Mediating compliance issues involving implementation of the OMB Bulletin.

5.1.4.      Overseeing development of procedures pertaining to the peer review administrative record.

5.1.5.      Attending all meetings convened by S-80/C-50[9] pertaining to the implementation of the OMB Peer Review Bulletin.

5.1.6.      Communicating changes and additions in peer review guidance to their respective OA management and staff.

5.1.7.      Overseeing the peer review process to ensure effective and timely compliance with the OMB Bulletin.

5.1.8.      Identifying and recommending enhancements in operating procedures to OA management and S-80/C-50.

5.1.9.      Coordinating with Data Quality Contacts in other OAs.

5.1.10.  Provide advice, guidance, and support to the Peer Review Leader.

5.1.11.  Work with the OA to identify major scientific and technical work products as ISI and HISA peer review candidates for the coming year.

5.1.12.  Designating (in conjunction with the Project Managers (s) and/or the Contracting Officer Technical Representative (COTR) a Peer Review Leader (PRL) for each peer review. The responsibilities of the PRL include:

1. Keeping the PRC informed of the status of the peer review project and notifying the PRC that the peer review is completed.
2. Providing advice, guidance, and support in the preparation, implementation, and completion of the peer review.
3. Reviewing working and final drafts, statements of work, charge, and procurement requests.
4. Ensuring that sufficient funds are designated in the office's budget request to conduct the peer review; also ensuring that adequate resources and or/extramural management support are available for the peer review.
5. Assisting in establishing a realistic peer review schedule of key deadlines.
6. Designating the stage(s) of product development where peer review is appropriate.
7. Ensuring issues and comments raised by the peer reviewer(s) are adequately addressed and documented for the record, and where appropriate, incorporated into the work product that is used as basis for decision-making.
8. Identifying and providing a rationale for the administrative record and the annual report of each decision not to peer review particular information dissemination.
9. Ensuring that peer review agenda submissions to the DOT-CIO are up-to-dated at least every six months.
10. Helping to identify peer reviewers in consultation with researchers and COTRs in accordance with guidance provided in the OMB Bulletin.
11. Advising peer reviewers of their responsibilities.
12. Notifying the peer reviewers, information submitters, and other interested parties that information regarding a peer review is subject to release under the Freedom of Information Act (FOIA), unless the peer review information meets the criteria for an exemption under the FOIA.
13. Providing information to the PRC and others in the concurrence chain regarding the peer review process and results, including the charge, profile of the peer reviewers, the peer review comments, and a proposal on how to address the comments.
14. Submitting the results to the DOT-CIO for submission in the annual peer review report.
15. Maintaining the OA-level peer review record in a manner consistent with OMB guidelines and Departmental procedures regarding the development and maintenance of the administrative record.
16. Ensure that all documents provided to Non-Federal peer reviewers are screened for information that could be claimed as Confidential Business Information (CBI).

5.2.

The procedures for processing peer review submission (agenda) and peer review plans are as follows:

5.2.1.      Peer review agenda entries should be tracked. Note: Appendix D provides an example of a tracking sheet that can be adapted for OA use.

5.2.2.      Peer review plans must include all elements called for in Paragraph V. (2) of the OMB peer review Bulletin. See Appendix E---Peer Review Plan Template.

5.2.3.      Each DOT organization preparing a peer review agenda entry or peer review plan must, after completing internal coordination required by its respective operating administration, transmit the document to S-80 and C-50 in the Office of the Secretary of Transportation for review.

5.2.4.      After all S-80/C-50 comments are addressed by the OA, S-80 will post the agenda entry and plan on other DOT peer review web page.

5.2.5.      DOT's peer review web page will be updated every six months and on an as-needed basis for highly influential scientific disseminations.

5.2.6.      The official posting dates for the Peer Review Agenda for highly influential information is June 16th.

5.2.7.      The official posting date for the Peer Review Agenda for influential scientific information is December 16th.

5.2.8.      Material intended for each June's agenda setting should be transmitted to S-80/C-50 by April 15, and for the December posting date, by October 15^th.

5.2.9.      Beginning with the December 2005 agenda, all influential, as well as highly influential

Back to Table of Contents

6.     PROCEDURES FOR ESTABLISHING AND MAINTAINING AN ADMINISTRATIVE RECORD

Each OA shall document peer review activities for both influential scientific information and highly influential scientific assessments through the DOT Docket Management System that is currently used to support DOT regulatory and adjudicatory actions.

6.1.

Whenever a project that is likely to produce highly influential or influential scientific information becomes active, or when there is a document ready for peer review and the peer review process is beginning, the OA PRC shall establish an official docket[10] within the DMS. The following types of information will be included in each docket:

6.1.1.      Draft work product of highly influential or influential scientific information submitted for peer review.

6.1.2.      Materials and information, including the charge, given to the peer reviewer(s).

6.1.3.      Written comments, information, and materials received from the peer reviewer(s) and the public.

6.1.4.      Information about the peer reviewers (names, affiliations, statement concerning potential conflicts and their resolutions).

6.1.5.      Logistical information about the conduct of the peer review (times and locations of meetings, etc.).

6.1.6.      A memorandum or written record responding to the peer review and public comments specifying acceptance or, where thought appropriate, rebuttal and non-acceptance.

6.1.7.      The peer review record is not complete until it contains a copy of the final work product that addresses the peer review comments.

6.1.8.      If the OA relies on influential scientific information or a highly influential scientific assessment subject to the requirements of the OMB Bulletin in support of a regulatory action, the OA shall include in the administrative record for that action documentation that explains how the agency has complied with the requirements of the Bulletin and the Information Quality Act. Relevant materials will be placed in the administrative record.

6.2.

Once a docket is established by the OA through the PRC, it will be used to:

6.2.1.      Post results of peer review activities.

6.2.2.      Post public comments on peer review plans.

6.2.3.      Post public comments on proposed disseminations of highly influential and influential scientific information.

6.2.4.      Track peer review activities.

6.2.5.      Track OA responses to peer review comments.

6.3.

It is the responsibility of each OA PRC to inform peer reviewers and the public how the DMS operates and how it will be used to facilitate the peer review process from beginning to end.

Back to Table of Contents

7.     PROCEDURES FOR CONDUCTING PEER REVIEWS

Peer review is used to ensure the quality of published information meets the standards of scientific and technical community. Peer review typically evaluates the clarity of hypotheses, the validity or research design, the quality of data collection procedures, the robustness of the methods employed, the appropriateness of the methods for the hypotheses being tested, the extent to which conclusions follow from the analysis, and the strength and limitations of the overall product.

7.1.

Procedures that apply to peer reviews of both highly influential and influential scientific information.

7.1.1.      The first step in the peer review process is to develop the "charge" that contains the instructions to the peer reviewers regarding the objective of the peer review and the specific advice sought. Every peer review must begin with a specific written charge. This charge is to be developed prior to selecting reviewers.

a.   Elements of the charge are:

1)       Brief overview or introduction (describe what work product is, how it was developed, how it will be used).

2)       Brief description or listing of any background materials provided to the peer reviewers.

3)       The issues or questions to be addressed by the peer reviewers.

4)       The due date of reviewer's comments.

5)       The format of reviewer responses.

6)       The point of contact in case peer reviewers have questions.

b.   In cases where the OA selects a contractor to perform peer review, the OA cannot submit the charge directly to the peer reviewers under contracting regulations. The OA can provide the contractor with a list of questions that the OA wants included in the charge letter. The list of charge questions can be incorporated into the Statement of Work by the OA. Based on this list, the contactor prepares and submits the charge letter to the peer reviewers, after the OA is given the opportunity to review the charge letter.

7.2.

Draft influential or highly influential information presented at scientific meetings or shared confidentially with colleagues for scientific input prior to or in tandem with a formal peer review shall include the disclaimer:

"THE FINDINGS AND CONCLUSIONS IN THIS REPORT (PRESENTATION) HAVE NOT BEEN FORMALLY DISSEMINATED BY (THE OA) AND SHOULD NOT BE CONSTRUED TO REPRESENT ANY OA [or DEPARMENTAL] DETERMINATION OR POLICY."

7.3.

Procedures for the determining the scope of peer reviews for influential scientific information.

7.3.1.      For influential scientific information, the OA determines the nature and scope of the peer review.

7.3.2.      Adequate peer reviews could include having the document reviewed by a referred scientific journal or a formal letter review by three or more reviewers outside the OA, but from another OA within DOT. The peer review might also involve convening a formal panel comprised of experts from other government agencies, universities, or the private sector to discuss and review the information.

7.3.3.      For peer reviews of influential scientific information, the OA may select reviewers outside the OA but still within the DOT.

7.3.4.      The OA shall disclose the names of the peer reviewers and their organizational affiliations in the report.

7.3.5.      Reviewers shall be notified in advance regarding the extent of disclosure and attribution planned by the OA.

7.3.6.      The final peer review report shall be posted on the DOT docket along with all materials related to the peer review (charge, peer review report, and OA response).

7.3.7.      The peer review report shall be discussed in the preamble to any related rulemaking and included in the administrative record for any related OA action.

7.4.

Procedures for determining the scope of peer reviews of highly influential scientific assessments

7.4.1.      The OMB Bulletin establishes a more rigorous set of minimum review requirements for scientific information that falls into the category of "highly influential scientific assessments." These documents are typically highly complex, multidisciplinary, and often, they are novel or precedent setting. The peer review planning review process should begin at the start of the research project.

7.4.2.      Peer reviews of highly influential scientific assessments should generally be conducted through a standing panel with a minimum of three members.

7.4.3.      Special care must be given to select panel members that have requisite expertise to review the various disciplines reflected in the highly influential scientific assessment.

7.4.4.      Reviewers must be independent of the DOT.

7.4.5.      Special Government employees, such as academics appointed to advisory committees can serve as reviewers.

7.4.6.      In those rare cases where expertise cannot be found outside the Department, an OA may use expertise that is found within the Department but outside the sponsoring OA.

7.4.7.      The OA must document the reasons for selecting a reviewer from within the OA.

7.4.8.      Repeated use of same reviewer (or reviewers) must be avoided unless it can be documented that the needed expertise cannot be found elsewhere.

7.4.9.      OA shall rotate membership in standing peer review committees across the available pool of qualified reviewers to promote intellectual diversity.

7.4.10.  OAs conducting peer reviews shall provide reviewers with sufficient background information, including access to key studies, data and models, to perform their role as peer reviewers.

7.4.11.  Research grants awarded to a potential reviewer do not automatically disqualify the individual from conducting peer reviews.

7.4.12.  The OA must consider whether a potential reviewer's consulting and contractual relationships with the OA might raise issues of perceived or actual conflicts of interest.

7.5.

Procedures for Preparing the Peer Review Report

7.5.1.      Peer reviewers must prepare a report that includes the charge to the peer reviewers, describes the nature and scope of their reviews and their findings and conclusions. a verbatim copy of each reviewer's comments (with or without specific attributions).

7.5.2.      The report should also document the process for selecting reviewers and why these reviewers were selected to conduct the peer review.

7.5.3.      The OA shall prepare a written response to the peer review report explaining (a) the OA's agreement or disagreement with the views expressed in the report, (b) the actions the OA has undertaken or will undertake in response to the report, and (c) the reasons the OA believes those actions satisfy the key concerns stated in the report (if applicable).

7.5.4.      The peer review report shall be discussed in the preamble to any related rulemaking and included in the administrative record for any OA action.

7.5.5.      The DOT is required to disseminate the peer review report and the Department's response to the report on the DOT's website, including all materials related to the peer review such as the "charge" settlement, peer review report, and the OA response to the review.

7.5.6.      The peer review report will include information on the slate of reviewers and the substance of their comments while avoiding specific attributions to particular reviewers.

7.6. Responsibilities of Peer Reviewers

Peer reviewers shall be willing participants and have a level of technical or scientific knowledge commensurate with that of the scientific/technical dissemination developers. They need to read all materials, participate fully, and protect confidential information that arises. Peer reviewers must be fully informed in advance as to all disclosure requirements of the peer review process.

Peer reviewers are to:

7.6.1.      Maintain the confidentiality of all products and deliberations.

7.6.2.      Perform reviews in a timely fashion.

7.6.3.      Maintain the highest standards of objectivity.

7.6.4.      Rescue themselves in cases where unexpected conflicts of interest arise.

7.6.5.      To assist OAs conduct peer reviews, the DOT Office of Chief Information Office (S-80) and Office of General Counsel (C-50) will develop and maintain a list of potential external and internal contacts organized in terms of technical expertise. This information will be maintained in strict accordance with Federal statutes and regulations governing privacy protections. Individuals on this list can be employed directly in the peer review process. Alternatively, they can identify other technical experts that may be more appropriate for a particular peer review project.

7.7.   Alternative Procedures

7.7.1.      In addition to developing a peer review process in accordance with OMB requirements and DOT implementing guidance, DOT OAs have several alternatives at their disposal:

a.   Rely on scientific information produced by the National Academy of Sciences (NAS).

b.   Commission the NAS to conduct the peer review.

c.   Employ an alternative procedure or set of procedures approved by the OIRA Administrator in consultation with the Office of Science and Technology Policy (OSTP).

Back to Table of Contents

8.     PROCEDURES FOR DEVELOPING PEER REVIEW PLANS

DOT OA's are required to begin a systematic process for peer review planning for all influential and highly influential information that the OA plans to disseminate in the "foreseeable" future.

8.1.

A key component of this planning process is a web-accessible listing of forthcoming influential and highly influential disseminations that together comprise an OA "agenda."

Each agenda item shall include the following items:

8.1.1.      A preliminary title for the planned dissemination.

8.1.2.      A short paragraph describing the subject and purpose of the dissemination.

8.1.3.      An OA contact person.

8.1.4.      A prediction of whether the information dissemination will be of an influential or highly influential nature.

8.1.5.      A designation of the type of peer review that will be undertaken.

8.1.6.      A draft schedule for the peer review with the notation that the schedule is subject to change.

8.1.7.      Specific deferrals or waivers that have been applied for.

8.1.8.      Any OMB "sanctioned" alternative peer review procedures that will be used.

8.2.

Developing the "Charge"

The "Charge" contains the instructions to the peer reviewers regarding the objectives of the peer review process. Each peer review request shall contain specific instructions to the peer reviewers regarding what is expected in terms of their review such as level of technical detail, etc. The "Charge" should be included in the Peer Review Plan.

8.3.

Each Peer Review Plan shall include the following elements:

(Please note that Appendix E provides a reporting template for the following categories of information)

8.3.1.      Contents

a.   A paragraph including the title, subject, and purpose of the planned report.

b.   A statement as to whether the dissemination is likely to be influential scientific information or a highly influential scientific assessment

c.   Schedule (timing) for peer review including deferrals.

d.   Structure of review (panel or individual letters) or whether alternative procedure will be exercised.

e.   The "Charge" to peer reviewers.

f.     Whether there will be opportunities for public to comment on the work product to be peer reviewed, and if so, how and when will these opportunities will be provided.

g.   Whether OA will provide significant and relevant comments to peer reviewers before they conduct their peer review.

h.   Anticipated number of reviewers 3 or fewer, 4-10, or more than 10.

i.     Description of the primary disciplines or expertise needed for reviews.

j.     Whether reviewers will be selected by the OA or by designated outside organization.

k.   Whether the public, including scientific or professional societies, will be asked to nominate reviewers.

l.     OA contact to whom inquiries may be directed to learn the specifics of the plan.

m. Links between plan and documentsmade public pursuant to the OMB Bulletin must be provided on the DOT web site along with public comment forms.

8.3.2.      Schedule

a.   For highly influential scientific disseminations, the peer review agenda should be updated as soon as a likely dissemination timeline is known.

b.   OAs should update their peer review agenda at least every six months for influential scientific information.

c.   It is strongly recommended that agendas be revised and schedules be updated as soon as additional information becomes available.

d.   OAs shall consider public comments on their peer review plans.

e.   OAs should provide a "list-serve" or similar mechanism to update the public via e-mail every time the peer review agenda has been updated.

Back to Table of Contents

9.     PROCEDURES FOR PUBLIC PARTICIPATION IN PEER REVIEWS

The OMB Bulletin requires that OAs shall, whenever feasible, provide for public participation. Public participation in the peer review process will vary in accordance with the type of scientific information to be disseminated.

In some cases, it may be appropriate for the scientific peer review process to precede an official dissemination so that the public receives the most scientifically sound product. Other times, it may be appropriate for an OA to make a draft assessment publicly available (i.e., a request for comment) at the onset of the peer review process. If an OA decides to release a product to the public prior to the completion of the peer review, the following procedures shall be followed:

9.1.

A process shall be provided for the public to submit either written comments or to make an oral presentation before a peer reviewer panel, or both.

9.1.1.      The public shall be provided information on how they can submit comments through the DOT DMS (See Section 6.0).

9.2.

When written comments from the public are received, the OA shall ensure that peer reviewers receive copies of public comments that address significant scientific issues.

9.3.

Peer reviewers shall be given sufficient time to fully consider all public comments.

9.4.

To avoid undue delay of OA activities, time limits for public participation in the peer review process shall be established and communicated to the public.

9.5.

OA may publish a "request for comment" or other notice in which public comment is solicited prior to a panel of peer reviewers commencing its work.

Back to Table of Contents

10. PROCEDURES FOR AVOIDING CONFLICTS OF INTEREST

The NAS defines "conflicts of interest" as any financial or other interest that conflicts with the service of an individual on the review panel because it could impair the individual's objectivity or could create an unfair competitive advantage for a person or organization. Department OAs will use the following procedures to avoid conflicts of interest:

10.1.

Reviewers of influential scientific information generally should not be employed by the OA or office producing the document.

10.2.

Employees of the DOT, or negotiating, or seeking employment with the DOT, are not permitted to be reviewers of highly influential scientific assessments. However, special government employees are not precluded from participating on peer reviews nor are academics appointed to OA Advisory Committees or scientific review groups.

10.3.

OAs shall make an effort to rotate peer responsibilities across the available pool of qualified reviewers.

10.4.

OAs will make a special effort to examine potential reviewers' financial conflicts, including significant financial investments, consulting arrangements, employer affiliations, grants, and contracts. Conflicts of interest may be real or apparent.

10.5.

Financial ties are to be scrutinized when the information is relevant to regulatory policy.

10.6.

Inquiries into potential conflicts must go beyond financial investments and business relationships and shall include work as an expert witnesses, family relationships, consulting arrangements, honoraria, and sources of grants and contracts.

10.7.

OAs shall consult Federal ethics requirements, applicable standards issued by the Office of Government Ethics, and the prevailing practices of the NAS.

10.8.

Peer reviewers that are Federal Employees (including special government employees) are subject to Federal requirements governing conflicts of interest.

10.9.

For non-Federal employees, the DOT shall adopt or adapt NAS policy for committee selection with respect to evaluating conflicts of interest.[11]

10.10.

    Under some conditions, conflicts are unavoidable. In those special circumstances, the DOT OA will publicly disclose those conflicts.

10.11.

The OA shall inform potential peer reviewers of any planned public disclosures (comment attributions) at the time they are recruited and obtain written permissions.

10.12.

DOT OAs conducting or overseeing peer reviews must ensure that proper documentation is received from nominated peer reviewers regarding the presence or absence of perceived or actual conflicts of interest.

10.13.

Reviewers must certify in writing that they will maintain the confidentiality of the review and associated materials and that they will not disclose to another individual any matter relating to the project.

10.14.

Conflict of Interest Waivers -- If no other reviewer is available with the expertise necessary to ensure competent reviews, a waiver may be granted to allow participation by an individual that may have a perceived or actual conflict of interest.

10.14.1.

Influential Scientific Information Waivers --- with documented explanations ---may be granted by an OA Administrator.

10.14.2.

Highly Influential scientific Disseminations Waivers --- with documented explanations --- may be granted by the Secretary or Deputy Secretary.

Back to Table of Contents

11. PROCEDURES FOR ANNUAL REPORTING

The DOT CIO shall prepare an annual report that summarizes key decisions made pursuant to the OMB Bulletin. The report, which is due December 15^th of each year, must include a summary of the peer reviews conducted by the agency during the fiscal year. The report should include the following items:

11.1.

Number of peer reviews conducted subject to the Bulletin (i.e., for influential scientific information and highly influential scientific assessments)

11.2.

Number of times alternative procedures were invoked.

11.3.

Number of times waivers or deferrals that were invoked.

11.4.

The length of time elapsed between the deferral and the peer review.

11.5.

Decisions to appoint reviewers pursuant to any exception to the applicable independence or conflict of interest standards of the OMB Bulletin.

11.6.

Number of peer review panels that were conducted in public and the number that allowed public comment.

11.7.

The number of public comments provided on the OA's peer review plans.

11.8.

The numbers of peer reviewers that the OA used that were recommended by professional societies.

Back to Table of Contents

12. PROCEDURES FOR PEER REVIEWS SUPPORTING RULEMAKING

Although the OMB Bulletin does not require a peer review to be conducted at a specific time during the regulatory rulemaking process, it does state, "It is most useful to consult with peers early in the process of producing information." It also notes that, when the information is a critical component of rulemaking, it is important to complete the peer review before the OA announces regulatory options.

OAs may decide that peer review should precede an opportunity for public comment in situations where parallel "public participation in peer review is an important aspect of obtaining a high-quality product."

If the OA relies on ISI or HISA that is subject to the OMB peer review requirements, in support of a rulemaking, the OA shall provide a certification in the administrative record that explains how the requirements of the Bulletin were met. Relevant materials will be placed in the administrative record.

Back to Table of Contents

13. BUDGETING FOR PEER REVIEW EXPENSES

The conduct of peer reviews will add administrative costs to scientific disseminations. These costs of peer review include both the direct costs of the peer review activity and those indirect costs stemming from the potential delay in government and private actions that can result from peer review. On the other hand, peer review, if performed fairly and rigorously, can build consensus among stakeholders and reduce the temptation for courts and legislators to second-guess or overturn OA actions. Moreover, comprehensive stakeholder review can be an effective mechanism for enhancing that rate at which scientific information and technologies are deployed into the market place.

13.1.

In all cases, OAs should use informal or formal cost-benefit methods to determine the appropriate scope and character of peer reviews. In this respect, the OMB Bulletin allows significant discretion regarding peer reviews involving influential scientific information.

13.2.

Resources needed to implement the OMB specified peer review process are to be requested through the usual Departmental and OA budgetary processes. For purposes of budget planning, the costs of peer review include:

13.2.1.  Additional FTE costs of staff dedicated to peer review process.

13.2.2.  Contract or other costs associated with the use of outside peer reviews.

13.2.3.  Administrative costs associated with conducting reviews (including travel expenses).

Note: Expenses pertaining to peer reviews can also be reduced by effectively using tracking tools available to OAs, including the DMS discussed Section 6.0. The DMS will enable OAs to track peer review activities and engage the public without the burden of developing an electronic peer review tracking system.

Back to Table of Contents

14. PEER REVIEW PROCESS DEFERRALS AND WAIVERS

The OMB Bulletin allows for a deferral or waiver of peer review requirements specified in the Bulletin where necessary.

Influential Scientific Information: The OA Administrators may waive or defer some or all of the peer review requirements for influential scientific information if there exists a compelling rationale. In such cases, the requesting program manager shall issue a memo through the appropriate Associate Administrator (or the equivalent) and Chief Counsel to the OA Administrator requesting a deferral or waiver. This memo will include information on the necessity of the deferral or waiver. When deferrals are granted by the OA Administrator, peer reviews should be conducted as soon as practicable thereafter. An estimated schedule for completing the peer review shall also be attached to the memo requesting a deferral.

If approved by the OA Administrator, written notification (including the planned completion schedule) shall be provided to the Director of OMB.

Highly Influential Scientific Disseminations: For waivers or deferrals involving highly influential scientific disseminations, the OA Administrator shall issue a memo through the OST/CIO and OST Assistant General Counsel for Regulation and Enforcement to the Secretary (or Deputy Secretary) requesting a deferral or waiver. This memo will include information on the necessity of the deferral or waiver. If and when deferrals are granted, peer reviews should be conducted as soon as practicable thereafter. An estimated schedule for completing the peer review shall also be attached to the memo requesting a deferral.

If approved by the Secretary or Deputy Secretary, written notification (including the planned completion schedule) shall be provided to the Director of OMB.

Waivers covering conflicts of interest are covered under Sections 10.14.

Back to Table of Contents

15. EXEMPTIONS TO THE PEER REVIEW PROCESS

Types of information specifically exempted from peer review include:

15.1.

Information that is related to national security, foreign affairs or negotiations involving international trade or treaties where compliance with the Bulletin would interfere with the need for secrecy or promptness.

15.2.

Information that is part of individual OA adjudications or permit proceedings.

15.3.

Time sensitive health or safety information.

15.4.

Routine statistical information released by Federal statistical OAs.

15.5.

Accounting budget, actuarial, or financial information generated by OAs that focuses on interest rates, banking currency, securities, commodities, futures, or taxes.

15.6.

Information disseminated in connection with routine rules that materially alter entitlement, grants, user fees, or loan programs, or the right and obligations of recipients.

15.7.

Licensing, approval, and registration processes for specific product development activities as well as site-specific activities.

15.8.

Official dissemination resulting in the course of adjudications and permit proceedings, unless the OA decides that peer review is practical and appropriate.

15.9.

Research disseminations resulting from DOT contracts or grants to universities that are clearly published under the imprimatur of the university receiving DOT funding for research.

15.10.

Deferrals for other types of information are expected to be highly unusual limited to compelled situations where flexibility is necessitated.

15.11.

Deadlines found in consent decrees agreed to by OAs after the Bulletin is issued.

15.12.

Regulatory Impact Analysis (RIAs) or regulatory flexibility analysis reports because they are already reviewed through an interagency review process under E.O.12866 that involves application of the principles and methods defined in OMB Circular A-4, except for underlying data and analytical models used.

15.13.

Derivative products (such as a product that only summarizes an already peer reviewed product or products such as fact sheets).

15.14.

Compendiums of existing models, methods and/or technologies.

15.15.

Minor intermediate products (i.e., a memorandum from a contractor) describing methods or results that will be incorporated in a larger product that will be peer reviewed.

15.16.

Preliminary or incidental analyses prepared separately from the work product ultimately used to support an OA action or decision (i.e., "what-if" analyses).

Back to Table of Contents

 APPENDIX A: KEY DEFINITIONS

Scientific dissemination

An OA initiated or sponsored distribution of scientific or technical information to the public.

Scientific information

Factual inputs, data, models, analyses, technical information, or scientific assessments based on the behavioral and social sciences, public health and medical sciences, life and earth sciences, engineering, or physical sciences.

Information quality act

Refers to Section 515 of Public Law 106-554 (Pub.L. No. 106-554,S 515, 112 Stat. 2763, 2763A-153-154 (2000))

Scientific assessment

Refers to an evaluation of a body of scientific or technical knowledge, which typical synthesizes multiple factual inputs, data, models, assumptions, and/or applies best professional judgment to bridge uncertainties in the available information.

Highly influential scientific assessment

Highly influential scientific assessments are those "that the agency or the OIRA Administrator determines could have a potential impact of more than $500 million in any one year on either the public or private sector or that the dissemination is novel, controversial, precedent setting, or has significant OA interest." A scientific assessment is an "evaluation of a body of scientific or technical information that typically synthesizes multiple factual inputs, data, models, assumptions, and/or applies best professional judgment to bridge uncertainties in the available information."Other criteria that can be used to determine whether a scientific work product is highly influential could include:

• Information that focuses on significant emerging scientific technical, or regulatory issues.
• Involves the application of an innovative approach for a previously defined problem/process/methodology.
• Satisfies a statutory or other legal mandate for peer review.
• Guidance for conducting economic or financial analysis.
• New financial or economic methodologies that serve as the primary method or protocol for conducting economic analyses within a program or operating administration.
• Unique, novel or previously un-validated applications of economic or financial methodologies.
• Broad scale assessments of regulatory programs, such as the required by Congressional-mandates.
• New preference surveys.
• New national surveys of costs and expenditures for environmental protection.
• Economic research plans developed to assess and advance the state-of-the-art science in economic theory.
• New meta-analysis that re-analyzes existing published literature and supporting data.
• New algorithms that will provide the basis of modeling and predicting traffic behavior.
• Large scale data sources.

Influential scientific information

Scientific information the OA can reasonably determine "will have or does have a clear and substantial impact on important public policies or private sector decisions."

Scientific integrity

In the context of peer review, scientific integrity refers to issues as expertise and balance of the panel members, the identification of the scientific issues and the clarity of the charge to the panel, the quality, focus, depth of the discussion of the issues by the panel, the rational and supportability of the panel's findings, and the accuracy and clarity of the panel report.

Process integrity

The term applies to issues such as transparency and openness, avoidance of real or perceived conflict of interest, a workable process for public comment and involvement, and adherence to defined submission and reporting procedures.

Conflict of interest

The National Academy of Sciences defines "conflicts of interest" as a financial or other interest that conflicts with the service of an individual on the review panel because it could impair the individual's objectivity or could create an unfair competitive advantage for a person or an organization.

Docket

A docket is an official public record. DOT publishes and stores on-line information about proposed and final regulations, copies of public comments on proposed rules, and related information in the Docket Management System (DMS). The DOT uses this docketed material when making regulatory and adjudicatory decisions, and makes docketed material available for review by interested parties. Specific documents covering the same issues are stored together in a docket. Therefore, a docket may contain several documents that are all related to the same subject matter.

DOT Docket Management System (DMS)

DMS is an electronic, image-based database in which all DOT docketed information is stored for easy research, and retrieval.

OIRA

Office of Information and Regulatory Affairs (OIRA) is located in the Office of Management and Budget.

Back to Table of Contents

APPENDIX B: LIST OF ACRONYMS AND ABBREVIATIONS

Back to Table of Contents

APPENDIX C: QUESTIONS AND ANSWERS REGARDING COMPLIANCE WITH OMB BULLETIN

What is peer review?

Peer review is intended to uncover any technical problems or unsolved issues in a scientific work product through the use of technically competent and independent (objective) experts. Peer review of a major scientific work product that will have the imprimatur of the Federal Government needs to be incorporated into the upfront planning of any action based in the work product. This includes obtaining the proper resources commitments (reviewers and money) and establishing realistic schedules.

What is a peer reviewer?

Peer reviewers are individuals who have technical expertise in the subject matter of the product undergoing peer review. Peer reviewers are generally external to the operating administration in cases where influential and highly influential information is involved. For highly influential information peer reviewers may come from outside the Federal Government such as universities. Peer reviewers must be objective experts that have not been associated with the generation of the specific influential or highly influential work product either directly by substantial contribution to its development or indirectly by significant consultation during the development of the product.

How is peer review different from public comment?

Peer review and public comment are often times mutually supportive and the OMB Bulletin encourages public comment on both peer review plans and highly influential and influential scientific information when feasible and appropriate. However, peer review is a rigorous process through which technical and scientific peers in a particular field (or fields) collectively work to ensure that the standards of the scientific method are adhered to during both research and publication phases.

What is a peer review panel?

A peer review panel can range from a few individuals to ten or more, depending on the nature of the product and the subject of the research. Individuals who serve as peer reviewers must always have appropriate scientific and technical expertise so that the complete range of issues presented in the peer review "charge" can be covered.

How should peer review records that pertain to a rulemaking action be handled?

The Peer Review Coordinator should work with the Department's CIO to ensure that proper docketing requirements are satisfied for a peer review of a work product supporting a new rule.

What is the "Charge" to the peer reviewers?

A key element of any effective peer review process is formulating a clear, focused charge that identifies recognized issues and invites comments or assistance. This request signals the OA's awareness of potential issues and its receptivity to expert recommendations. A "Charge" is important for the following reasons:

• First, it focuses the review by presenting specific questions and concerns that the OA expects the peer reviewers to address.
• Second, it invites general comments on the entire work product. The specific and general comments should focus mostly on the scientific and technical studies that have been applied in a sound manner.

When and how often should peer review take place?

As a general rule, it is most usual to consult with peer reviewers early in the research and development process. On large, complex research projects it may be prudent to establish a multi-stage peer review process as part of the overall peer review planning process.

What content should be in the peer review record?

Items that should be part of the record include:

• Draft work product of highly influential or influential scientific information submitted for peer review.

• Peer Review Plan
• Materials and information, including the charge, given to the peer reviewers.
• Peer Selection Process (methods/approaches used)
• Peer reviewer names and biographies (subject to privacy regulations) statement concerning potential conflicts and their resolutions.
• Peer Report
• Public comments and responses to public comments
• A written response to the peer review report explaining the OA's agreement or disagreement with peer comments and recommendations, the actions that OA has undertaken or will undertake in response to peer review comments, and (if applicable) the reasons the OA believes those actions satisfy key concerns or recommendations
• Any deferrals and/or waivers and rationale for granting them

       Compliance with Privacy Act, U.S.C. 552A, as amended, and OMB Circular A-130, Appendix I, 61 Fed. Reg. 6428 (February 20, 1996)

Back to Table of Contents

APPENDIX D: SUMMARY TRACKING MATRIX USING FHWA SUBMISSIONS TO ACHIEVE COMPLIANCE TEMPLATE

Back to Table of Contents

APPENDIX E: SUBMISSION FORM AND PEER REVIEW PLAN

EXAMPLE: NON-DESTRUCTIVE TESTING AND EVALUATION

FOR STRUCTURAL HEALTH MONITORING

FHWA OFFICE OF BRIDGE TECHNOLOGY

Back to Table of Contents

APPENDIX F: SAMPLE CONFLICT OF INTEREST FORM

Dear (Peer Reviewer -- insert name):

You have been requested by (Agency) to serve as a Peer Reviewer for______ __________(name of project)____________. Your participation in this review will be greatly appreciated. However, it is possible that your personal affiliations and involvement in particular activities could pose a conflict of interest or create the appearance that you lack impartiality in your involvement for this peer review. Although your involvement in these activities is not necessarily grounds for exclusion from the peer review, you should consult the contact named below or other appropriate official to discuss these matters. Affiliations or activities that could potentially lead to conflicts of interest might include:

a) work or arrangements concerning future work in support of industries or other parties that could potentially be affected by regulatory developments or other actions based on material presented in the document (or review materials) that you have been asked to review;

b) your personal benefit (or benefit of your employer, spouse or dependent child) from the developments or other actions based on the document (or review materials) you have been asked to review;

c) any previous involvement you have had with the development of the document (or review materials) you have been asked to review,

d) any financial interest held by you (or your employer, spouse or dependent child) that could be affected by your participation in this matter; and

e) any financial relationship you have or have had with (Agency) such as research grants or cooperative agreements.

Please contact ___________________ (name and contact info for Agency peer review official or primary contractor) to discuss any potential conflict of interest issues at your earliest convenience, but no later than _________.

[Be sure to date and sign this inquiry]

Back to Table of Contents

APPENDIX G: PEER REVIEW CHECKLIST

[1] A complete list of definitions is provided in Appendix A and a list of Acronyms and Abbreviations is provided in Appendix B.

[2] OMB Bulletin titled, "Final Information Quality Bulletin for Peer Review." (December 16, 2004), p. 11.

[3] OMB Bulletin titled, "Final Information Quality Bulletin for Peer Review." (December 16, 2004), p. 23.

[4]DMS is an electronic, image-based database in which all DOT docketed information is stored for easy research and retrieval. See http://www.dms.dot.gov

[5] The costs of peer review include both the direct costs of the peer review activity and those stemming from the potential delay in government and private actions that can result from peer review. On the other hand, peer review, if performed fairly and rigorously, can build consensus among stakeholders and reduce the temptation for courts and legislators to second-guess or overturn OA actions. Moreover, comprehensive stakeholder review can be an effective mechanism for enhancing that rate at which scientific information and technologies are deployed into the market.

[6] Balance refers to ensuring a broad and diverse representation of respected perspectives and intellectual traditions within the scientific community.

[7] Scientific integrity in the context of peer review, refers to the expertise and intellectual balance of the panel members; the identification of the scientific issues and clarity of the charge to the panel; the quality, focus, and depth of the discussion of the issues by the panel; the rationale and supportability of the panel's findings; and the accuracy and clarity of the panel report. Process integrity includes such issues as transparency and openness, avoidance of real or perceived conflicts of interest, a workable process for public comment and involvement, and adherence to defined procedures (OMB Bulletin, pg. 13).

[8] The OMB Bulletin strikes a compromise by disclosure of the identity of the reviewers, but not the public attribution of specific comments to specific peer reviewers.

[9] The Office of the Secretary (OST) Chief Information Officer and the OST Associate General Counsel are the primary contact organizations for Bulletin-related questions.

[10]A docket is an official public record. Specific documents covering the same issues are stored together in a docket. Therefore, a docket may contain several documents that are all related to the same subject matter. These two terms, docket and document, are used frequently throughout the DOT docket site. A docket is a folder that can contain several pages, which are the documents.

[11] National Academy of Sciences, "Policy and Procedures on Committee Composition and Balance and Conflicts of Interest for Committees Used in the Development of Reports," May 2003: Available at: http://www.nationalacademies.org/coi/index.html.