Detailed Information on the
Science and Technology: Homeland Security University Fellowships Assessment

Develop additional annual and long-term performance metrics and targets that focus on outcomes that accurately measure the program.

Consolidate and align Centers of Excellence based on the new structure of the S&T Directorate.

Conduct assessment of program and project milestones.

Develop additional research goals and outcomes for the DHS Centers of Excellence.

Conduct independent assessments of sufficient scope and quality.

Conduct assessment of program and project milestones.

Develop FY 2007 Execution Plan.

Develop a comprehensive five year strategic plan.

Develop 5 Year Research and Development Plan.

Measure: Percent of peer review ratings on University Programs' management and research and education programs that are very good or excellent.

Explanation:

Measure: Number of University Centers of Excellence.

Explanation:The University Centers of Excellence are mission-focused university consortiums that leverage the multi-disciplinary capabilities of universities to address the Department of Homeland Security needs.

Measure: Number of Science, Technology, Engineering and Mathematics (STEM) students supported.

Explanation:The Science, Technology, Engineering and Mathematics may include undergraduates, graduate students, post-docs - the University Centers can make the awards for scholars and fellowships in their disciplinary areas. The University Centers of Excellence are mission-focused university consortiums that leverage the multi-disciplinary capabilities of universities to address the Department of Homeland Security needs.

Measure: Months required to set up a University Center of Excellence

Explanation:This measure identifies the time it takes from awarding a Center of Excellence to distributing funding. During this time period the program works closely with those academic institutions selected to run the Center to develop plans and goals.

Measure: Percentage of University Program milestones that are met, as established in the fiscal year's budget execution plan. (New measure, added August 2007)

Explanation:The program has established a set of milestones that are necessary for achieving the goals and objectives of the program. These milestones are presented in the program's portion of the Science and Technology Directorate's fiscal year budget execution plan, which details the allocation of dollars and projected accomplishments for the year.

Measure: Percentage of University Program's research program costs allocated for administration. (New measure, added August 2007)

Explanation:This measure identifies the amount of research and development funding that is being used for administration which includes travel, SETA costs, IPAs, etc.

Is the program purpose clear?

Explanation: University Programs was created by the Homeland Security authorizing legislation (P.L. 107-296), as amended, to coordinate, leverage and utilize the unique intellectual capital of the university community to address current and future homeland research and technology needs and to provide educational support for the purpose of attracting and engaging highly talented individuals. The investment in the university community and individuals is essential for the future health of the Nation and in particular the development of a future cadre of homeland security experts. Homeland security research requires the knowledge and understanding of teams of experts from multiple disciplines especially those from the science and engineering fields. The enabling legislation also calls for building U.S. leadership in scientific areas of importance to homeland security. In that regard, University Programs has created the Scholars and Fellows Program to: (1) attract and engage highly talented individuals with a demonstrated record of high performance, skills and abilities; and (2) create a cadre of homeland security experts that understand how to work on problems that are multidisciplinary in nature, multifaceted and complex, to present them with interesting challenges and provide rewarding opportunities to address homeland security needs. A workforce with strengths across disciplines is imperative if experts from differing backgrounds are to be able to bring complementary perspectives to bear on complex problems. Another important factor underlying the importance of the disciplinary breadth of the workforce is the inability to predict the areas that will contribute to any given advancement in the future. The Scholars and Fellows Program will enhance the quality of the Nation's scientific leadership.

Evidence: 1) a. Authorizing legislation, Pub. L. 107-296, as amended, Section 308(b)(2), University-Based Centers for Homeland Security: "(2) UNIVERSITY-BASED CENTERS FOR HOMELAND SECURITY.?? ''(A) DESIGNATION ''(B) CRITERIA FOR DESIGNATION ''(C) DISCRETION OF SECRETARY ''(D) REPORT TO CONGRESS ''(E) AUTHORIZATION OF APPROPRIATIONS b. C.Rpt. 108-280 FY2004 and S.Rpt.108-280 FY2005 Appropriations "The Committee encourages the Department to consider all colleges and universities that meet the requirements of 6 USC 188 in the selection of university-based centers, including Historically Black Colleges and Universities, Tribal Colleges and Universities, Hispanic Serving Institutions and Native American Serving Institutions and Alaskan Native Serving Institutions." 2) Administration Policy Documents including: a. Homeland Security Presidential Directives: i. HSPD-5 "Management of Domestic Incidents" ii. HSPD-6 "Integration and Use of Screening Information" iii. HSPD-7 "Critical Infrastructure Identification, Prioritization, and Protection" iv. HSPD-8 "National Preparedness" v. HSPD-9 "Defense of United States Agriculture and Food" b. National Security Presidential Directives vi. NSPD-33 "Biodefense for the 21st Century" vii. NSPD-41 "Maritime Security Policy" c. Presidential Executive Orders - Education for Underrepresented Groups viii. 13230, 13256, 13270, 13336 3) Program Documentation a. Draft University Programs Strategic Plan b. Program Execution Plan c. FY2006 Congressional Justification 4) External Agency Documentations a. National Research Council publication, Making the Nation Safer b. DHS National Incident Management Plan

Does the program address a specific and existing problem, interest, or need?

Explanation: University Programs is designed to address the critical needs of science and technology research and education which is crucial to national security and to our Nation's future. This program establishes a coordinated, university-based system to enhance the Nation's homeland security and to support United States' leadership in science and technology, in order to address homeland security-related knowledge, technology and intellectual capital gaps. Existing and specific problems addressed by the University Programs Portfolio are further articulated in: ?? The Road Map for National Security, US Commission on National Security/21st Century, February 2001 ?? Innovate America, The Council on Competitiveness, December 2004 ?? NRC publication, Making the Nation Safer ?? Homeland Security Act of 2002, as amended

Evidence: ?? Section 308(b)(2), Homeland Security Act of 2002, as amended (P.L. 108-7) ?? National Research Council Workshop entitled "University Research Centers of Excellence on Homeland Security", January 29, 2004 ?? Administration Policy Documents including: o Homeland Security Presidential Directives:  HSPD-5 "Management of Domestic Incidents"  HSPD-6 "Integration and Use of Screening Information"  HSPD-7 "Critical Infrastructure Identification, Prioritization, and Protection"  HSPD-8 "National Preparedness"  HSPD-9 "Defense of United States Agriculture and Food" ?? FY2006 Congressional Justification ?? Program Execution Plan ?? The Road Map for National Security, U.S. Commission on National Security/21st Century, February 2001 ?? Innovate America, The Council on Competitiveness, December 2004 ?? NRC publication, Making the Nation Safer

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: University Programs is unique in that it serves the Nation's need to bring the intellectual capital from the university community and the future science and engineering talent to bear on complex problems in the Homeland Security mission space. University Programs addresses a unique need in government to create a university-based homeland security complex to fill a critical gap in the RTD&E mission space with respect to homeland security. While other Federal programs are involved in various other education efforts, DHS' University Programs are mission-informed to enhance the security of our Nation. The multidisciplinary character of universities provides for the uniqueness of this effort and fulfills a "compelling national need". University Programs has created an integrated system that leverages the expertise and research capabilities of individual national research centers to create an integrated system of centers called the Integrated Network of Centers or INC. Justification for this comes from section 308(b)(2) of the Homeland Security Act of 2002, as amended, which states that the Department is to establish a coordinated, university-based system of Centers to enhance the Nation's homeland security. All Centers were designed and selected in close consultation with other agencies to ensure complementary activities. The portfolio collaborates closely with other Federal agencies such as USDA, FDA, EPA, NSF, DoE, State, and Commerce, as well as a number of the DHS operating directorates, to leverage resources and to eliminate overlapping efforts. As an example of DHS Center and federal government collaboration, the two agro-security centers are collaborating closely with USDA and FDA. Specifically, they have active collaborative research programs with the Plum Island Animal Disease Center (DHS/APHIS) and the Food Safety Response Network (USDA/FDA). The University Programs' Scholarship and Fellowship program activities also provide for significant interaction with other Federal agencies and institutions. Scholars and Fellows must complete a required summer internship, and internship placements are frequently at DoE National Labs such as Lawrence Livermore and Sandia. University Programs staff participate on interagency groups such as the National Science and Technology Council; the National Academies of Science -- Government, University, Industry Research Roundtable (GUIRR); and the Federal Demonstration Partnership (FDP). Other Federal agency representatives are invited to participate in University Programs review panels as subject matter experts. Further, responsibility is delegated by S&T leadership to University Programs for additional research and educational programs, including outreach to Minority Serving Institutions. Representatives from other agencies across the government recently attended University Programs DHS Centers of Excellence Symposium on Science-Based Risk Analysis: Examining Threats, Vulnerabilities, Consequences and Responses. The goal was to identify the different strategies needed to anticipate, prevent, respond, and recover from both naturally occurring events and those initiated by an intelligent adversary.

Evidence: ?? Section 308, Homeland Security Act of 2002, as amended (P.L. 108-7) ?? Administration Policy Documents including: o Homeland Security Presidential Directives:  HSPD-5 "Management of Domestic Incidents"  HSPD-6 "Integration and Use of Screening Information"  HSPD-7 "Critical Infrastructure Identification, Prioritization, and Protection"  HSPD-8 "National Preparedness"  HSPD-9 "Defense of United States Agriculture and Food" ?? Program Execution Plan ?? Draft University Programs Strategic Plan

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The program provides generous stipends to university students but only requires that students participate in a brief internship during the summer with a DHS affiliated entity. Thus the benefits derived to the federal government are tenuous and based on a hope that students will choose to work in a career field that benefits the country. There is no assurance whatsoever that this will be the case. Our investment is therefore not clearly effective in provided a linkage of investment of dollars to results in terms of protection to the country. S&T argues that in an attempt to harness the country's capability to confront future threats, DHS believes that University Programs' competitive review processes are the best way to harness the multidisciplinary strength of universities and to attract students to homeland security disciplines. The topics chosen for the Centers of Excellence are determined through a detailed process to ensure the program's effectiveness. Specifically, after understanding the Department's mission and goals and then determining where science and technology can provide the greatest value both through leveraging developed tools as well as providing a force multiplier to operational end-users, clear knowledge and capability gaps are identified across the spectrum of the DHS mission space. Because of the unique characteristics of the university complex, which provides different research and development capability from the National and federal laboratory sector as well as from private industry, topical areas are chosen such that the research completed will provide the most beneficial solutions to the DHS mission space. The involvement of operational end-users in the selection process, on advisory boards of the Centers, and through feedback mechanisms will ensure the relevance and effectiveness of these centers. Competitive process structures are detailed in the Broad Agency Announcements for proposals for the DHS Centers of Excellence and in application requirements for the Scholars and Fellows programs. Centers of Excellence have external advisory committees with DHS representation to ensure that effective programs are developed and implemented to support homeland security needs. The Scholars and Fellows Programs will be independently reviewed next year to benchmark issues like retention, program satisfaction and career placement. Competitive grants encourage innovation from grant recipients, and the competition process allows S&T to emphasize different homeland security priorities responding to changing national needs. Center grants are three year grants, unlike five year grants more commonly used in other government programs. This allows for further flexibility to adapt to changing needs through reorientation of existing programs or recompetition. Different center topics are chosen to highlight DHS priorities. Each year the Scholarship and Fellowship program selects students studying in homeland security disciplines, and the required internship allows students to perform research in priority areas. These educational programs were created similarly to other successful educational programs within the Federal government to ensure their program effectiveness and efficiency. For all programs, UP utilizes continual assessment to adapt program needs to changing conditions in the field - through site visits and reverse site visits, progress, performance and financial reports, and other meetings with grantees, an example being UP's recent symposium.

Evidence: ?? University Programs Strategic Plan ?? University Programs policy documentation of a competitive, three-tiered, merit-based awards of Centers ?? Draft Oak Ridge Institute for Science and Education Annual Report ?? University Programs staff participation on interagency groups such as National Science and Technology Council; the National Academies of Science -- Government, University, Industry, Research Roundtable (GUIRR); and Federal Demonstration Partnership (FDP) ?? Participant feedback used to assess effectiveness of Scholars and Fellows programs ?? University Programs Report to Congress

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: University Programs is effectively targeted to address the program purpose of establishing Department of Homeland Security Centers of Excellence and the Scholars and Fellows programs. The program's highest priorities are to strengthen U.S. scientific leadership, generate and disseminate knowledge and research in areas critical to advancing homeland security missions, foster a homeland security culture within the academic community through educational and research programs, integrate homeland security activities across all agencies engaged in relevant academic research and create intellectual capital, and nurture a homeland security science and engineering workforce within the academic community.

Evidence: University Programs uses a rigorous, competitive, three-tiered, merit-based review process to select the most qualified group of Centers to maximize the interaction with the homeland security mission. The three-tiered process includes: Technical merit (external review) consists of expert review for scientific merit and technical quality, conducted by scientists or other subject matter experts from academia, industry, and government; Mission relevance (internal review) consists of programmatic review for scientific merit, technical quality, and mission relevance, conducted by scientists and subject matter experts from Federal agencies; and Management effectiveness (site visits) consists of expert and programmatic review for the purpose of interviewing finalists??lead universities and their partners??conducted by scientists, subject matter experts from academia, industry and government leadership. The Scholars and Fellows Programs were developed to effectively attract, engage, and retain the unique intellectual capital in the academic community to address current and future homeland security challenges. DHS is committed to educating the nation's next generation of workers that have an acute understanding and skills that can contribute to the Department's homeland security mission. In selecting the scholarship and fellowship recipients, University Programs uses a competitive, two-tiered merit-based review process to ensure that recipients are qualified and their academic interests are compatible and relevant to the Department's mission. Meeting high standards and excellence in academic performance, writing skills, meaningful references, and test performance are required and evaluated by external university experts. Following an external review, an internal review is conducted and comprised of subject matter experts within the Department to ensure compatibility and relevance to the Department's mission and goals. UP activity funds are almost exclusively allocated through rigorous peer reviewed merit-based competitions, with the exception of funding for workshops aimed at outreach to Minority Serving Institutions.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The long term outcome performance goal for University Programs is to coordinate, leverage, and utilize the unique intellectual capital of the U.S. university community to address current and future homeland research and technology needs and to provide educational support for the purpose of attracting and engaging highly talented individuals to become scientists with specialized skills that can be utilized to counter the threats possed by terrorists. It should also be remembered that the University Programs is a newly established program only two years in existence and as such, the nearer term performance measures have been associated with standing up and creating effective and efficient programs. The University Programs has additionally established longer-term measures of output to ensure the effectiveness of these programs to meet the desired outcomes. The program is in the process of developing long-term outcome measures that focus on the reduction of risk as a result of implementing program objectives. We are establishing specific outcome goals to be evaluated over the next several years and to be affected by strategic adjustments of input and throughput parameters. UP is now developing a performance model with specific outcome measures.

Evidence: ?? DHS Strategic Plan ?? FY 2007 - 2011 DHS FYHSP ?? Draft University Programs Strategic Plan

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: University Programs has ambitious but realizable multi-year targets and timeframes established, recognizing that University Programs has only been in existence two years with the first Center in operation only about one year. DHS Centers have taken on average six months to set up from the award of the grant, when the average in the federal government is eighteen months. In addition, UP established four separate education programs in two years, when the average in the federal government is about one program per year. The following are UP throughputs which show ambitious targets: ?? University Centers of Excellence throughput targets and timeframes: ?? Three DHS Centers were established in FY2004 ?? An additional four DHS centers (bringing the total to seven) are planned by the end of FY2005 ?? The grants for the Centers of Excellence are of three year duration, potentially renewable, but continually monitored and reviewed ?? Cooperative Centers throughput targets and timeframes: ?? Two Cooperative Centers with other federal entities are planned in FY2005 ?? Scholars and Fellows Program throughput targets and timeframes include the establishment of 4 separate programs (including AAAS Fellowships) in 2 years: ?? FY2003: Scholarship and Fellowship Program established; approximately 100 graduate and undergraduate students in relevant fields of study important to homeland security ?? FY2004: approximately 200 graduate and undergraduate students in relevant fields of study important to homeland security ?? FY2005: approximately 200 graduate and undergraduate students in relevant fields of study important to homeland security ?? FY2006: approximately 300 graduate and undergraduate students in relevant fields of study important to homeland security ?? FY 2005: Postdoctoral Program established ?? FY 2006: Postdoctoral Program students double from 10 to 20 ?? FY 2005: Summer Faculty and Student Research Team Program established Budget allocations have largely determined this number of Centers and students. However, UP has continually made careful analyses on how to best use program funds. As an example of this, with regard to its education activities, most of the resources are allocated to the Scholarship and Fellowship Program, where DHS believes homeland security long term needs can best be promoted. Significant progress is expected in University Programs' research and educational program output, which can then be directly incorporated in support of DHS operational requirements. Targets and time frames will be evaluated on a frequent basis by DHS personnel and subject matter experts. In terms of longer-term measures, a successful program would be University Programs serving as a nucleus for a much longer, sustained effort in homeland security, attracting additional support and personnel from other Federal agencies, state and local government and the private sector.

Evidence: University Programs' measures are included in the UP Program Execution Plan, Strategic Plan and the S&T 5-Year performance based budget. Additional specific evidence includes: ?? University Programs Centers of Excellence Work Plans ?? University Centers of Excellence work plans are tied to the DHS Strategic Plan ?? All Centers of Excellence provide work plans, deliverables and time lines, reviewable on at least an annual basis by external committees ?? Students will be tracked to determine subsequent career paths

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: In its model of a university community devoted to mission-related research, UP will be tracking annual measures such as the number of researchers and students in homeland security-related disciplines, and the number of other individuals working in homeland security-related fields. In tracking such annual measures, UP will benchmark other agencies and the metrics they utilize. It should also be remembered that the University Programs is a newly established program only 2 years in existence and as such, the nearer term performance measures have been associated with standing up and creating effective and efficient programs. The University Programs has additionally established longer-term measures of output to ensure the effectiveness of these programs to meet the desired outcomes. Furthermore, additional measures will be considered and applied as the program matures so that quality "leading indicators" of outcome are identified. Attention is also referred back to the table provided in section 2.1 which lists specific performance measures that provide indication of meeting the programs long-term goals. UP will also track the yearly success of the research efforts of its Centers and their contribution to DHS strategic objectives. Feedback continues to be obtained from DHS operating directorates to indicate the impact of the Centers' research on the war on terrorism. UP is planning a major comprehensive program review of the research programs for September 2005 and evaluation of the first Center for renewal in FY2006. All these activities will include participation from outside advisory groups and other stakeholders. UP performance will be supplemented by Centers' progress reports, site visits and reverse site visits. Progress reports, site visits and reverse site visits will be conducted of the education programs as well.

Evidence: ?? Centers of Excellence work plans and progress reports ?? Student Program Reports

Does the program have baselines and ambitious targets for its annual measures?

Explanation: University Programs (UP) has ambitious but realizable multi-year targets and timeframes established, recognizing that UP has only been in existence two years with the first Center in operation only about one year. DHS Centers have taken on average six months to set up from the award of the grant, when the average in the federal government is 18 months. In addition, UP established four separate education programs in two years, when the average in the federal government is about one program per year. UP created its programs during the last two years, and is developing a baseline and ambitious targets for its annual measures. Research outputs from the Centers of Excellence will be measured using a number of parameters: ?? Academic measures of quality will include numbers of refereed publications, citations, and number of Masters and Ph.D. graduates and other appropriate measures ?? Relevancy to DHS mission will be assessed by the number and quality of the interactions between Centers both internally to DHS and externally with other Federal agencies, and detailed feedback from operational parts of DHS ?? Technical objectives and quality of research will be monitored ?? Program planning and risk management activities will be monitored ?? Technology transfer to demonstration or operational deployment stages for end users will be monitored The Scholars and Fellows Program will be assessed by program applicant numbers, number and quality of graduates, student and mentor feedback and participant persistence in fields of study. For all our programs we have set high standards for impact on mission and for expanding available workforce.

Evidence: ?? University Programs -- Program Execution Plan ?? Draft University Programs Strategic Plan ?? NSF performance measures parameters ?? Centers' annual progress, final reports and site reviews ?? Student feedback and tracking ?? University Programs annual review and program evaluation for FY2007

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: Grantee partners are working towards the goals of the program -- addressing knowledge gaps, and providing unique homeland security mission-relevant information, products and solutions available to government decision makers and others to make our Nation more secure. As examples of current products of research that are mission relevant: The Center for Risk and Economic Analysis of Terrorism Events (CREATE) is developing a Terrorism Modeling System (CTMS) as both a methodology and software system for assessing the risks and consequences of terrorism within the framework of economic analysis and structured decision-making. The essential elements of CTMS include: ?? Terrorism Threat Characterization (TTC) system for problem definition and decision-making; ?? Staircase model of terrorism intervention, which affords multiple opportunities to defeat a terrorist conspiracy, and considers feedback effects; ?? Four-step modeling system, consisting of risk assessment, consequence assessment, emergency response and economic analysis; ?? An integrated family of software tools, linked through the Risk Analyst Workbench (RAW), that utilizes a geographic-information-system (GIS) for data integration and display Centers also have advisory groups that oversee their research. As an example, CREATE has user advisory groups that include members from the Los Angeles Fire Department and the Port of Los Angeles, the FBI, DHS' Transportation Security Administration, and private industry such as AT&T and General Dynamics. The National Center for Food Protection and Defense is developing a computer-based predictive modeling tool that depicts the geospatial movement of selected, vulnerable food products (such as fresh produce, fluid milk, ground beef, and seafood) along the food chain from primary production to the consumer. It will visually demonstrate key health effects, including morbidity rates, medical intervention, and mortality rates. This tool will also profile the expected cost to individual consumers, the food industry, and the public health system. The National Center for Foreign Animal and Zoonotic Disease Defense is developing new classes of candidate vaccines for testing protective immunity against River Valley Fever, Brucella and Avian Influenza. Brucella vaccine candidates have been identified based on vaccine trials in mice and in small ruminants. Aerosol chambers are being setup for challenge studies. These candidates will ultimately be tested in the target livestock hosts in small scale studies under appropriate containment conditions in USDA and CDC approved bio-containment laboratories at Texas A&M University. UP ensures that grantees are working towards the goals of the program through the following procedures: University Centers of Excellence ?? For the selection of each University Center of Excellence, key partner commitment is validated during site visits and monitored throughout the grant award ?? Following selection, Centers and their sub-grantees convene to review proposed plans, goals and deliverables with University Programs. Progress is regularly monitored with individual Centers, individual Centers and their major partners, and with the system of Centers that has been developed (Integrated Network of Centers). Regular monitoring includes weekly phone conversations with Center principal investigators, monthly INC meetings (face-to-face, teleconferencing or phone conferencing). At these meetings, shared goals and issues and relevance to mission are discussed

Evidence: ?? DHS Strategic Plan ?? University Programs Broad Agency Announcements ?? Scholars & Fellows application materials ?? DHS and University Programs Strategic Plans, University Programs strategic goals matrix ?? Centers of Excellence Grantee Reports

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: Since the first Center of Excellence has only been in operation for one year out of a three year grant period (all Centers have three year competitive grants), now is the appropriate time to begin evaluation activities. UP will have a Centers' program review in September 2005, and a program evaluation focusing on the education/student programs in FY2006, following three years of funding for students with the first group of students going through the academic pipeline. A similar evaluation of the Centers of Excellence program will occur after more Centers have been established and agreed upon research plans can be evaluated. An external program evaluation is planned for FY2007 for the Integrated Network of Centers, which will cover mission and user relevance, technical competency, communications and outreach, and management effectiveness of the Center Complex. All these independent assessments will focus on the outcomes of UP's investments. University Programs recently held a Centers of Excellence Symposium where feedback was asked of participants. Data from the feedback indicated that participants responded positively when asked how well the Symposium illustrated how university-based research can contribute to the defense and security of our Nation's assets. The ORISE draft annual report provided information on the Scholarship and Fellowship Program activities. Specifically, feedback was gathered from the 2003 class of Scholars and Fellows who attended the Washington, D.C. orientation for students. Two-thirds of the students indicated that they were very satisfied with the orientation experience. In terms of more direct monitoring of UP activities, UP staff regularly interact with Center leadership and researchers. Center advisory committees include S&T and other subject matter experts. Each Center will undergo a comprehensive site visit review before the end of the second year of their grants. Student academic progress is tracked on a regular basis through communication with the students' faculty advisors, annual reports, and during internships. Formal written (as well as informal verbal) feedback is obtained from students and mentors during their required internships.

Evidence: ?? Quarterly, semi-annual and annual reports of Centers ?? Program Execution Plan ?? Integrated Network of Centers (INC) meetings ?? Student and internship mentor feedback forms ?? Academic student progress reports ?? University Program Symposium feedback responses ?? Draft ORISE Annual Report

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: Budget requests are tied to the accomplishment of the preceding discussion of performance goals, and are also linked to the Science and Technology Directorate's Five-Year planning, programming, budgeting, and execution processes, which include the following components: ?? Enabling legislation ?? Associated Homeland Security Presidential Directives ?? DHS Strategic Plan ?? DHS Performance-Based Budget Targets ?? S&T Performance-Based Budget Targets ?? Draft University Programs Strategic Plan ?? University Programs - Program Execution Plan (PEP) ?? University Programs 5 Year Resource Allocation Plan (RAP) The program baseline of 7 DHS Centers of Excellence, 2 Cooperative Centers, and 300 Scholars and Fellows supported was established in FY2004 and based upon the FY2004 budget estimate and subsequent yearly budget requests. A model has been developed to assess the potential impact of funding, policy, or legislative decisions that may affect the baseline development or implementation of University Programs though 2010. This model clearly articulates to senior decision makers the impacts of decisions to the baseline numbers of University Centers of Excellence, the numbers of Cooperative Centers, and the numbers of Scholars and Fellows that will enable the program to achieve its stated program goals. In a separate model, University Programs estimates all direct and applicable indirect costs to support budget requests and tracks these estimates to actual costs incurred. Attention is referred to the table provided in section 2.1 which conveys the outputs and outcomes that the budget is supporting.

Evidence: ?? Enabling legislation ?? Associated Homeland Security Presidential Decision Directives ?? DHS Strategic Plan ?? DHS Performance-Based Budget Targets ?? S&T Performance-Based Budget Targets ?? University Programs - Program Execution Plan (PEP) ?? University Programs 5 Year Resource Allocation Plan (RAP) ?? Draft University Programs Strategic Plan ?? Spreadsheet Model Estimating University Centers of Excellence, Cooperative Centers, and Scholars and Fellows ?? Spreadsheet Model Estimating cost of Scholars & Fellows (Salaries, Stipends, Health Insurance, and other overhead)

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: University Programs has developed a draft strategic plan that cascades from the Department's Strategic Plan. University Programs staff regularly meets to review program progress. In addition, University Programs participates in an annual offsite meeting with the Office of Research and Development. The University Programs Annual Program Review, attended by internal and external subject matter experts, will evaluate the success of University Programs meeting its planned goals and objectives.

Evidence: ?? DHS Strategic Plan ?? Draft University Programs Strategic Plan

If applicable, does the program assess and compare the potential benefits of efforts within the program and (if relevant) to other efforts in other programs that have similar goals?

Explanation: University Programs is unique in that it serves the Nation's need to bring the intellectual capital from the university community and the future science and engineering talent to bear on complex problems in the homeland security mission space. University Programs addresses a unique need in government to create a university-based homeland security complex to fill a critical gap in the RTD&E mission space with respect to homeland security currently not being similarly addressed across the federal system. While other federal programs are involved in various other education efforts, DHS' University Programs are mission-informed to enhance the security of our Nation.

Evidence:  

Does the program use a prioritization process to guide budget requests and funding decisions?

Explanation: University Program's highest priorities are to strengthen U.S. scientific leadership, generate and disseminate knowledge and research in areas critical to advancing homeland security missions, foster a homeland security culture within the academic community through educational and research programs, integrate homeland security activities across all agencies engaged in relevant academic research and create intellectual capital, and nurture a homeland security science and engineering workforce within the academic community. The topics chosen for the Centers of Excellence are determined through a detailed process to ensure the program's effectiveness. Specifically, after understanding the Department's mission and goals and then determining where science and technology can provide the greatest value both through leveraging developed tools as well as providing a force multiplier to operational end-users, clear knowledge and capability gaps are identified across the spectrum of the DHS mission space. Because of the unique characteristics of the university complex, which provides different research and development capability from the National and federal laboratory sector as well as from private industry, topical areas are chosen such that the research completed will provide the most beneficial solutions to the DHS mission space. The involvement of operational end-users in the selection process, on advisory boards of the Centers, and through feedback mechanisms, will ensure the relevance and effectiveness of these centers. Selection of topics and preparation of the BAAs are also performed in close consultation with subject matter experts within S&T, and other DHS Directorates and other Federal agencies. University Programs is working closely with relevant Federal agencies including USDA, FDA, EPA, and NSF to ensure relevancy and a better understanding of the importance of the research being carried out by the Centers. We communicate regularly through committees of the National Science Technology Council (NSTC) to assess the requirements and quality of the educational programs. In selecting research areas for the DHS Centers of Excellence, University Programs has sought input from a variety of sources. These sources include the Homeland Security Act of 2002, as amended, the National Research Council (NRC), the Homeland Security Presidential Directives (HSPDs), other DHS directorates, and DHS S&T subject matter experts (SMEs). Prioritization of Center topics is determined, among other things, by S&T subject matter experts familiar with scientific and knowledge gaps, University Programs Integrated Process Team (IPT) within S&T, other DHS directorates, and developing relationships with other Federal agencies. Current priorities are aligned with the Department's Strategic Plan, the Homeland Security Act of 2002, as amended, and NRC publication, Making the Nation Safer.

Evidence: ?? DHS Strategic Plan ?? Draft University Programs Strategic Plan ?? University Programs Report to Congress ?? Homeland Security Act of 2002, as amended ?? NRC Publication, Making the Nation Safer ?? University Programs -- Program Execution Plan ?? Integrated Network of Centers (INC) meetings to optimize interactions and share other opportunities for collaborations ?? University Programs Resource Allocation Plan (RAP) ?? Homeland Security Presidential Directives

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: University Programs collects program performance reports from its grant and other recipients. Centers of Excellence provide quarterly, semiannual, annual and final reports. Final reports are required by the grant award; however, University Programs has required additional reporting to assess grantees' progress. Scholars and Fellows' academic institutions provide progress reports and students provide annual reports as well. Performance reports are regularly evaluated in context of goals of the programs and goals of the DHS Strategic Plan. These evaluations enable both policy and science-based decision makers at varying levels within the S&T Directorate to adjust program priorities or take other appropriate management actions. Performance of key program partners are also assessed for key program activities through site visits and reverse site visits. Furthermore, University Programs is conducting an internal program review in September 2005, and thereafter annually. A program evaluation is being planned for FY2007 for the DHS Centers of Excellence and Integrated Network of Centers and for FY2005 for the Scholars and Fellows Program, and thereafter to be performed every 3 years.

Evidence: ?? University Programs Centers of Excellence quarterly, semi-annual, annual, and final reports ?? Oak Ridge Institute of Science and Education quarterly financial reports and annual performance reports

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) are held accountable for cost, schedule and performance results. Performance expectations for program partners are established at the time the grants, contracts or work orders are negotiated. Performance expectations are clearly articulated with key partners and it is clearly understood that they must meet all contractual program reporting requirements as well as financial record keeping requirements. Performance is monitored by University Programs' staff. Corrective actions are taken as needed to ensure accountability. Direct oversight from the Office of Naval Research (ONR), which administers Center grants, is through quarterly financial reports which ensure that grant terms and conditions are met (e.g. cost requirements). Centers are also required to submit program activity progress reports that are used to verify schedule and scope of effort are as agreed to in the grants. The Oak Ridge Institute for Science and Education (ORISE), which administers and oversees the student education programs for UP, submits financial and program activity progress reports at least quarterly from which cost, schedule, and scope status of programs can be evaluated. The University Programs staff also conducts site visits and reverse site visits with grantees. Centers also have advisory groups that oversee their research. As an example, CREATE has user advisory groups that include members from the Los Angeles Fire Department and the Port of Los Angeles, the FBI, DHS' Transportation Security Administration, and private industry such as AT&T and General Dynamics. Federal managers' performances are linked to the University Programs Strategic Plan's goals and objectives. The performance standards for these managers have clearly defined or quantifiable performance standards for each manager.

Evidence: ?? Draft University Programs Strategic Plan ?? Grant Award Terms and Conditions Documents ?? Performance Standards for University Programs Manager (Annual Performance Standards) ?? University Programs Centers of Excellence quarterly, semi-annual, annual, and final reports ?? Oak Ridge Institute of Science and Education quarterly and annual reports

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: University Programs recognized the need for a startup period for Centers and other activities to allow for merit based selection of Centers of Excellence and individual student awards. No-year funds were provided to facilitate this startup. Our budget plan incorporates carry over of funds, with all funds to be obligated by 2010 as UP reaches its steady state of 7 Centers of Excellence, 2 Cooperative Centers, and 300 students supported. UP Programs and UP partners establish schedules for obligations that properly correspond to the resource needs of the UP Program Execution Plan (PEP), part of the execution plan of the Office of Research and Development of S&T. Adequate procedures exist for reporting actual expenditures, comparing them against the intended use, and taking timely and appropriate action to correct any possible single audit findings when funds are not spent as intended. Furthermore, program funding is tracked regularly to ensure timely and accurate execution. DHS Centers of Excellence have taken on average six months to establish from the award of the grant, while the average in the federal government for similar centers establishment is eighteen months. In addition, UP established four new separate education programs in two years, while the average number of new programs established in the educational community in the federal government is one program per year. The nature of competitive grant making requires a certain amount of time to set up centers and student education programs and funds can not be immediately obligated at the beginning of the fiscal year. However, Unversity Programs has significantly reduced the time it has taken for our award activities to be completed from previously accepted standards. Funds are obligated as awards are being made and the five year budget plan is only sized properly to sustain the existing programs.

Evidence: ?? University Programs - Program Execution Plan (PEP) ?? Budget execution reports from the Federal Financial Management System (FFMS) ?? Performer's financial reports ?? University Programs Expenditures Tracking Document

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: University Programs has standard operating procedures to ensure that competitive sourcing and cost comparisons are used to measure and achieve efficiencies and cost effectiveness in program execution. University Programs' Centers of Excellence are awarded through full and open competitions to ensure cost effectiveness. Furthermore, once grants are awarded, program performance reports are reviewed and analyzed by program managers to determine cost, schedule, scope, and quality performance. University Programs is benchmarking the process of creating and the subsequent performance of DHS Centers of Excellence with other similar programs. Additionally, the Scholars and Fellows Programs benchmark program elements and processes with those of other federal agencies that are similar in design. An efficiency measure for UP is the time it takes for the selection and announcement procedures for both students and centers. DHS is moving towards incorporating an in-house grants management system. Once this is accomplished, another efficiency measure will be UP's costs with running its research and education competitions.

Evidence: ?? Centers of Excellence Broad Agency Announcements ?? Scholarship and Fellowship Program Announcements ?? NSF Selected Graduate Fellowship Programs ?? University Programs Centers of Excellence quarterly, semi-annual, annual, and final reports ?? Oak Ridge Institute of Science and Engineering quarterly and annual reports

Does the program collaborate and coordinate effectively with related programs?

Explanation: University Program collaborates and coordinates effectively with other related federal government programs by promoting partnerships, intra-agency and interagency efforts. The topics chosen for the Centers of Excellence are determined through a detailed process to ensure the program's effectiveness as well as ensure the program is coordinated with any other similar efforts. Specifically, after understanding the Department's mission and goals and then determining where science and technology can provide the greatest value both through leveraging developed tools as well as providing a force multiplier to operational end-users, clear knowledge and capability gaps not presently being addressed are identified across the spectrum of the DHS mission space. Because of the unique characteristics of the university complex, which provides different research and development capability from the National and federal laboratory sector as well as from private industry, topical areas are chosen such that the research completed will provide the most beneficial solutions to the DHS mission space. The involvement of operational end-users in the selection process, on advisory boards of the Centers, and through feedback mechanisms ensures the relevance, coordination with existing programs and effectiveness of these Centers. Cooperative Centers have been established to increase the ability of the Federal government to collaborate with other agencies that have inter-related but separately budgeted efforts for homeland security initiatives. For example, DHS & EPA are collaborating on a Cooperative Center on microbial risk assessment, to be announced in FY2005. A second cooperative center is planned for FY2005. In addition, in creating its Broad Agency Announcements for the DHS Centers of Excellence, University Programs works with DHS S&T component portfolios, the other DHS Operational Directorates, and other federal agencies and entities in identifying topics for Centers and throughout the selection process. This ensures that the best topics are identified and also prevents overlap of mission with other agencies. DHS Centers of Excellence collaborate with other federal agencies such as HHS, FDA, and USDA as examples.. In addition, University Programs staff participates on interagency groups such as the National Science and Technology Council (NSTC); the National Academy of Sciences -- Government, University, Industry Research Roundtable (GUIRR); and the Federal Demonstration Partnership (FDP). Centers also have advisory groups that oversee their research. As an example, the CREATE Center has user advisory groups that include members from the Los Angeles Fire Department and the Port of Los Angeles, the FBI, DHS' Transportation Security Administration, and private industry such as AT&T and General Dynamics. As another example of collaboration, DHS engaged in a partnership with the National Research Council (NRC) to operate its Postdoctoral Program. NRC has a long history of supporting federal education efforts, and runs very effective similar programs for other federal agencies. DHS determined that approaching the NRC to operate its program would be the most effective way to identify the best students through NRC's academic networks. For the University Program staff to attempt to reach these students on its own would have been less cost effective.

Evidence: ?? Joint DHS-EPA Request for Proposals for Cooperative Center on Microbial Risk Assessment ?? NSTC, GUIRR, and FDP documentation ?? Postdoctoral Program documentation

Does the program use strong financial management practices?

Explanation: The Directorate of Science and Technology practices sound financial management consistent with the Department of Homeland Security guidance. Specifically, the Office of Research and Development, in which University Programs resides, has established a financial management process to ensure payments are made accurately and timely consistent with section 5 CFR 1513, Office Management and Budget Final Rule of 1999. Program received a clean financial audit and no material weaknesses.

Evidence: ?? Budget Execution Reports from the Federal Financial Management System (FFMS)

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The formal University Programs Performance Management Plan is in development. As the Performance Management Plan is being developed, S&T leadership and University Programs staff convene regularly scheduled meetings to address management deficiencies and take corrective action. All University Programs activities will be included in annual program reviews (first program review scheduled September 2005). The membership of the program review panel will include subject matter experts from other federal agencies, National Laboratories, DHS S&T senior level managers and individuals from other DHS elements outside of S&T having an interest in University Programs activities. Currently, monthly teleconference meetings are held and University Programs program managers communicate on a weekly basis with DHS Centers principal investigators to ensure that issues or challenges are identified and resolved in a timely manner. University Programs also participates in an S&T Integrated Product Team (IPT) and members are included in UP activities to enhance internal communication and continuous feedback.

Evidence: ?? Staff Meetings ?? Weekly Reports for the Office of Research and Development ?? IPT member involvement in University Programs workshops

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: All University Programs grant awards are based on a clear competitive process that includes a qualified assessment of merit. Independent merit review and ranking of applications occurs in the following competitions: Centers of Excellence, Cooperative Centers, Scholarship and Fellowships, Postdoctoral Program, AAAS Fellowships, and the Summer Faculty and Student Research Team Program. University Programs uses a rigorous, competitive, three-tiered, merit-based review process to select the most qualified group of Centers to maximize the interaction with the homeland security mission. The three-tiered process includes: Technical merit (external review) consists of expert review for scientific merit and technical quality, conducted by scientists or other subject matter experts from academia, industry, and government; Mission relevance (internal review) consists of programmatic review for scientific merit, technical quality, and mission relevance, conducted by scientists and subject matter experts from Federal agencies; and Management effectiveness (site visits) consists of expert and programmatic review for the purpose of interviewing finalists??lead universities and their partners??conducted by scientists, subject matter experts from academia, industry and government leadership. The Scholars and Fellows Programs were developed to effectively attract, engage, and retain the unique intellectual capital in the academic community to address current and future homeland security challenges. DHS is committed to educating the nation's next generation of workers that have an acute understanding and skills that can contribute to the Department's homeland security mission. In selecting the scholarship and fellowship recipients, University Programs uses a competitive, two-tiered merit-based review process to ensure that recipients are qualified and their academic interests are compatible and relevant to the Department's mission. Meeting high standards and excellence in academic performance, writing skills, meaningful references, and test performance are required and evaluated by external university experts. Following an external review, an internal review is conducted and comprised of subject matter experts within the Department to ensure compatibility and relevance to the Department's mission and goals. Vehicles that provide fair and open competition, and that prove a reasonable amount of outreach to encourage participation, are made available to the public through: ?? BAAs that are posted on Fedbizopps.gov ?? Grants.gov ?? DHS Homepage website (dhs.gov) ?? Oak Ridge Associated Universities (ORAU) website (with a link for potential collaborators). Scholars and Fellows activities are posted on the DHS website and on the ORAU website ?? National Research Council's Research Associateships website

Evidence: ?? University Programs Report to Congress ?? Centers' BAAs ?? Scholars and Fellows Programs' applications and program guidelines ?? ORISE Draft Annual Report

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: University Programs has an oversight practice that provides sufficient insight and knowledge of grantee planning and execution activities. The current reporting system for each University Program requires that: ?? University Centers of Excellence ?? The Office of Naval Research serves as the grant award agent for University Programs. During grant negotiations, cost categories are established andnegotiated to collect planning and execution information. As each program begins its execution phase, financial and technical reports are used to verify grantees' use of funds in eligible activity categories established at grant award. ?? University Program conducts site visits to DHS Centers of Excellence and reverse site visits to DHS by Center principal investigators and their major partners' grantees on a regular basis. Site visits are also made to Scholars and Fellows internship sites. ?? University Program managers and S&T senior managers regularly participate in Center activities. The University Programs Director and/or Program Managers serve as ex-officio members of the Centers' Science Advisory Committees. Teleconferences are conducted monthly with DHS principal investigators and University Programs staff. University Programs has created an Integrated Network of Centers process to ensure that Centers' research is shared and leveraged among all Centers. ?? University Programs reviews grantee performance, and receives reports from the Office of Naval Research which track expenditures to verify that funds are used for their designated purpose. On a quarterly basis, University Program staff reviews budget and performance information provided by the DHS Centers of Excellence. ?? Institutions bill for students' tuition and fees, and student performance is reported and monitored. Students also provide an annual progress report. A reporting system is in place to document grantees' use of funds in eligible activity categories. ?? Student internship sites, i.e, DHS Laboratories, National Laboratories, and other DHS affiliated venues are visited by University Programs staff. University Programs conducts a review of approximately one-third of the internship sites annually. Site visits include meeting with mentors, students and internship coordinators to discuss the students' progress. ?? Quarterly financial reports are submitted by program administrators for the purpose of tracking actual expenditures to verify that funds are used for their designated purpose. Students are required to submit a written report at the end of each academic year as well as at the conclusion of their required internship (or create a scientific poster). Students are also expected, during their internship, to attend a minimum of three scientific topics, make an oral presentation, and provide feedback about their research experience. Mentors provide feedback on the program and students progress. Lastly, incoming students are required to participate in a fall DHS S&T orientation to familiarize them with the DHS mission, to interest them in the variety of research opportunities being offered and to educate them on critical S&T homeland security issues.

Evidence: ?? Grant terms and conditions ?? Financial quarterly reports ?? Report to Congress ?? DHS Program Descriptions (on www.dhs.gov) ?? Scholarship and Fellowship Program Orientation Agendas

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: University Programs collects grantee performance data on an quarterly, semiannual and annual basis and makes the semiannual and annual performance information publicly accessible through the Centers' websites. Currently, the semiannual reports are available on the DHS Centers' websites. Annual and final reports are not due until the summer of 2005, but will be made available to the public through the same websites. University Programs has planned for a "one-stop" Integrated Network of Centers website. The number of applications and the numbers of scholars and fellows selected by discipline from the previous year's competition are provided on the ORISE website.

Evidence: ?? DHS Centers Websites (University of Southern California, University of Minnesota, and Texas A&M University) ?? ORISE website

For R&D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain program quality?

Explanation: As stated in the enabling legislation and explained in question 3C01 above, University Programs uses a broadly competitive merit based process to allocate R&D funding to grantees. University Programs reviews are consistent with the definitions in Circular A-11 and ensure that intramural and extramural research programs are competitively funded based on scientific and technical merit by peer review.

Evidence: A description of the awards process and results of external assessments of project performers are provided in: ?? University Programs Report to Congress

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The long term performance goal for University Programs is to coordinate, leverage and utilize the unique intellectual capital of the university community to address current and future homeland research and technology needs and to provide educational support for the purpose of attracting and engaging highly talented individuals. UP's current performance measure is the establishment of seven Centers of Excellence and two Cooperative Centers, and support of 300 students. So far they have established four Centers of Excellence and awarded funding to over 270 students. UP is now developing a performance model with specific outcome measures. This model consists of the following conceptual framework: Through puts are under the control of UP through program activities and program funding. The program is establishing specific outcome goals to be evaluated over the next several years and which can be affected by strategic adjustments of input and throughput parameters. Until then, a score of Small Extent is appropriate considering some process has been made establising Centers and funding students but outcome measures are under development.

Evidence: ?? FY2004 DHS Budget ?? FY2005 DHS Budget ?? Draft University Programs Strategic Plan ?? University Programs -- Program Execution Plan

Does the program (including program partners) achieve its annual performance goals?

Explanation: University Porgrams has established four Centers of Excellence as was their annual performance goal. They also awarded funding to the number of students they had expected for each year in furtherance of the annual goal. UP is now in a better position to begin tracking the results of its annual performance goals, given that its first Center is only one year old and just submitted its progress report. UP has received initial draft findings of its Scholars and Fellows annual report to be used as the basis for more explicit program goals. However, as a general practice, in its model of a university community devoted to mission-related research, UP will be tracking annual measures such as the number of researchers and students in homeland security-related disciplines, and the number of other individuals working in homeland security-related fields. In tracking such annual measures, UP will benchmark other agencies and the metrics they utilize. UP will also track the yearly success of the research efforts of its Centers and their contribution to DHS mission objectives. Such feedback would be from DHS operating directorates that indicate to what extent the Centers' research supports the war on terrorism. UP will hold its first comprehensive Center program review in September 2005 and a similarly structured evaluation for the education activities in FY 2006. These activities will involve extensive input from outside advisory groups and other stakeholders. Additionally, UP performance will be monitored by periodic progress reports, site visits and reverse site visits. Similar evaluations will be conducted for the student education programs as well.

Evidence: ?? Draft University Programs Strategic Plan ?? University Programs -- Program Execution Plan ?? Center Progress Reports ?? Draft ORISE Annual Report

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: Since the beginning (2003) of the scholarship and fellowship component of the Scholars and Fellows Programs, University Programs has paid particular attention to efficiencies including moving away from paper to electronic application processes, increasing the diversity of reviewers including those from Minority Serving Institutions, reducing the number of application reviewers by prescreening applications for eligibility, reducing the number of days to review to conform with acceptable practices used by other similar federal programs, and increasing the number of internal reviewers to ensure that mission relevancy is thoroughly incorporated. University Programs' Centers of Excellence are awarded through full and open competitions to ensure cost effectiveness. Furthermore, once grants are awarded, program performance reports are reviewed and analyzed by program managers to determine cost, schedule, scope, and quality performance. DHS Centers have taken on average six months to set up from the award of the grant, when the average in the federal government for similar centers is eighteen months. UP has awarded four Centers, with increasing numbers of proposals, in a limited amount of time. In addition, UP established four separate education programs with increasing numbers of students in two years, when the average in the federal government is one program per year. The nature of competitive grant making requires a certain amount of time to set up centers and student education programs. However, we have reduced the time it has taken for our award activities. UP has undertaken more activities with only a modest increase in staff. University Programs is now able to initiate benchmarking of the process of creating DHS Centers of Excellence and their subsequent performance with other similar programs. Additionally, the Scholars and Fellows Program is benchmarking program elements and processes with those of other Federal agencies that are similar in design. An efficiency measure for UP is the time it takes for the selection and announcement procedures for both students and centers. For both, our processing time is among the shortest, compared to other similar agency programs. DHS is moving towards incorporating an in-house grants management system. Once this is accomplished, an efficiency measure will be UP's costs of running its merit-based research and education competitions. [NEED TO ADD TARGETS AND BASELINES ON EFFICIENCY MENTIONED HERE]

Evidence: ?? University Programs Scholars and Fellows Annual Report ?? University Programs Report to Congress

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: DHS Centers' performance goals, objectives and targets are anticipated to receive positive evaluations in terms of specific responses to DHS mission goals and when compared to other similar federal programs. The strong response of the university community in applying for Center grants and for education programs attests to their attractiveness and to UP's effectiveness in articulating the national importance of the efforts. DHS Centers have taken on average six months to set up from the award of the grant, when the average in the federal government for similar centers is eighteen months. In addition, UP established four separate education programs in two years, when the average in the federal government is one program per year. The nature of competitive grant making requires a certain amount of time to set up centers and student education programs. However, we have reduced the time it has taken for our award activities. As previously discussed, UP is planning program reviews for September 2005 and 2006 for the research and education parts of the programs, respectively. These activities include evaluations from outside advisory groups and other stakeholders.

Evidence: ?? University Programs participation in National Science and Technology Council's Education and Workforce Development Subcommittee ?? National Institute of Allergy and Infectious Diseases' Centers (as models) ?? ORISE Draft Annual Report ?? UP Report to Congress

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: Both the University Centers of Excellence and the Scholars and Fellows Programs have been in existence for two years or less and are just beginning to have sufficient history on the programs accomplishments such that a meaningful review process would be appropriate. Consequently, the following independent review schedules are anticipated for each program: ?? An external evaluation of the progress of the current Centers of Excellence will take place in late FY2005 when the first three centers will each have at least one year of performance to evaluate. In FY2006, the Scholars and Fellows Programs will be evaluated, which will coincide with the completion of the first 3-year award terms. ?? An external evaluation of the entire University Programs is planned for FY2007. However, other external evaluations and monitoring has occurred in lieu of the formal external reviews to ensure the programs are on track. University Programs recently held a Centers of Excellence Symposium where feedback was asked and provided by participants on the effectiveness of these centers. Data from the feedback indicated that participants responded positively when asked how well the Symposium illustrated how university-based research can contribute to the defense and security of our Nation's assets. The University Program staff provides direct and regular monitoring of the Centers through regular interaction with Center leadership and researchers. Center advisory committees include S&T and other subject matter experts that also provide regular feedback. Additionally, each Center will undergo a comprehensive site visit review before the end of the second year of their grants, as part of the renewal process. In the education programs, student academic progress is tracked on a regular basis through formal communication with the students' faculty advisors, annual reports, and during internships. Formal written (as well as informal verbal) feedback is obtained from students and mentors during their required internships.

Evidence: ?? University Programs -- Program Execution Plan ?? Draft University Programs Strategic Plan ?? Work plans for University of Southern California, University of Minnesota, Texas A&M and University of Maryland Centers of Excellence