Detailed Information on the
National Institute of Justice Assessment

Improving grant monitoring to address OIG-identified weaknesses.

Ensuring that future budget requests explicitly link to the long-term and annual goals for the program.

Updating and refocusing NIJ's strategic plan to better communicate the program's investment priorities to the Congress and others.

Planning for an independent evaluation of key aspects of the program.

Measure: Number of fielded technologies

Explanation:

Measure: Number of citations of NIJ products in peer reviewed journals:

Explanation:

Measure: Total Number of NIJ electronic and hard copy documents/publications/other requested:

Explanation:

Measure: Average Days Until Closed Status for Delinquent NIJ Grants by FY

Explanation:

Measure: Percent Reduction in the DNA Backlog (Casework/Offender)

Explanation:The "casework" backlog refers to DNA evidence taken from crime scenes; the "offender" backlog refers to DNA samples taken from convicted offenders. Reduction of the casework backlog translates into additional criminal cases solved. Reduction of the offender backlog assists in assembling a database of known offenders and can lead to solving additional crimes for which the offenders have not yet been prosecuted.

Measure: Total number of NIJ electronic and hard copy documents/publications requested.

Explanation:

Measure: CODIS hits resulting from Convicted Offender funds

Explanation:

Measure: Number of new NIJ final grant reports, NIJ research documents, and grantee research documents published

Explanation:NIJ/Grantee/FGR

Is the program purpose clear?

Explanation: NIJ was established under the Omnibus Crime Control and Safe Streets Act of 1968 to "advance scientific research, development, and evaluation to enhance the administration of justice and public safety" (NIJ Mission Statement). This statement of purpose is derived from the statute that governs the Institute. The mission, as expanded upon in NIJ's governing statute, grants responsibility for research and development; evaluation of program effectiveness; development and demonstration of new approaches or techniques; identification of successful programs or projects that will improve the justice system; and dissemination of results.

Evidence: The Omnibus Crime Control and Safe Streets Act of 1968, as amended (42 USC § 3721-3723); NIJ Mission Statement, Strategic Goals and Research Priorities (www.ojp.usdoj.gov/nij/about.htm); NIJ Strategic and Performance Plan for Fiscal Years 2003-2005; OJP Organization and Program Information, Legislative Authority: Part 4, Ch 2, 16.560 (www.ojp.usdoj.gov/FinGuide/part4chap2.htm#nij).

Does the program address a specific and existing problem, interest, or need?

Explanation: In 1967, the President's Commission on Law Enforcement and Administration of Justice observed that "the greatest need is the need to know" and that "[W]e need to know much more about crime. A national strategy against crime must be in large part a strategy of search." (1967, pp. 273, 279). Follow-up legislation was introduced that same year and later enacted as the Omnibus Crime Control and Safe Streets Act of 1968. This Act established the Law Enforcement Assistance Administration (LEAA), an organization that was later replaced in part with NIJ. In establishing NIJ, Congress wrote, "there is an urgent need to encourage basic and applied research?? and to evaluate methods of preventing and reducing crime." The "need to know" persists. As Professor James Q. Wilson wrote in 1996, "The Federal Government has a unique opportunity to be the research and development (R&D) arm for law enforcement. It is a task that no city or State will undertake in any meaningful way, and is also one that with a few exceptions private foundations will not support." The "key role," he continues, "ought to be to do the one thing local authorities cannot and will not do on their own. That is to design and test new crime-control strategies." NIJ fulfills this role by providing objective, independent, evidence-based knowledge and tools to meet the challenges of crime and justice, particularly at the State and local levels. NIJ provides a public good by conducting research, development, and evaluation that States and localities will not fund on their own. Because of its expertise and long-standing participation in research and development in support of the forensic sciences, NIJ also is tasked with the management of programs to improve the capacity and capabilities of the Nation's State and local crime laboratories.

Evidence: James Q. Wilson quotes from "What, If anything, Can the Federal Government Do About Crime?" available at www.ncjrs.org/pdffiles/166609.pdf; Omnibus Crime Control and Safe Streets Act of 1968, as amended (42 USC § 3721-3723); NIJ, Office of Science and Technology, Research, Development, Testing and Evaluation (RDT&E) Process; Census of State and Local Law Enforcement Agencies, 2000 (www.ojp.usdoj.gov/bjs/abstract/csllea00.htm); Electronic Crime Needs Assessment for State and Local Law Enforcement, March 2001 (www.ojp.usdoj.gov/nij/pubs-sum/186276.htm); Forensic Sciences: Review of Status and Needs, February 1999 (www.ojp.usdoj.gov/nij/pubs-sum/173412.htm); Inventory of State and Local Law Enforcement Technology Needs to Combat Terrorism, January 1999 (www.ojp.usdoj.gov/nij/pubs-sum/173384.htm); Report to Congress on the National Law Enforcement and Corrections Technology Center, July 21, 2004.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: NIJ is the research, development, and evaluation (RD&E) agency of the U.S. Department of Justice and is the primary means for OJP and DOJ to achieve research-related goals. Overlap exists with some other Federal agency missions. Recognizing these commonalities of mission, NIJ has engaged these other agencies and developed an extensive set of formal and informal relationships to coordinate research activities and minimize inappropriate duplication of effort. For example, NIJ participates in the Technology Policy Council (TPC), which includes representation from the Departments of Homeland Security, Treasury, and Transportation, and others with technology interests and applications relevant to law enforcement and public safety. Also, at the inception of the Department of Homeland Security (DHS), NIJ worked closely with DHS to identify research areas most relevant to everyday law enforcement that were most appropriately maintained within NIJ and those areas more exclusively relevant to counterterrorism that would be transferred to DHS. As a result of this review of DHS and NIJ activities, NIJ got out of the business of developing chemical and biological protective and detection technologies and moved three congressionally directed counterterrorism centers to DHS. NIJ has participated in the "What Works Repository," an innovative effort sponsored by its parent, the Office of Justice Programs, to apply rigorous criteria for evaluating the effectiveness of State and local criminal justice grant projects. While NIJ's participation has mitigated redundancy of effort, this activity should be supported and funded within NIJ in the future.

Evidence: National Academy of Sciences reports: "Fairness and Effectiveness in Policing" pp. 327 and 30, "Firearms and Violence: A Critical Review," pp. 2-3, "Advancing the Federal Research Agenda on Violence Against Women," p. 96 (www7.nationalacademies.org/claj/CLAJ_Publications.html); NIJ's Border Research and Technology Center, Affiliated Agencies: (www.justnet.org/brtc/affil.html) "What Works Repository Presentation for OMB," February 2, 2005. Sampling of Technology Policy Council Agenda and Minutes, provided by NIJ.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: NIJ's investments in its research, development, and evaluation (RD&E) program rely on an external, competitive, merit-based, peer review process managed by specialized NIJ Program Managers and comprising panels of external, subject-area experts to ensure program effectiveness and efficiency. Merit review by peers has been recognized as a best practice for administering RD&E programs. Independent external groups (such as National Academy of Sciences, National Governors Association, International Association of Chiefs of Police, and others) provide scrutiny and advice to achieve portfolio diversity and maintain programmatic goals.

Evidence: OMB/OSTP Research and Development Criteria (June 2003); OMB Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies; Federal Research: Peer Review Practices at Federal Science Agencies Vary, GAO/RCED-99-99 (www.gao.gov/archive/199/rc99099.pdf); NIJ Office of Research and Evaluation Program Review Overview; 2005 Consolidated Appropriations Act, Conference Report;

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: NIJ's beneficiaries exert strong influences on the resource investments because NIJ's RD&E activities rely extensively on practitioner-driven needs assessments. NIJ then attempts to ensure that resources are being targeted effectively to achieve programmatic goals by including clear statements of program purposes in competitive, externally peer-reviewed solicitations for research and evaluation. NIJ's Office of Science and Technology accomplishes these tasks and helps to ensure relevance, quality, and performance through their RDT&E process. NIJ's Office of Research and Evaluation accomplishes this through internal Program Reviews and external National Academy of Sciences evaluations. For example, NIJ's recent solicitation concerning community corrections prioritized research on supervision strategies/case management and revocation policies because these were cited by the Community Corrections Research Network (a collection of recognized experts in probation and parole research, practice, and policy) in a previous meeting sponsored by NIJ. However, NIJ is limited in its ability to optimally allocate research funds because of earmarking. Congressional earmarking notwithstanding, NIJ holds earmarks to high standards of external peer review to ensure that the maximum benefits possible accrue to intended beneficiaries.

Evidence: OMB Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies (February 22, 2002); NIJ Technical Assistance and Support Program, Standard Technical and Practitioner Review Questions; NIJ, Office of Science and Technology Research, Development, Testing, and Evaluation Process; Report to Congress on National Law Enforcement and Corrections Technology Center July 21, 2004; NIJ Office of Research and Evaluation Program Review Overview; Community Corrections Network and Solicitation; Other Documented Solicitations with Needs Assessments; 2005 Consolidated Appropriations Act, Conference Report.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: Because NIJ conducts substantial amounts of basic and applied research, long-term outcomes are uncertain and impose measurement difficulties. NIJ has framed its long-term performance measures around process metrics, including those built around OMB's Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies; and output metrics (that presage long-term outcomes of basic and applied research): "Percent of NIJ RD&E applications subjected to external peer review" provides a measure of the extent to which NIJ-funded activities are subjected to the review of research practitioners and experts, whose input is important for guiding such investments. "Average score by the public on the Customer Satisfaction Index Survey given to website visitors" is a proxy measure for customer satisfaction with NIJ-funded research products. "Number of new NIJ grantee, Final Grant Report research papers/documents published/online" is a good indicator for NIJ's level of research output.

Evidence: NIJ Mission Statement & Strategic Goals and Research Priorities (www.ojp.usdoj.gov/nij/about.htm#strat); NIJ Strategic and Performance Plan for Fiscal Years 2003-2005; Memorandum to Deborah Daniels from Sarah Hart Regarding Strategic Plan for NIJ, August 15, 2002; DOJ Strategic Plan (www.usdoj.gov/jmd/mps/strategic2003-2008/index.html).

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: NIJ engages in ongoing portfolio review to assess and validate the continuing relevance of research goals and priorities, and further define program-specific requirements and outcomes. The long-term targets established are ambitious and hold NIJ to a strong standard for performance over the timeframes specified.

Evidence: NIJ Strategic and Performance Plan for Fiscal Years 2003-2005; See Performance Measures section of this PART.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: NIJ tracks several annual performance measures that tie to fundamental outcome objectives of the program: to conduct meaningful research, issue products that provide information and results from NIJ research, develop technology that will be used by law enforcement, and satisfy customers of its research products. In each case, the measures are discrete, quantifiable, and measurable.

Evidence: NIJ Strategic and Performance Plan for Fiscal Years 2003-2005; DOJ Strategic Plan (www.usdoj.gov/jmd/mps/strategic2003-2008/pdf.html).

Does the program have baselines and ambitious targets for its annual measures?

Explanation: Baselines have been established for each of NIJ's annual performance measures. NIJ also maintains annual targets that are quantified and designed to capture the progress toward management and programmatic goals. The targets are reasonably ambitious and, in most cases, signal incremental improvement in performance. In some cases, decreasing targets either reflect NIJ efforts at substitution (e.g., moving from print to electronic dissemination) or projections of decreased funding for specific program areas (e.g., counterterrorism prototype development and projects on new forensic DNA markers).

Evidence: NIJ Strategic and Performance Plan for Fiscal Years 2003-2005.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: All NIJ partnership documents include program goals and milestones relevant to NIJ's mission and R&D objectives. NIJ memorializes agreements with other Federal agencies using MOUs or interagency agreements, which outline NIJ's mission and role and how the other participating agency's mission(s) and role(s) complement NIJ's within the context of the relevant partnership. NIJ solicitations outline R&D objectives and, where relevant and appropriate, measures, outputs, and program outcomes for which grantees are held responsible. For contracts, the statement of work places all proposed tasks within the context of NIJ's overall mission or ties them to a specific R&D activity for which NIJ seeks consulting or other relevant services. Project quality and timeliness are monitored through milestones for task completion. Final products are reviewed for quality and relevance. As an example, NIJ seeks to create relevant, high quality social science knowledge about community corrections. In October 2004, the Community Corrections Research Network identified the need for knowledge on revocation policies and case management/offender oversight. NIJ then released a solicitation asking for research on those particular aspects of Community Corrections. Independent, merit-based peer review then evaluated these applications on their quality, expected impact, and dissemination plan, and NIJ staff then made recommendations to the Director for funding. The use of open, competitive, peer-reviewed solicitations for research, prioritized according to the needs of practitioners and policymakers, is a recognized method for creating high-quality, relevant information. Grantees are held to their proposals by Special Conditions and are committed to the annual and long-term goals of the program.

Evidence: Examples of Memoranda of Understanding submitted to OMB; Solicitation Calendar and Example NIJ 2005 Solicitations submitted to OMB; Memorandum of Understanding between NIJ and NAS submitted to OMB; NIJ Technical Assistance and Support Program, Standard Technical and Practitioner Review Questions submitted to OMB.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: NIJ evaluations do not satisfy PART definitions of scope, quality, independence, and regularity. Until recently, GAO audits and OIG findings served as primary sources of independent evaluations. In addition, NIJ has commissioned evaluations and reports from scientific advisory groups and engaged criminal justice professional associations, scientific bodies, and practitioner groups. NIJ's response to these audits and evaluations included a number of critical management initiatives and structural changes detailed elsewhere that include the consolidation of evaluation activities within one Division and an evaluability assessment strategy to target funding decisions. NIJ evaluates its dissemination processes through customer satisfaction surveys. In 2004, NIJ's science and technology program established the Research, Development, Testing, and Evaluation (RDT&E) process to document needs and requirements and translate them into validated program objectives that can be tested against stated requirements. This process establishes a structure for conducting independent evaluations. The Grant Progress Assessment Program was also established to conduct project-level audits and collect performance data from State and local crime laboratories. On the social science side, NIJ has used and will continue to rely on the National Academy of Sciences for independent evaluative studies, reviews, and recommendations regarding NIJ's specific research areas. To date, these efforts have included studies on violence prevention, gambling, policing, firearms, terrorism, the death penalty, and research methodology.

Evidence: Office of Inspector General Reports (05-02, 02-20) (www.usdoj.gov/oig/igauojp1.htm); GAO reports (GAO-03-1091, GAO-02-309, GAO-04-666, GAO-04-740, GAO-04-198) (www.gao.gov/aac.html); NFSTC Audit (www.nfstc.org/audit_assess_gpa.htm); National Academy of Sciences Reports (www7.nationalacademies.org/claj/CLAJ_Publications.html); NIJ, Office of Science and Technology, Research, Development, Testing, and Evaluation (RDT&E) Plan Process submitted to OMB.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: NIJ's internal DOJ budget requests attempt to link resources requested to performance goals. However, the budget requests to OMB and Congress do not provide sufficient information about how the resources requested will assist NIJ in meeting key annual and long-term performance goals. The Department's performance plan and budget present separate requests for core NIJ research and development activity and other NIJ-administered initiatives, such as the Administration's DNA Initiative. While the core NIJ budget is well justified, the Department's budget justifications to OMB and to Congress provide little information about the linkage between budget and performance goals. In fact, little performance information is presented at all. For other NIJ-administered initiatives, such as the DNA Initiative, considerable performance information is presented in the submissions, providing a greater linkage to the resources requested.

Evidence: 2006 DOJ Congressional Budget Submission; 2006 DOJ OMB Budget Submission; NIJ Strategic and Performance Plan for Fiscal Years 2003-2005.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: While significant performance information for NIJ has been lacking from prior DOJ budget submissions, NIJ is working with the Department to ensure that future budget requests include key NIJ performance measures. NIJ also is working with OMB to develop a plan for regular, independent evaluations of its programs to evaluate their effectiveness.

Evidence: 2005 DOJ Congressional Budget Submission; 2005 DOJ OMB Budget Submission; NIJ Strategic and Performance Plan for Fiscal Years 2003-2005.

If applicable, does the program assess and compare the potential benefits of efforts within the program and (if relevant) to other efforts in other programs that have similar goals?

Explanation: NIJ participates in a number of RD&E coordinating entities such as DOJ's Technology Policy Council and the interagency Technical Support Working Group. NIJ's participation in such entities provides an opportunity to ensure that NIJ programs are yielding significant benefits compared to other, related programs. When NIJ solicits targeted research or technology, it employs external peer reviews to compare strengths and weaknesses among competing solutions to the target problem.

Evidence: For example, see Technical Support Working Group (www.tswg.gov/tswg/home/home.htm); 2005 Explosives Detection and Remediation Panel and Funding Recommendations; 2005 Electronic Crime Panel and Funding Recommendations submitted to OMB; 2005 ORE Continuations (Evaluations of OJJDP Discretionary Fund Project) Panel and Funding Recommendations submitted to OMB.

Does the program use a prioritization process to guide budget requests and funding decisions?

Explanation: NIJ prioritizes programmatic objectives in its planning and operational actions. In planning, NIJ uses Program Reviews, independent subject-area experts and executive sessions, and the Research, Development, Testing, and Evaluation (RDT&E) process to analyze tradeoffs between alternative funding options. These analytic processes help determine what areas are deserving of increased attention and what RD&E questions need to be addressed. In operation, NIJ uses competitive, independent peer review and managerial overview and analysis to prioritize grant applications for funding decisions by the Director. In this way, applications in response to targeted solicitations are screened to prioritize those that will provide the most relevant and useful knowledge and tools. For example, NIJ's Office of Research and Evaluation used a prioritized research agenda created by practitioners and policymakers on its Community Corrections Research Network to guide its recent solicitation for research in Community Corrections. Similarly, NIJ's Office of Science and Technology used a prioritized agenda created by its Forensic DNA Testing Working Group to guide funding decisions.

Evidence: NIJ Strategic and Performance Plan for Fiscal Years 2003-2005; NIJ, Office of Science and Technology, Research, Development, Testing, and Evaluation Process (document submitted by NIJ); NIJ Funding Decision Process (www.ojp.usdoj.gov/nij/faq_funding.htm#sandf2); Community Corrections Network and Solicitation (document submitted by NIJ).

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: NIJ measures its ability to create relevant knowledge and tools, disseminate those products, and uphold high standards of management by making use of performance information that ties to NIJ's Strategic and Performance Plan. Sources of data include the OJP Grant Management System, quarterly financial reports that track grant expenditures, semi-annual progress reports that detail progress, Pipeline (NIJ's data management system), NIJ's Web Management Tool, the Communications Department, and monthly telephone contacts by program managers. NIJ establishes oversight plans for grantees and guidelines for grant managers that serve to measure and improve the performance of program partners. All projects contain task-based milestones to monitor progress. NIJ holds regular cluster meetings of grantees in order to monitor performance and improve grantee problem definition, program design, data collection and analysis, and dissemination of findings.

Evidence: NIJ Strategic and Performance Plan for Fiscal Years 2003-2005; NIJ Evaluation Research Oversight Plan; NIJ Guidance Regarding the Evaluability Assessment Process; Grant Progress Assessment Program documentation (www.nfstc.org/audit_assess_gpa.htm); Example Program Review: Community Corrections; DNA Research & Development Documents (submitted to OMB).

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: NIJ enforces the guidelines in OJP's Office of the Comptroller's Financial Guide 2005, which incorporates provisions of OMB circulars and governmentwide common rules applicable to grants and cooperative agreements. NIJ ensures cost accountability with its program partners through its rigorous external peer review process, which is conducted on a pre-award basis. As part of this process, experts measure the cost effectiveness of proposed projects. NIJ seriously considers this peer review recommendation and ensures that the recommended applicant has no other financial delinquencies with the government prior to issuing an award. All projects contain task-based milestones to monitor progress and deviations from plans. Departures from the original plan are documented in writing and, if financial redirection is necessary, authorized by NIJ through grant adjustment notices. NIJ holds regular cluster meetings of grantees in order to monitor performance and improve grantee problem definition, program design, data collection and analysis, and dissemination of findings. NIJ holds its program partners/grantees accountable with special grant conditions that regulate and withhold the flow of grant awards based on performance. Contracts contain Quality Assurance Surveillance Plans (QASP) that grade the contractor performance and cost control. If standards are not met, NIJ withholds a predetermined percent of funding. Each contract has language that includes procedures for acceptance and rejection of products or services. NIJ also uses Blanket Purchase Agreements (BPAs) that involve cost ceilings that can be modified only by the Contracting Officer (CO). Contractors may not bill above the ceiling or perform work not authorized by the COTR or CO. NIJ holds its managers accountable through performance work plans; timelines for posting, processing, and awarding grant information and grant monies; and supervisor review. The managers held to these standards include both Office Heads (Science & Technology and Research & Evaluation) as well as all Division Chiefs. These standards include links to annual and long-term performance goals as well as dated milestones. Grant managers comply with the policies and procedures as delineated in the OJP Grants Management Manual including programmatic, administrative, and financial review of grantee performance in addition to regular contact, interviews, and site visits where appropriate. Additionally, on June 30, 2004, the Attorney General transmitted decisions on the SES Performance-based Pay System to OPM and OMB with the Human Capital report, including a generic work plan for all Departmental SES members, with accompanying Performance "contract" that must explicitly relate to the Department's, the President's or the AG's defined goals. On December 10, 2004, DOJ obtained approval for the SES and GS/Prevailing Rate performance orders. DOJ has completed the application package for OPM/OMB certification of the DOJ SES Performance Management and Compensation Plan. DOJ components implemented five-level performance plans for all SES. New SES and manager plans include cascading tasks/assignments that are linked to the DOJ Strategic Plan and the PMA. By Dec 30, 2004, all DOJ components certified to the Attorney General that all SES and direct report performance work plans are in place. By the end of the December 2004, the Department had performance appraisals for most of the work force that: link to agency mission, goals and outcomes in DOJ's Strategic Plan; hold employees accountable for results appropriate to their level of responsibility; effectively differentiate between various levels of performance; and provide consequences based on performance.

Evidence: All managers are subject to the new DOJ Performance Management System. Attorney General's Memorandum dated August 10, 2004 - "Performance Management Plans for SES Members and Managers" www.usdoj.gov/jmd/ps/memagpma-ses1.htm NIJ Evaluation Research Oversight Plan; OJP Financial Guide 2005 (www.ojp.usdoj.gov/FinGuide/); OJP Grants Manager's Manual, Chapter 8: Performance Monitoring; NIJ Application Guidelines (www.ncjrs.org/pdffiles1/nij/202948.pdf); NIJ Special Conditions for Grants and Examples of 2005 NIJ Grant Solicitations (submitted by NIJ).

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: NIJ ensures efficient and appropriate administration of funds with enforced schedules and monthly oversight by senior management. OJP's legal and accounting departments ensure that all solicitations and grants adhere to restrictions that may be set in the appropriations process; external peer reviews ensure that all grant applications are responsive; and grant managers oversee compliance by grantees with all goals, objectives, and activities associated with individual research grants. NIJ manages a comprehensive budget planning process that nominates funding for specific projects and ensures (over the life of the project) that funds are used for the intended purpose. NIJ participates in weekly budget reconciliation exercises with OJP's Budget and Management Office to review accounts including awards, contracts and reimbursable agreements. This year OJP implemented a new policy that improves the collection of quarterly financial reports, and expedites grant closeout and deobligation of unexpended funds. Through this initiative, NIJ collected 585 financial reports, closed over 100 grants, and established a procedure to continue this process as part of standard grant administration operating procedures. While the share of unobligated funds at year-end has been significant in recent years, 2003 and 2004 have seen dramatic reductions in these balances.

Evidence: OJP Financial Guide 2005 (www.ojp.usdoj.gov/FinGuide); NIJ Obligation of Direct Appropriated Funds: 2001 BA: $69.8M; 2001 Unob. End of Year Bal.: $21.2M; 2002 BA: $54.9M; 2002 Unob. End of Year Bal.: $23.7M; 2003 BA: $59.5M; 2003 Unob. End of Year Bal.: $4.0M; 2004 BA: $47.5M; 2004 Unob. End of Year Bal.: $1.1M. Grant Award Special Conditions, documentation supplied by NIJ.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: In January 2003, NIJ reorganized, streamlining its structure from three operating offices to two, consolidating divisions, and reducing management layers for improved coordination and program execution. Key process initiatives designed to improve program efficiencies have included the development of NIJ's Pipeline system, a comprehensive project-tracking database for use by all NIJ personnel; the institution of electronic processing for all grant awards (reducing the time to award approval of approximately 90 to 60 days - reducing the processing time to close out projects and recover unused funds from 511 in 2003 to 275 in 2004. These process metrics have recently been incorporated into NIJ's Strategic and Performance Plan to assess the timeliness, reach, and impact of NIJ's RD&E and dissemination efforts. NIJ also has streamlined its procurement processes by greatly increasing its use of pre-qualified vendors with approved Government rates on various GSA schedules or via Blanket Purchase Agreements for repetitive requirements. In this way, NIJ ensures the robustness of competitions for the procurement of a variety of services while dramatically reducing the time to contract award. Contracts are awarded in as little as 3-6 months versus more complex open competition processes, including sometimes unqualified applicants, that require up to 12 months to complete. Activities to increase NIJ's return on RD&E investments incorporate both agencywide and program-level approaches. NIJ recently instituted evaluability assessments to attain more efficient and effective evaluation investments. At the project level, NIJ subjects all grant applications to independent peer review. This process analyzes proposals in terms of cost-benefit, budget relative to effort, and use of non-Federal resources. Applications approved for award receive a secondary analysis via OJP's Comptroller of cost reasonableness and other factors indicating the fiscal integrity of the project. NIJ is tracking the cost savings during its shift from print to electronic dissemination. NIJ has implemented a number of initiatives targeted to increase the efficiency and effectiveness of its RD&E and dissemination activities. Initiatives have focused on management controls, processes, and the overall return on NIJ's investments in the development of knowledge and tools to enhance the administration of justice and improve public safety. The OJP Competitive Sourcing study will enable the Department of Justice to show continued progress in advancing the President's Management Agenda and the commitment to government efficiency and effectiveness. The OJP study is anticipated to affect 115 full time equivalent personnel throughout NIJ's parent organization, the Office of Justice Programs, who are involved with the functions and activities of Grant Management Support Services. This 18-month study is scheduled to be completed in January 2006. NIJ's grants management support activities are included in this competitive-sourcing (A-76) activity that is designed to achieve program-wide efficiencies in NIJ's core awards management functions. Critical program support administrative and program cost centers are included in the scope of work, such as peer review logistics, application and awards processing, and the drafting and editing of products for dissemination to the public.

Evidence: NIJ Grant Application Guidelines (www.ncjrs.org/pdffiles1/nij/202948.pdf); Funding Decision Process (www.ojp.usdoj.gov/nij/faq_funding.htm#sandf2); NIJ Guidance Regarding the Evaluability Assessment Process; PWS solicitation for competitive sourcing, reference number 2004PR784 (www.fedbizopps.gov); FY03 NIJ reorganization under "Organization" (www.ojp.usdoj.gov/nij/about.htm#org); Awardee Correspondence.

Does the program collaborate and coordinate effectively with related programs?

Explanation: NIJ's leverages funds through a tradition of collaboration and coordination with public and private sector entities. NIJ representatives participate on coordinating committees ranging from the White House Office of Science and Technology Policy to the United Nations Office on Drugs and Crime. NIJ receives substantive input from independent planning and advising groups including the National Academy of Sciences (NAS), the National Governors Association and the International Association of Chiefs of Police. NIJ's partnership with NAS has produced 14 publications, which often include policy and program recommendations. NIJ often participates in collaborative efforts in planning and evaluating research programs such as a recent Conference on Suicide Terrorism Research cosponsored with the Department of Homeland Security, an upcoming planning meeting on School Violence Prevention Research, and a recently completed conference on less-lethal technology.

Evidence: NIJ Partnerships in FY2004 (documents submitted by NIJ); Examples of Memoranda of Understanding (documents submitted by NIJ).

Does the program use strong financial management practices?

Explanation: Auditors were unable to express an opinion on the Department's 2004 financial statements because of an auditor's disclaimer on the Office of Justice Programs' 2004 financial statements. The disclaimer was the result of an issue with grant accruals, or the estimates of unspent funds. These estimates can be prepared in a variety of ways depending upon the assumptions made about to grantee spending behavior--- the auditors required OJP to devise a new procedure for producing such estimates. OJP is working with auditors to resolve the issue in time for the 2005 financial statements. Conversations with DOJ and OIG staff indicate, barring an unforeseen auditor issue, that OJP is on track to resolve this issue. NIJ adheres to the guidelines in OJP's Office of the Comptroller's Financial Guide 2005 that incorporates the provisions of OMB circulars and governmentwide common rules applicable to grants and cooperative agreements. All internal transfers of money are tracked through a budget obligation process. Electronic systems that require grantees to enter specific accounting data, passwords, and other criteria ensure security of financial draw downs. All OJP/NIJ financial systems meet statutory requirements. All financial processes and systems are compliant with law and regulations related to financial management. With OJP's Office of the Comptroller, NIJ has taken steps to ensure future compliance with financial management requirements.

Evidence: DOJ 2004 Performance and Accountability Report, Independent Accountants' Report on Financial Statements, www.usdoj.gov/ag/annualreports/pr2004/P3/p009-10.pdf Conference call with DOJ financial management and OIG officials and staff, June 16, 2005; OJP Financial Guide 2005 (www.ojp.usdoj.gov/FinGuide).

Has the program taken meaningful steps to address its management deficiencies?

Explanation: With OJP's Office of the Comptroller, NIJ has implemented new policies on grantee financial reporting. New automated financial reporting and tracking procedures will freeze grantee funds and prevent further grant draw downs if financial reporting is delinquent. NIJ has expanded its use of Pipeline, the grant program monitoring information system, and uses it to record all grantee correspondence and grant program oversight activities. NIJ also has taken specific steps to respond to GAO findings on evaluation grant monitoring activities. NIJ has initiated policies of front-end evaluability assessments to improve the likelihood of obtaining a rigorous outcome evaluation and mid-evaluation reassessments to identify evaluations that are faltering. NIJ also has revitalized its dissemination process in order to focus more directly on practitioner-oriented publications and electronic dissemination. NIJ has instituted policies for evaluation grant oversight; revitalized a series of program reviews intended to increase efficiency and high-quality, relevant research; and recommitted itself to ensuring that all employees develop, and are held to individual development plans. NIJ has an established annual target of 100 percent of employees receiving training, as well as complete resolution of any findings of waste, fraud, or abuse. Recently, NIJ established separate Technical and Practitioner Review Criteria to determine which research products are worthy of publication.

Evidence: Memorandum to Glen A.. Fine from Tracy A. Henke Regarding Response to OIG's Audit Report; GAO Reports (GAO-03-1091, GAO-02-309, GAO-04-666, GAO-04-740, GAO-04-198) (www.gao.gov/aac.html); NIJ Evaluation Research Oversight Plan; NIJ Strategic and Performance Plan for Fiscal Years 2003-2005; NIJ Technical Assistance and Support Program, Standard Technical/Practitioner Review Questions; OIG reports (05-02 and 02-20) (www.usdoj.gov/oig/igauojp1.htm); Awardee Correspondence: Pipeline Reports.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: NIJ has implemented strong oversight practices for its three formula grant activities (DNA Initiative-Forensic Casework; DNA Initiative-Capacity Enhancement; and Coverdell Forensic Sciences Improvement Act Grants). Since 2002, NIJ has provided DNA audit services to grantee and other State and local laboratories through its cooperative agreement with the National Forensic Science and Technology Center (NFSTC). NIJ requires grantee laboratories to meet quality standards established by both the Federal Bureau of Investigation (FBI) and the American Society of Crime Laboratory Directors as a condition of their receipt of federal funds for DNA capacity enhancement. The audit services provided via NFSTC review operating DNA laboratories for their compliance with FBI and ASCLD standards. NIJ has performed 172 such audits between 2002 and 2004. As an extension of the DNA quality standards audit activities, NIJ established the Grant Progress Assessment (GPA) program at NFSTC in November of 2004. Through the GPA program, NIJ assesses grantee progress in meeting program goals and objectives, reviews program documentation, and assesses the impact of the grant funding. In addition to the GPA program, face-to-face contact opportunities also occur at NIJ's annual DNA Grantees meeting. This meeting connects forensic capacity enhancement funding recipients with NIJ-funded researchers, and provides NIJ with opportunities to address program issues with grantees from across the Nation. Program managers are in frequent contact with grantees. In addition, NIJ receives quarterly financial reports and periodic progress reports from all grantees. Program managers access expenditure data at the grantee level via OJP's Integrated Financial Management Information System as a further check on project-level financial accountability and performance against project targets and milestones.

Evidence: OJP Grants Management Manual; NIJ Evaluation Research Oversight Plan; NIJ Technical Assistance and Support Program; Standard Technical and Practitioner Review Questions; Data Resources Program (www.ojp.usdoj.gov/nij/dataprog.htm); Grant Progress Assessment Program (www.nfstc.org/audit_assess_gpa.htm); Paul Coverdell Forensic Science Improvement Grant Program, Grant Announcement.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: NIJ primary grant programs are the three formula-based grant programs: the DNA Initiative-Forensic Casework, the DNA Initiative-Capacity Enhancement, and the Coverdell Forensic Sciences Improvement Act grant programs. The DNA Initiative was first funded in FY 2004. Program funding announcements provide instructions to all grantees as to program activity and performance reporting requirements. Baseline data are being established and gathered as reports from the first year of funding become available. NIJ's Grant Progress Assessment activity will support the consolidation and analysis of performance information. For the Coverdell program, first funded during FY 2002, program announcements provide instructions to grantees as to how performance data must be collected and reported. A Congressionally mandated Coverdell annual report is available to the public via NIJ's Web site and contains annual updates regarding grantees' use of Coverdell funds.

Evidence: Coverdell Annual Report to Congress (www.ojp.usdoj.gov/nij/topics/forensics/funding/nfsia/fy2004_report.htm); Paul Coverdell Forensic Science Improvement Grant Announcement; Grant Progress Assessment Program (www.nfstc.org/audit_assess_gpa.htm) ; DOJ Strategic Plan (www.usdoj.gov/jmd/mps/strategic2003-2008/pdf.html, see Goal III - Assist State, Local, and Tribal Efforts to Prevent or Reduce Crime and Violence); FY 2004 Performance and Accountability Report (www.usdoj.gov/ag/annualreports/pr2004/TableofContents.htm, see Strategic Goal 3) and Management's Discussion and Analysis.

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: The core of NIJ's grant award decision-making process is the competitive, merit review process (including external peer review). In 2004, 99 percent of all grant applications were subject to external, peer review. Each solicitation is posted on the Internet for open competition by agencies in the public, for-profit, and private not-for-profit sectors and includes a set of selection criteria to evaluate the applicant's understanding of the problem; research capability, demonstrated productivity, and experience; the quality and technical merit of the proposal; the impact of the proposed project; the budget; and the dissemination plan. Independent peer reviewers (both researchers and practitioners) evaluate each application using these criteria and reach consensus on the quality of and need for the proposed research. In addition to these reviews, NIJ management considers and evaluates applications in the context of portfolio diversity and budget restrictions before making final awards. NIJ consistently reaches out to new applicants through presentations at professional conferences where the application process, selection criteria, and forthcoming solicitations are discussed.

Evidence: Funding Decision Process (www.ojp.usdoj.gov/nij/faq_funding.htm#sandf2); Peer Review Documents (submitted by NIJ); NIJ Grant Application Guidelines (www.ncjrs.org/pdffiles1/nij/202948.pdf); NIJ Funding Opportunities (www.ojp.usdoj.gov/nij/funding.htm); Solicitation Calendar and Examples of NIJ 2005 Solicitations (submitted by NIJ).

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: DOJ's OIG has indicated that improvements are needed in the Office of Justice Programs' grant management and oversight activities. While steps are being taken to remedy oversight weaknesses, the OIG has not yet confirmed resolution of this weakness. For its part, NIJ is taking steps in conjunction with OJP to attempt to improve grant oversight. Grant managers are requested to comply with the OJP Grants Manager's Manual that includes responsibilities of programmatic, administrative, and financial review. They also conduct frequent contacts, interviews, and site visits where appropriate. NIJ adheres to the guidelines in OJP's Office of the Comptroller's Financial Guide 2005 that incorporate the provisions of OMB circulars and governmentwide common rules applicable to grants and cooperative agreements. Semi-annually, NIJ reviews selected evaluation projects requiring intensive monitoring. It also has standardized criteria for grantee final report screening and rating. NIJ uses the National Archive of Criminal Justice Data along with the NIJ Data Resources Program to test reproducibility of findings. NIJ uses Pipeline, its internal grant management information system, to track grant manager contact with grantees. NIJ maintains access to its semi-annual progress reports, quarterly financial reports, and annual audits that are submitted to the Office of the Comptroller.

Evidence: DOJ 2004 Performance and Accountability Report, Top Management Challenges: www.usdoj.gov/ag/annualreports/pr2004/P4/p15-44.pdf OJP Grants Manager's Manual; NIJ Evaluation Research Oversight Plan; NIJ Technical Assistance and Support Program: Standard Technical and Practitioner Review Questions; Data Resouces Program (www.ojp.usdoj.gov/nij/dataprog.htm); GPA Program (www.nfstc.org/audit_assess_gpa.htm).

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: The Department of Justice (DOJ) includes a select example of Department-wide performance measures in its strategic and annual performance plan documents, which are available to the public on the DOJ Web site, although grantee performance information is not regularly made available to the public by NIJ for all major programs. NIJ regularly announces all grant awards online. NIJ deposits all grantee final reports with the National Criminal Justice Reference Service, announces final reports, and makes them available online in the Research Review. Data resulting from grantee research is deposited and made publicly available through the National Archive of Criminal Justice Data. An NIJ measure related to the DNA Backlog Reduction program is currently included in the DOJ Strategic Plan and in the Department's annual Performance and Accountability Report. Program output data for the Paul Coverdell National Forensic Sciences Improvement Act program is available via NIJ's Web site; as this is a relatively new program (first funded in FY 2002), annual performance data are being collected on an ongoing basis for analysis against established targets and the results of this analysis will be published as data become available from all participating jurisdictions. Similar data will be published for DNA-Initiative activities (Forensic Casework and Capacity Enhancement grant programs that were first funded in FY 2004) as program performance data currently being collected become available from all participating jurisdictions for analysis against baselines and established program targets.

Evidence: Award information from 1995 - 2004 (www.ojp.usdoj.gov/nij/funding.htm); National Criminal Justice Reference Service (www.ncjrs.org/); National Criminal Justice Data Archive (www.icpsr.umich.edu/NACJD/welcome.html); Research Review (www.ojp.usdoj.gov/nij/rr/index.html); Data Resources Program (www.ojp.usdoj.gov/nij/dataprog.htm); FY 2004 Performance and Accountability Report (www.usdoj.gov/ag/annualreports/pr2004/TableofContents.htm, at Strategic Goal 3); DOJ Strategic Plan for Fiscal Years 2003 - 2008 (www.usdoj.gov/05publications/05_4.html, at Goal III: Assist State, Local, and Tribal Efforts to Prevent or Reduce Crime and Violence); Coverdell Report (www.ojp.usdoj.gov/nij/topics/forensics/funding/nfsia/fy2004_report.htm).

For R&D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain program quality?

Explanation: Because of congressional direction and earmarks, NIJ is unable to allocate funding for all of its program activity using a clear, competitive method. NIJ awarded more than $100 million noncompetitively in FY 2004 (from discretionary and reimbursable activities). NIJ requires an objective, merit-based review of earmark proposals to document input regarding relevance and quality from scientists and practitioners. These merit-based reviews improve their quality, ensure collaboration where relevant and appropriate, and minimize redundancy of proposed research. All final products are subjected to an additional external technical review to document the quality of deliverables. NIJ employs various other checks to ensure the technical merit, quality, and relevance of its awards to other Federal science agencies. Through these interagency activities, NIJ engages the technical expertise of a partner entity to perform research or support NIJ in managing research activities of mutual relevance. Interagency relationships that involve the allocation of program funds to partner agencies receive an Economy Act (31 USC § 1535) review by OJP's General Counsel to ensure that required services cannot be provided more conveniently and cheaply through other means.

Evidence: NIJ Guidelines for Submitting Applications (www.ncjrs.org/pdffiles1/nij/202948.pdf) NIJ 2005 report to the Committees on Appropriations on NIJ's management plan for aligning the National Law Enforcement Corrections Technology Center (NLECTC) system within the NIJ Research, Development, Test and Evaluation (RDT&E) process.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: With sustained incremental improvement, as seen in NIJ's recent performance, NIJ is on track to achieve its long-term performance goals. (NIJ receives a "Large Extent" instead of a "Yes" for this question because it has developed a baseline for its customer satisfaction index, but reported no actual progress.)

Evidence: See Performance Measures Section of PART.

Does the program (including program partners) achieve its annual performance goals?

Explanation: NIJ has established baselines and targets for its annual performance measures, and can report actual performance beginning with 2003. While actual performance for 2004 is available, in most cases performance targets were not set for that year. With no target setting, achievement of the annual performance goals is impossible to determine.

Evidence: See Performance Measures Section of the PART.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: NIJ is undertaking a number of activities to ensure efficiencies and cost effectiveness in program execution. NIJ increasingly relies on OJP's computerized Grant Management System to focus grant management support functions and target science priorities and program planning. A semiannual oversight process for grants involving intensive monitoring. One piece of evidence that improvements in monitoring are materializing is in the decrease in the average time to closeout projects from 511 days in 2003 to 275 in 2004. Other process initiatives include the development of NIJ's Pipeline system, a comprehensive project tracking database for the use of all grant awards (reducing the time from selection to award approval from a range of 90 to 60 days). NIJ also has streamlined its procurement processes by greatly increasing its use of prequalified vendors with approved Government rates on various GSA schedules or via Blanket Purchase Agreements for repetitive requirements. In this way, NIJ ensures the robustness of competitions for the procurement of a variety of services while dramatically reducing the time to contract award. Finally, NIJ is currently participating in an A-76 competition that will address NIJ's key cost and performance drivers. NIJ has reduced its grants application processing time from 94 days on average in 2003 to 88 days in 2004, improving the efficient delivery of services to potential grantees. Further, printing costs have been reduced from $314,000 in 2003 to $213,000 in 2004.

Evidence: PWS solicitation for competitive sourcing, reference number 2004PR784 (www.fedbizopps.gov); See Performance Measures Section of the PART.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: Although NIJ is the only federal organization dedicated to funding basic and applied research on U.S. criminal justice, it can be compared with other research, development and evaluation entities. In 2004, 99 percent of awards were made competitively and 100 percent of applications were independently peer reviewed. While comparable statistics for similar R&D agencies were not readily available, it would be very difficult for any agency to score much higher than this. NIJ was able to compare grant award rates (number of awards/number of applications) with similar R&D agencies. In FY 2004, the National Science Foundation (NSF), the National Institute on Drug Abuse (NIDA), and the National Institutes of Health (NIH) recorded award rates of 27 percent, 27 percent, and 24 percent, respectively. The comparable rate at NIJ was 9 percent. This seems to reflect a more selective award process relative to demand for NIJ funds. In order to establish benchmarks for evaluation, NIJ has recently solicited proposals for an independent assessment of evaluation of management practices across federal agencies. As NIJ migrates its knowledge and tool dissemination function from paper to electronic documents, it places increased emphasis on Web-based customer satisfaction. NIJ's most recent Customer Satisfaction Index places it well within the range of similarly situated agencies.

Evidence: See Performance Measures Section of the PART for percentage of awards made competitively and applications peer reviewed. Comparative Data from Similar R&D Agencies, documented in email correspondence with NIH and NSF staff. Comparable Web-Based Customer Satisfaction Indices (documentation submitted by NIJ); NIJ Statement of Work (Improving the Return on Evaluation Investments).

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: Based on evaluations conducted to date, the evidence for NIJ's effectiveness is somewhat mixed. While no comprehensive review has been undertaken, positive reviews of some of the specific program areas come from the National Academy of Sciences. On the other hand, GAO has recently questioned the methodological rigor of some of the research funded by NIJ. GAO's findings concerning the rigor of the research raise questions about the extent to which NIJ's program outcomes are sufficiently successful. NIJ would benefit from a comprehensive review that could provide specific evidence of NIJ's impact, in both its social science and technology portfolios.

Evidence: National Academy of Sciences studies include: "Firearms and Violence: A Critical Review," "Advancing the Federal Research Agenda on Violence Against Women," "Fairness and Effectiveness in Policing: The Evidence," "Deadly Lessons: Understanding Lethal School Violence," "Elder Mistreatment: Abuse, Neglect, and Exploitation in an Aging America," and many others.(www7.nationalacademies.org/claj/CLAJ_Publications.html), GAO Report, "Justice Outcome Evaluations: Design and Implementation of Studies Require More NIJ Attention," GAO-03-1091, September 2003.