Enclosure 6 -- Integrated T&E

1. Overview
2. T&E Planning
3. DT&E
4. Readiness for IOT&E
5. OT&E
6. OSD T&E Oversight List
7. LFT&E
8. M&S
9. Foreign Comparative Testing (FCT)
10. Testing Increments of an Evolutionary Acquisition Program

1. OVERVIEW

1. The fundamental purpose of T&E is to provide knowledge to assist in managing the risks involved in developing, producing, operating, and sustaining systems and capabilities. T&E measures progress in both system and capability development. T&E provides knowledge of system capabilities and limitations to the acquisition community for use in improving the system performance, and the user community for optimizing system use in operations. T&E expertise must be brought to bear at the beginning of the system life cycle to provide earlier learning about the strengths and weaknesses of the system under development. The goal is early identification of technical, operational, and system deficiencies, so that appropriate and timely corrective actions can be developed prior to fielding the system.
2. The PM, in concert with the user and the T&E community, shall coordinate DT&E, OT&E, LFT&E, family-of-systems interoperability testing, information assurance testing, and modeling and simulation (M&S) activities, into an efficient continuum, closely integrated with requirements definition and systems design and development. The T&E strategy shall provide information about risk and risk mitigation, provide empirical data to validate models and simulations, evaluate technical performance and system maturity, and determine whether systems are operationally effective, suitable, and survivable against the threat detailed in the STAR or STA. The T&E strategy shall also address development and assessment of the weapons support equipment during the EMD Phase, and into production, to ensure satisfactory test system measurement performance, calibration traceability and support, required diagnostics, and safety. Adequate time and resources shall be planned to support pre-test predictions and post-test reconciliation of models and test results, for all major test events. The PM, in concert with the user and the T&E community, shall provide safety releases (to include formal Environment, Safety, and Occupational Health (ESOH) risk acceptance in accordance with Section 6 of Enclosure 12) to the developmental and operational testers prior to any test using personnel.
3. The PM shall design DT&E objectives appropriate to each phase and milestone of an acquisition program. Testing shall be event-driven and monitored by the use of success criteria within each phase, OT&E entrance criteria, and other metrics designed to measure progress and support the decision process. The OTA and the PM shall collaboratively design OT&E objectives appropriate to each phase and milestone of a program, and these objectives shall be included in the TEMP. Completed IOT&E and completed LFT&E shall support a beyond-LRIP decision for ACAT I and II programs for conventional weapons systems designed for use in combat. For this purpose, OT&E shall require more than an OA that was based exclusively on computer modeling, simulation, or an analysis of system requirements, engineering proposals, design specifications, or any other information contained in program documents (sections 2399 and 2366 of Reference (k)).
4. Systems that provide capabilities for joint missions shall be tested in the expected joint operational environment.

2. T&E PLANNING

1. Test and Evaluation Strategy (TES). At Milestone A, the PM shall submit a TES that describes the overall test approach for integrating developmental, operational, and live-fire test and evaluation and addresses test resource planning. The TES shall include a test plan that addresses Technology Development phase activity, including the identification and management of technology risk, and the evaluation of system design concepts against the preliminary mission requirements resulting from the AoA. Test planning shall address the T&E aspects of competitive prototyping, early demonstration of technologies in relevant environments, and the development of an integrated test approach. The Milestone A test plan shall rely on the ICD as the basis for the evaluation strategy. For programs on the OSD T&E Oversight List, the TES shall be submitted to the USD(AT&L) and the DOT&E for approval.
2. Test and Evaluation Master Plan (TEMP). The PMs for MDAPs, MAIS Acquisition Programs, and programs on the OSD T&E Oversight List shall submit a TEMP to the USD(AT&L) and the DOT&E for approval to support Milestones B and C and the Full-Rate Production decision. The TEMP shall describe planned developmental, operational, and live-fire testing, including measures to evaluate the performance of the system during these test periods; an integrated test schedule; and the resource requirements to accomplish the planned testing. The MDA or designee shall ensure that IOT&E entrance criteria, to be used to determine IOT&E readiness certification in support of each planned operational test, are developed and documented in the TEMP.
3. Planning Requirements
   1. Planning shall provide for completed DT&E, IOT&E, and LFT&E, as required, before entering full-rate production.
   2. Test planning for commercial and non-developmental items shall recognize commercial testing and experience, but nonetheless determine the appropriate DT&E, OT&E, and LFT&E needed to ensure effective performance in the intended operational environment.
   3. Test planning and conduct shall take full advantage of existing investment in DoD ranges, facilities, and other resources. Embedded instrumentation shall be designed and developed to facilitate training, logistics support, and combat data collection.
   4. Planning shall consider the potential testing impacts on the environment (sections 4321-4347 of Reference (ac) and Reference (ad)).
   5. The concept of early and integrated T&E shall emphasize prototype testing during EMD and early OAs to identify technology risks and provide operational user impacts.
   6. Appropriate use of accredited models and simulation shall support DT&E, IOT&E, and LFT&E.
   7. The DOT&E and the Director, SSE, shall have full and timely access to all available developmental, operational, and live-fire T&E data, records, and reports.
   8. Interoperability Testing: All DoD MDAPs, programs on the OSD T&E Oversight list, post-acquisition (legacy) systems, and all programs and systems that must interoperate, are subject to interoperability evaluations throughout their life cycles to validate their ability to support mission accomplishment. For IT systems (including NSS) with interoperability requirements, the Joint Interoperability Test Command (JITC), regardless of ACAT, shall provide system interoperability test certification memorandums to the Deputy Under Secretary of Defense (Acquisition and Technology) (DUSD(A&T)), the ASD(NII)/DoD CIO, and the Director, Joint Staff J-6, throughout the system life-cycle.

3. DT&E. During DT&E, the materiel developer shall:

1. Identify the technical capabilities and limitations of the alternative concepts and design options under consideration;
2. Identify and describe design technical risks;
3. Stress the system under test to at least the limits of the Operational Mode Summary/ Mission Profile, and, for some systems, beyond the normal operating limits to ensure the robustness of the design;
4. Assess technical progress and maturity against critical technical parameters, to include interoperability, documented in the TEMP;
5. Assess the safety of the system/item to ensure safety during OT and other troop-supported testing and to support success in meeting design safety criteria;
6. Provide data and analytic support to the decision process to certify the system ready for IOT&E;
7. Conduct information assurance testing on any system that collects, stores, transmits, or processes unclassified or classified information;
8. In the case of IT systems, including NSS, support the DoD Information Assurance Certification and Accreditation Process and Joint Interoperability Certification process; and
9. Prior to full-rate production, demonstrate the maturity of the production process through Production Qualification Testing of LRIP assets.

4. READINESS FOR IOT&E

1. The DoD Components shall each establish an Operational Test Readiness Process for programs on the OSD T&E Oversight List, consistent with the following requirements:
   1. The process shall include a review of DT&E results; an assessment of the system's progress against critical technical parameters documented in the TEMP; an analysis of identified technical risks to verify that those risks have been retired during developmental testing; and a review of the IOT&E entrance criteria specified in the TEMP. Programs shall provide copies of the DT&E report and the progress assessment to USD(AT&L) and DOT&E.
   2. At each test readiness review, the PM shall ensure that the impact of deviations and waivers is considered in the decision to proceed to the next phase of testing.
2. The DUSD(A&T) shall conduct an independent Assessment of Operational Test Readiness (AOTR) for all ACAT ID and special interest programs designated by the USD(AT&L). Each AOTR shall consider the risks associated with the system's ability to meet operational suitability and effectiveness goals. This assessment shall be based on capabilities demonstrated in DT&E and OAs and criteria described in the TEMP. Where feasible, the AOTR shall be performed in conjunction with the program's review and reporting activities as described in subparagraph 4.a.(1) of this Enclosure. The AOTR report shall be provided to the USD(AT&L), DOT&E, and CAE.
3. The CAE shall consider the results of the AOTR prior to making a determination of materiel system readiness for IOT&E.

5. OT&E

1. OT&E Requirements
   1. OT&E shall be used to determine the operational effectiveness and suitability of a system under realistic operational conditions, including joint combat operations; determine if thresholds in the approved CPD and critical operational issues have been satisfied; assess impacts to combat operations; and provide additional information on the system's operational capabilities.
   2. The lead OTA shall brief the DOT&E on concepts for an OT&E 120 days prior to start. They shall submit the OT&E plan 60 days prior, and shall report major revisions as they occur.
   3. Typical users shall operate and maintain the system or item under conditions simulating combat stress and peacetime conditions.
   4. The independent OTAs shall use production or production-representative articles for the dedicated phase of IOT&E that supports the full-rate production decision (or for ACAT IA or other acquisition programs, the full-deployment decision).
   5. Hardware and software alterations that materially change system performance, including system upgrades and changes to correct deficiencies, shall undergo OT&E.
   6. OTAs shall conduct an independent, dedicated phase of IOT&E before full-rate production to evaluate operational effectiveness and suitability, as required by section 2399 of Reference (k).
   7. All weapon, information, and Command, Control, Communications, Computers, Intelligence, Surveillance, and Reconnaissance programs that depend on external information sources, or that provide information to other DoD systems, shall be tested and evaluated for information assurance.
   8. The DOT&E, following consultation with the PM, shall determine the quantity of articles procured for IOT&E for MDAPs; the cognizant OTA, following consultation with the PM, shall make this decision for non-MDAPs (section 2399 of Reference (k)).
   9. The DOT&E shall assess the adequacy of IOT&E and LFT&E and evaluate the operational effectiveness, suitability, and survivability, as applicable, of systems under DOT&E oversight. DOT&E-oversight programs beyond LRIP shall require continued DOT&E test plan approval, monitoring, and FOT&E reporting to:
      1. Complete IOT&E activity;
      2. Refine IOT&E estimates;
      3. Verify correction of deficiencies;
      4. Evaluate significant changes to system design or employment; and
      5. Evaluate whether or not the system continues to meet operational needs and retain operational effectiveness in a substantially new environment, as appropriate.
2. OT&E Information Promulgation
   1. The responsible test organization shall release valid test data and factual information in as near-real-time as possible to all DoD organizations and contractors with a need to know. Data may be preliminary and shall be identified as such.
   2. To protect the integrity of the OTA evaluation process, release of evaluation results may be withheld until the final report, according to the established policies of each OTA. Nothing in this policy shall be interpreted as limiting the statutory requirement for immediate access to all OT&E results by DOT&E.
   3. The primary intent of this policy is to give developing agencies visibility of factual data produced during OT&E, while not allowing the developmental agency any influence over the outcome of those evaluations.
3. Use of Contractors in Support of OT&E
   1. Per section 2399 of Reference (k), persons employed by the contractor for the system being developed may only participate in OT&E of MDAPs to the extent they are planned to be involved in the operation, maintenance, and other support of the system when deployed in combat.
   2. A contractor that has participated (or is participating) in the development, production, or testing of a system for a DoD Component (or for another contractor of the Department of Defense) may not be involved in any way in establishing criteria for data collection, performance assessment, or evaluation activities for OT&E.
   3. The DOT&E may waive the limitations in 5.c.(1) and 5.c.(2) of this enclosure if the DOT&E determines, in writing, that sufficient steps have been taken to ensure the impartiality of the contractor in providing the services. These limitations do not apply to a contractor that has participated in such development, production, or testing, solely in test or test support on behalf of the Department of Defense.

6. OSD T&E OVERSIGHT LIST

1. The DOT&E and the DUSD(A&T) shall jointly, and in consultation with the T&E executives of the cognizant DoD Components, determine the programs designated for OSD T&E oversight. The OSD memorandum, "Designation of Programs for OSD Test and Evaluation (T&E) Oversight" (Reference (bb)) will identify these programs.
2. Programs may be placed on the OSD T&E Oversight List in one or more of the following categories: developmental testing, operational testing, or live fire testing. Unless otherwise indicated, programs designated for operational and live-fire test and evaluation are to be considered MDAPs and covered programs subject to the provisions of sections 139, 2366, and 2399 of Reference (k) and the requirements of this Instruction, including the submission of T&E Strategies, DIA or DoD Component-validated STARs, TEMPs, Operational Test Plans, and reporting of test results.
3. Force protection equipment (including non-lethal weapons) will be identified as a separate category on the OSD T&E Oversight List. With respect to OT&E and/or survivability testing of such equipment, DOT&E will provide guidance to and consult with senior Defense officials to expedite suitable OT&E; provide objective subject-matter expertise; encourage data sharing between DoD Components; and facilitate the use of common test standards. The DOT&E will not delay deployment of, nor require approval of test plans for, such equipment (section 139 of Reference (k)). Force protection programs are not MDAPs unless they meet the requirements specified in sections 2430 and 2399 of Reference (k).

7. LFT&E. (Not applicable to ACAT IA programs.) Section 2366 of Reference (k) mandates LFT&E and formal LFT&E reporting for all covered systems. The DOT&E shall approve the LFT&E strategy for covered systems prior to Milestone B.

8. M&S. The PM shall plan for M&S throughout the acquisition life cycle. The PM shall identify and fund required M&S resources early in the life cycle.

9. FOREIGN COMPARATIVE TESTING (FCT). Paragraph (g) of section 2350a of Reference (k) prescribes funding for U.S. T&E of selected allied and friendly foreign countries' equipment and technologies when such items and technologies have potential to satisfy approved DoD requirements. The USD(AT&L) shall centrally manage FCT and notify the congressional defense committees of the intent to obligate funds made available to carry out FCT not less than 7 days before such funds are obligated.

10. TESTING INCREMENTS OF AN EVOLUTIONARY ACQUISITION PROGRAM. The structure of these test activities depends on the program acquisition strategy. In general, all programs shall:

1. Provide for early involvement of the Service OTA/JITC in DT&E and test planning;
2. Conduct adequate DT&E, LFT&E, and OT&E of the first and each successive increment of capability;
3. Integrate, as appropriate, and without compromising the specific requirements of the different types of testing, successive periods of DT&E, LFT&E, and IOT&E;
4. Tailor test content and reporting against earlier test results, evaluating at a minimum the increment of mission accomplishment and survivability required of the new increment, plus whether or not performance previously demonstrated by the previous increment has been degraded;
5. For programs under OSD OT&E and/or LFT&E oversight, support DOT&E's intended schedule for reporting to the Secretary of Defense and congressional defense committees.