9.3. Test and Evaluation
DoD employs three formal types of T&E (directed by statute) in the acquisition of weapon systems, business systems, NSS, and joint systems administered by OSD: DT&E, OT&E, and LFT&E. The TRMC, also directed by statute, oversees the MRTFB to ensure availability of capabilities to support the three T&E types. Within these broad categories, the military departments and Defense Agencies have their own directives, guidance, organizations, T&E resources, ranges, and facilities specific to their needs. This section provides distinguishing features of each type.
9.3.1. Developmental Test and Evaluation
Programs conduct DT&E throughout the systems life cycle, from program initiation through system sustainment, to reduce design and programmatic risks and provide assessments. DT&E can occur as either contractor testing or government testing or a mix of both. As such, DT&E:
- Assesses achievement of Critical Technical Parameter(s) (CTPs) and Key System Attribute(s) (KSAs) along with assessment of progress toward achievement of KPPs and Critical Operational Issue(s) (COIs).
- Assesses system satisfaction of the thresholds as described in the capabilities requirements documentation.
- Supports progress toward and final characterization of the system readiness for dedicated IOT&E via the AOTR process and document.
- Characterizes system functionality and provides information for cost, performance, and schedule tradeoffs.
- Assesses system specification compliance.
- Reports progress to plan for Reliability Growth and characterizes reliability and maintainability.
- Identifies system capabilities, limitations, and deficiencies.
- Assesses system safety.
- Assesses compatibility with legacy systems.
- Stresses the system within an intended mission environment.
- Supports the joint interoperability certification process and achieves information assurance certification and accreditation.
- Documents achievement of contractual technical performance and verifies incremental improvements and system corrective actions.
In general, “DT&E is the disciplined process of generating experimental performance data from systems, subsystems, components and materiel for the purpose of informing optimum solutions and the state of performance progress toward design performance goals.”
Evaluation in the context of DT&E refers to evaluating the generated performance data to ensure it appropriately depicts the performance of the item as tested in the conditions of the test.
Testing in the context of DT&E refers to the process of establishing appropriate conditions and generating performance data from systems, subsystems, components and materiel.
9.3.2. Operational Test and Evaluation
Service and Defense Agency OTAs have a responsibility for OT&E. OT&E determines the operational effectiveness and operational suitability of a system under realistic operational conditions, including joint combat operations; determines the satisfaction of thresholds in the approved JCIDS documents and critical operational issues; assesses impacts to combat operations; and provides additional information on the system’s operational capabilities.
OTAs have a responsibility for early involvement in a system’s acquisition; for example, EOAs during the Technology Development (TD) phase, OAs during engineering and manufacturing development (EMD) phase, and review of Capabilities Documents to assess measurability, testability, and operational relevancy of requirements in the JCIDS documents (that is, Capability Development Document (CDD) and Capability Production Document (CPD)). OTAs also have responsibility for the assessment and evaluation of systems operational effectiveness, operational suitability, and survivability or operational security completed in IOT&E, and when necessary, Follow-on Operational Test and Evaluation (FOT&E).
General guidelines for the conduct of OT&E include:
- For dedicated OT&E, typical users operate and maintain the system under test conditions simulating combat and peacetime operations.
- OT&E uses threat or threat representative forces, targets, and threat countermeasures, validated by the Defense Intelligence Agency (DIA) or the DoD Component intelligence agency, as appropriate, and approved by DOT&E during the test plan approval process.
- Conducting IA Testing and evaluation for all weapon, information, and C4ISR programs depending on external information sources, or providing information to other DoD systems.
- Persons employed by the contractor for the system under development may only participate in the OT&E of MDAPs to the extent the PM planned for their involvement in the operation, maintenance, and other support of the system when deployed in combat.
- Testing production representative systems, which include any system accurately representing its final configuration using mature and stable hardware and software; that accurately mirrors the production configuration, but not produced on a final production line (although production tooling may account for some components).
9.3.2.1. Evaluation of Operational Effectiveness
DoD defines operational effectiveness as the overall degree of mission accomplishment of a system when used by representative personnel in the environment planned or expected for operational employment of the system considering organization, training, doctrine, tactics, survivability or operational security, vulnerability, and threat.
The evaluation of operational effectiveness links to mission accomplishment. The early planning for the evaluation should consider any special test requirements, such as the need for large test areas or ranges or supporting forces, requirements for threat systems or simulators, new instrumentation, or other unique support requirements.
For weapon systems, integrate LFT&E of system lethality into the evaluation of weapon system effectiveness. For example, operational testing could identify likely shot lines, hit points, burst points, or miss distances providing a context for LFT&E lethality assessments. Fuse performance, as determined under DT&E, can provide information for both OT&E and LFT&E assessments.
9.3.2.2. Evaluation of Operational Suitability
Operational suitability defines the degree in which a system satisfactorily places in field use, with consideration given to reliability, availability, compatibility, transportability, interoperability, wartime usage rates, maintainability, safety, human factors, manpower supportability, logistics supportability, documentation, environmental effects, and training requirements.
Early planning for the operational suitability evaluation should include any special needs for the number of operating hours, environmental testing, maintenance demonstrations, testing profiles, usability of DT&E data, or other unique test requirements.
Operational suitability evaluates a mission context to provide meaningful results. For example, maintaining a required operations tempo over an extended period while conducting realistic missions gives insight into the interactions of various suitability factors.
9.3.2.3. Evaluation of Survivability or Operational Security
Survivability or operational security includes the elements of susceptibility, vulnerability, and recoverability. As such, survivability or operational security acts as an important contributor to operational effectiveness and suitability. All systems under OT&E oversight should receive survivability or operational security assessment if exposed to threat weapons in a combat environment or to combat-induced conditions that may degrade capabilities, regardless of designation for LFT&E oversight. For example, unmanned vehicles may not have a requirement to undergo survivability LFT&E under section 2366 of title 10 USC, but should receive an assessment for survivability or operational security. The assessment may identify issues needing addressed through testing.
Integrate DT&E, OT&E, and LFT&E strategies to ensure the consistent assessment of the full spectrum of system survivability or operational security. The COIs should include any issues needing addressed in the OT&E evaluation of survivability or operational security. Systems under LFT&E oversight must address personnel survivability (reference section 2366 of title 10 USC) and integrate it into the overall system evaluation of survivability or operational security conducted under OT&E.
Generally, LFT&E address vulnerability while OT&E addresses susceptibility, but areas of overlap exist. The evaluation of LFT&E results requires realistic hit distributions. The OT&E evaluation of susceptibility might identify realistic hit distributions of likely threats, hit/burst points, and representative shot lines providing a context for LFT&E vulnerability assessments. DT&E and OT&E testing of susceptibility may provide other LFT&E insights, such as information on signatures, employment of countermeasures, and tactics used for evasion of threat weapons. Similarly, LFT&E tests, such as Total Ship Survivability trials, may provide OT&E evaluators with demonstrations of operability and suitability in a combat environment.
Recoverability addresses the consequences of system damage. Typically, LFT&E addresses recoverability; however, both OT&E and LFT&E have an interest in tests relating to recoverability from combat damage or from peacetime accidents, battle damage assessment and repair, crashworthiness, crew escape, and rescue capabilities.
LFT&E conducts real time casualty assessment (RTCA) during IOT&E to ensure assumptions supporting the RTCA remain consistent with LFT&E results.
Networked and C3I systems evaluation should include effectiveness of IA and Computer Network Defense (CND) measures against cyber threats in accordance with the DOT&E memo “Clarification of Procedures for Operational Test and Evaluation of Information Assurance in Acquisition Programs,” dated November 4, 2010, and “Procedures for Operational Test and Evaluation of Information Assurance in Acquisition programs,” dated January 21, 2009.
DAG Tutorial 
