Developmental Test and Evaluation

Dr. J. Brian HallActing Deputy Assistant Secretary

Developmental Test and Evaluation

3 October 2017

International Test and Evaluation Association

Distribution Statement A: Approved for public release; distribution is unlimited. Cleared 17 –S-1321

Agenda

• Office of The DASD(DT&E)– Mission, Roles, and Responsibilities– Differences between DT and OT

• Miscellaneous Technical Areas– Developmental Evaluation Framework– Design of Experiments– Design for Reliability and Reliability Growth– Cybersecurity T&E

• Emerging Areas – Frontiers of T&E– DT&E / TRMC Initiatives

2Distribution Statement A: Approved for public release; distribution is unlimited. Cleared 17 –S-1321

Office of The DASD(DT&E)

3

Major Program

EngagementDAWIA T&E

Workforce

Policy, Guidance,

Congressional Reporting

DT&E Mission: Improve acquisition outcomes by advancing the DT&E “state of the practice,” engaging acquisition programs to position them for success, and by executing Title 10 oversight responsibilities.

Distribution Statement A: Approved for public release; distribution is unlimited. Cleared 17 –S-1321

DT&E OT&E

Tests • Prototypes of systems, subsystems, and components

• M&S, HITL, ISTF, Open Air Range

• Units equipped with production representative systems

• SoS or system-level Open Air Range

Evaluates • Performance and Interoperability• Reliability and Maintainability• Cybersecurity Posture• KPP/KSA Compliance

• Operational Effectiveness• Operational Suitability• Survivability• Test Adequacy

Informs • Design improvements• Production readiness

• Improvements to TTPs and Training• Combat Readiness

Organized • Controlled by the acquisition chain of command

• Independent of the acquisition chain of command

Differences between DT and OT

4Distribution Statement A: Approved for public release; distribution is unlimited. Cleared 17 –S-1321

DT (can be, but) is frequently not a good predictor of OT outcomes.

Developmental Evaluation Framework

5Distribution Statement A: Approved for public release; distribution is unlimited. Cleared 17 –S-1321

• Evaluates:– Performance - Interoperability– Reliability & Maint. - Cybersecurity Posture

• Includes:– CTR Testing - Lab Testing– Government DT - Cat./Trap Testing– Sea Trials

• Informs:– Long Lead Item Procurement– First Flight Readiness– Carrier Suitability– Initial Sea Trial Readiness– Production Readiness– Follow on Sea Trial Readiness– OT&E Readiness

MQ-25A (Unmanned Carrier Aviation)

6

MQ-25A is a carrier-based unmanned aircraft that supports long-endurance, Intelligence, Surveillance, Reconnaissance (ISR) and aerial refueling

Long Lead Item

Procurement

Design Reviews MS-C / LRIP

Design maturity of LL components?

Performance & integration

status

Air Vehicle performance?

Control System

performance?

SoS integration?

Air vehicle performance?

Landing system

integration?

Flight deck compatability

?

Air vehicle performance?

Control system

integrated performance?

Air ship integration

(ASI)?SoS maturity?

Family of Systems (FoS) performance?

Training status?

FoS performance?

Logistics supportability?

System CapabilitiesTechnical Reqmts

Document Reference

DescriptionData Sources

(Test, M&S events)

PerformanceAir refuel (give)

External carriage

Envelope

Endurance

Sensor Integration

Sensor Performance

Envelope

Air worthiness

CVN suitability

System survivability

Envelope

Environmental E3

InteroperabilityLOI 1-5

LOI 1-5

LOI 1-5

A/V - CS

A/V - Fuel receiver

CybersecuritySystem/SW Assurance

RMF Measure1

RMF Measure2

Vulnerability Assessment Vuln Assess Measure #1

Adversarial Assessment Adversarial Measure #1

ReliabilitySoS reliablity

A/V Service Life

Logistics Supportability

Training

System Requirements / MeasuresDevelopmental

Evaluation Objectives

Compliance

First Flight Readiness

ISR

Aerial Refueling

CVW Integration

NR-KPP

Carrier Suitability Initial Sea Trial Readiness Follow on Sea Trial Readiness

PEO Certification (OT&E Readiness)

Decisions Supported

Distribution Statement A: Approved for public release; distribution is unlimited. Cleared 17 –S-1321

Design of Experiments and the STAT COE

7Distribution Statement A: Approved for public release; distribution is unlimited. Cleared 17 –S-1321

Initial System Reliability

Req. Dev.

Design AoA

Rel. Allocation

FRACAS

PoF ESSALT/HALT

DoEQual.

Testing

Tolerance Analysis

Root Causes

Corrective ActionFailure

Models

Fault Tree Analysis

Reliability Block Diagrams

FMEA/FMECA

Prod. Testing

FPRB

• GEIA-STD-009, Contract Language Promoting RAM Best Practices

• AMSAA Program Scorecard to Assess

Reliability Program Plan

• DFR improves product quality and reliability

• DFR Reduces Lifecycle Costs

Design for Reliability

Distribution Statement A: Approved for public release; distribution is unlimited. Cleared 17 –S-1321

Reliability Growth Planning

9

80

114

175

215220

CAP1

CAP2

CAP3

CAP4

0

50

100

150

200

250

300

Idealized Curve Customer Test Initial DT LUTLUT Excursion IOT Requirement ASA(ALT) Threshold

$869 M$894 M$1,103 M

$1,701 M

$2,457 M

Test Time (Hours)CAP – Corrective Action Period

The most critical step …- Designing for Reliability -

Mea

n Ti

me

Bet

wee

n Fa

ilure

(MTB

F)

Distribution Statement A: Approved for public release; distribution is unlimited. Cleared 17 –S-1321

( )t

th B

⋅+=

βλ

1

ttt⋅+

⋅=

ββθ

1)(

Figure 1. Expected No. Modes. Figure 2. Percent λB Observed.

Figure 3. ROC of New Modes. Figure 4. Reliability Growth.

( ) ( ) ( ){ } ( )1

1A d BM t h t h tλ µ λ−

= + − ⋅ − +

( ) ( )tt B ⋅+

= β

βλµ 1ln

( )

⋅+

=t

th B

βλ

1

Assessing Reliability Maturity

Distribution Statement A: Approved for public release; distribution is unlimited. Cleared 17 –S-1321

• Requirements– January 2017, cybersec in SS KPP

• Policy– January 2017 - D0DI 5000.01

Cybersecurity T&E

• Guidance– July 2015 – DoD Cybersecurity T&E Guidebook – September 2015 – DoD PM’s Cybersecurity Guidebook– January 2017 – Cyber Survivability Implementation Guide– February 2017 – Defense Acquisition Guidebook

• Training– Defense Acquisition University– Cyber Tabletop Events– Cross-Service Working Groups

• Investments– T&E infrastructure– Manning for cyber test teams– Cybersec evaluations of Major Defense Programs

11

DT&E Phases OT&E Phases

Distribution Statement A: Approved for public release; distribution is unlimited. Cleared 17 –S-1321

Previous Offset Strategies

First Offset Strategy. Emphasis on nuclear deterrence to avoid the large increase in defense expenditures necessary to conventionally deter Warsaw Pact forces during the 1950s.

These Offset Strategy’s technologies continue to enable U.S. global precision strike today

12

Second Offset Strategy. Emphasized: Intelligence, Surveillance, and Reconnaissance (ISR) platforms; Precision-Guided Weapons; Stealth Technology; and the expansion of space’s role in military communications and navigation.

Distribution Statement A: Approved for public release; distribution is unlimited. Cleared 17 –S-1321

A Third Offset Strategy

• Autonomous Learning Systems– Delegating decisions to machines in applications that

require faster-than-human reaction times

– Cyber Defense, Electronic Warfare, Missile Defense

• Human-Machine Collaborative Decision Making– Exploiting advantages better and faster human decisions

– “Human strategic guidance combined with the tactical acuity of a computer”

• Assisted Human Operations– Helping humans perform better in combat

• Advanced Manned-Unmanned System Operations– Employing innovative cooperative operations

– “Smart swarm” operations and tactics

• Network-enable, autonomous weapons hardened to operate in a future Cyber/EW Environment

– Allowing for cooperative weapon concepts in communications-denied environments

13Distribution Statement A: Approved for public release; distribution is unlimited. Cleared 17 –S-1321

DT&E / TRMC Initiatives

14

1. Improve TRMC/DT&E collaboration2. Emphasize Shift Left3. Institutionalize the Developmental

Evaluation Framework4. Implement the TEMP at MS A5. Advocate for the T&E Workforce6. Improve support to PMs and Chief

Developmental Testers7. Improve reliability T&E8. Improve cybersecurity T&E 9. Improve interoperability T&E10. Improve hypersonics infrastructure11. Incorporate big data/knowledge

management into T&E12. Understand/improve T&E of

autonomous systems13. Improve Mission Context in DT

Distribution Statement A: Approved for public release; distribution is unlimited. Cleared 17 –S-1321

Questions?

ARMY NAVY

AIR FORCEDEFENSEAGENCY

15Distribution Statement A: Approved for public release; distribution is unlimited. Cleared 17 –S-1321