33nd Annual International T&E Symposium

33nd Annual International T&E Symposium

How to Manage the Complexity

of Testing the SoS

Presented By:

Ms. Paola Pringle, NAVAIR

Mr. Todd Standard, NAVAIR

Enhancing our Competitive Edge: T&E to Meet the Pace of Need

October 4-6, 2015 Hyatt Regency~ Reston, VA

Distribution Statement A - approved for public release; distribution is unlimited,

as under NAVAIR Public Release Authorization 2016-775

BLUF

• Integrated and Interoperable (I&I) systems must be defined and

designed as such

• Test cannot create I&I

• I&I requires more than a single platform

• Our Naval Forces operate in a communication rich System of Systems

(SoS) and Family of Systems (FoS) that accomplish mission objectives

using capabilities

• Test focus must shift to testing capabilities within a mission context to

determine

How effective our platform is within each required missions

How confident we are that the platform can execute capabilities to accomplish

the required missions

Distribution Statement A - approved for public release; distribution is unlimited,

as under NAVAIR Public Release Authorization 2016-7752

Distribution Statement A - approved for public release; distribution is unlimited, as under NAVAIR Public Release Authorization 2015-710Distribution Statement A - approved for public release; distribution is unlimited, as under NAVAIR Public Release Authorization 2015-7103

Universal High-Level OV-1

4 Distribution Statement A - approved for public release; distribution is unlimited, as under NAVAIR Public Release Authorization 2016-775

P-8A Platform Centric View

5

Requirements•Multi-static Active Coherent

Acoustic Sensors

•Automatic Identification System

•Rapid Capability Insertion

•High Altitude ASW

•Weapons

•Communications & Sensor

Upgrades

•Net-Ready KPP

•Net-enabled ASuW Weapon

•Enhanced Security Architecture

5

6

Communications

Distribution Statement A - approved for public release; distribution is unlimited, as under NAVAIR Public Release Authorization 2016-775

System of Systems View

7

Capabilities

• Communications

• Acoustics

• Service Oriented Architecture (SOA)

• Track Management

• Targeting

• WeaponsTRITON UAS

TRITON UAS MCS

Distribution Statement A - approved for public release; distribution is unlimited, as under NAVAIR Public Release

Authorization 2016-7757

8

~1.4X

(Convergence Zone)

Acoustics


9

Weapons and Targeting


• Capabilities-based perspectives are more likely to identify

needed relationships among what were previously considered

independent systems

−SoS evolve over time making requirements

development an ongoing activity

• SoS systems engineer integrates the end-to-end functionality

and performance across the SoS mission capability

• SoS T&E engineer applies capabilities based test methods to

a mission (CBT&E) is required to verify and validate integration

of the end-to-end functionality and performance

ANALYSIS AND EVALUATION CHANGES FROM A ONE-TIME, UP-FRONT PROCESS

TO A MORE-OR-LESS CONTINUOUS PROCESS

10

CBT&E Complexity


• CBTE supports the continuum of upfront and early materiel solutions

analysis, completes the Systems Engineering “V”, and addresses testing from

a capabilities viewpoint

− T&E process is both iterative and recursive with each pass providing an increasing

level of complexity and realism to the system under test with a corresponding level

of confidence

CBTE CLOSES THE LOOP ON MATERIEL SOLUTION ANALYSIS THROUGH

VERIFICATION AND VALIDATION OF INTEGRATED WARFIGHTING CAPABILITIES

SYSTEM T&E IN A MISSION CONTEXT


DoD Architecture Framework

12Distribution Statement A - approved for public release;

distribution is unlimited, as under NAVAIR Public Release Authorization 2016-775

System of Systems (SoS) Engineering

Product Integration

Operational Evaluation Views

System/Services

Functionality

System/Services Evaluation Views

Operational

Concept

Operational

Activity

Component &Qualification Testbeds

LVC Distributed Events

Mission

and

Capabilities

Based

Architectures

Live, Virtual,

Constructive

(LVC) Test

Environments

Desired Capability

PerformanceGap: Mission

Level Assessment

& Evaluation

Across SoS

Solution:

Capabilities

Based T&E

(CBTE)

Mission

Based

Verified &

Validated

Capability

BaselinesCapability Actual

Performance

13

CBTE is a Catalyst for Architecture-driven

Distributed Capabilities Verification and Validation


CBTE Intent

14

Cross-competency expanded processes

required to rapidly assess SoS capabilities

Development

SoS / SUT Performance

Requirements:

Unambiguous,

Measurable, Testable

Tasks/Conditions:

UNTL, UJTL

Mission

Desired Effects Fleet Exercise

Operational

Test

Development

Test

CBTE Precursor Products

• Mission Effectiveness Baseline (MEB)

• Mission Confidence Baseline (MCB)

Mission Context (MTB / ICTB)

Mission Evaluation Structure (Mission Study Views / MVM)

Operational

Capabilities

Operational Capability

Needs

ICTB – Integrated Capability Technical

Baseline

MTB – Mission Technical Baseline

MVM – Mission Verification Matrix

SoS – System of Systems

SUT – System Under Test

UJTL – Universal Joint Task List

UNTL – Universal Naval Task List

Mission Based Test & Evaluation “V”

CBTE Products:

• Mission Test Strategy (MTS)

• SoS Test Environment Investment

Strategy (STEIS)

• Mission Test Matrix (MTM)

with Mission Test Architecture

baseline (MTA)

• Provides

traceability from

mission capability

requirements to

mission testing

• Provide timely,

credible, and

continuous

feedback to

decision makers


MTB / ICTB Product One Liners

MTB

• Documents a mission based need using scenarios

ICTB Level 1

• Naval Force apportionment for mission area

ICTB Level 2

• Describes how systems are organized and allocates performance requirements

ICTB Level 3

• Used to manage requirements across multiple programs

15Distribution Statement A - approved for public release; distribution is unlimited, as under NAVAIR Public Release Authorization 2016-775

Mission Array Inputs to Mission Verification Matrix (MVM) DoDAF Viewpoint Model

16


Mission Array Inputs to Mission Verification Matrix (MVM)

17

W1


Outcomes for SoS using CBT&E

• CBTE has the potential to increase acquisition efficiency for the SoS

SoS viewed within a data driven high accuracy mission context

Improved I&I of the SoS throughout the acquisition lifecycle

SoS V&V across mission, task, and system requirements

Better coordinated, more efficient acquisition workflow

• System Under Test has high confidence that mission-level

requirements are met when fielded

• CBTE facilitates IT: Complements MBTD, COMOPTEVFOR’s

planning method for OT promoting operational view of SUT in DT

• CBTE promotes end-to-end testing of the SUT using LVC test

environments in an operationally relevant mission context

• “Closes the loop” on materiel solution analysis

• Focuses on continuous improvement of mission / SoS capabilities


Questions


Contact Information

Ms. Paola Pringle

Integrated Warfare Test and Evaluation Division, 5.1L

Cyberspace T&E Branch 51L300E

P-8A Increment 3 Interoperability LTE

(805) 816-3038

[email protected]

Mr. Todd Standard

Integrated Systems Evaluation, Experimentation and Test, AIR 5.1G

Capabilities-Based T&E Chief Test Engineer

(301) 757-7070

[email protected]

Hank 'Wizard' Steinfeld, APM T&E P-8A Inc 3, P-3C

APM T&E PMA-264

ASW IWC APM T&E

PMA-290 Cyber Test Strategy Lead

301-342-3041

[email protected]


mailto:[email protected]

