An International Code of Best Practice for T&E & Experimentation of Complex, Adaptive Aerospace Mission Capabilities

– sure we can do that, it can’t be that hardWing Commander Malcolm G. Tutty MEng, CPEng, FIE(Aust), FRAeS

Director Simulation, Trials & Ranges – Royal Australian Air Force,

PhD Candidate & Vice President, Southern Cross Chapter, ITEA

Si ego Certiorem Faciam, Mihi Tu Delendus Eris – the Greek Goddess of Chaos! HomerSi ego Certiorem Faciam, Mihi Tu Delendus Eris – the Greek Goddess of Chaos! Homer

Nature

Design

By suitably arranging these patterns, By suitably arranging these patterns, one can simulate a Turing Machine. one can simulate a Turing Machine. Paul Rendell.Paul Rendell. http://rendell.server.org.uk/gol/tmdetails.htmhttp://rendell.server.org.uk/gol/tmdetails.htm

By suitably arranging these patterns, By suitably arranging these patterns, one can simulate a Turing Machine. one can simulate a Turing Machine. Paul Rendell.Paul Rendell. http://rendell.server.org.uk/gol/tmdetails.htmhttp://rendell.server.org.uk/gol/tmdetails.htm

The “glider” pattern

The “glider” pattern

Complex systemsComplex systems

Mechanism Function Purpose

Complex systemsComplex systems• Emergence: the creation of a new entity, one which has new properties (often a the creation of a new entity, one which has new properties (often a

group or a system), through the interaction among multiple autonomous elements. group or a system), through the interaction among multiple autonomous elements. – Multiscalarity: everything is both an entity and a group.

• A level of abstraction has both a specification (requirements) and an level of abstraction has both a specification (requirements) and an implementationimplementation.

– Throwing away the specification once an implementation exists produces a reductionist blind spot.– It’s the specification (of the interface) that ensures loose coupling.

• Interaction—even (or especially) intra-systemInteraction—even (or especially) intra-system —occurs through an environment.– An environment that provides functionality that facilitates interaction is a platform.– Architectures: agents and platforms vs. stovepipes and functional decomposition.– Platform governance becomes a fundamental issue. Who owns it, runs it, controls it?

• Evolutionary processes are unavoidableEvolutionary processes are unavoidable — leading to unexpected consequences. They are also the source of all creativity.

– Their essence combines exploration with exploitation of discoveries.– Organizations can plan to be innovative.

• Groups are nature’s way to build systems.Groups are nature’s way to build systems.– We can build powerful groups because we evolved to live in groups and we can learn. – How can a group’s wisdom be distilled as action? Bottom-up resource allocation.

• Nature and markets have self-validating criteria: reproductive success and profits.• By looking carefully you can see the world in a grain of sand.

Two levels of emergenceTwo levels of emergence

• No individual chemical reaction inside the ants is responsible for making them follow the rules that describe their behavior.

• That the internal chemical reactions together do is an example of emergence.

• No individual rule and no individual ant is responsible for the ant colony gathering food.

• That the ants together bring about that result is a second level of emergence.

Colony results

Ant behaviors

Ant chemistry

Each layer is a level of abstraction

Notice the similarity to layered communication protocols

No horizontal communication.No dashed lines. (Is that good?)

It’s not accurate as a communication or

operational structure.

It may represent how authority is delegated,and it may represent how responsibility is assigned,but it doesn’t represent how communication occursor how organizations really work.

Can be implemented with point-to-point communication links.

Organizational/system structure:What’s wrong with this picture?

Functional decompositionFunctional decomposition

Downward pointing arrows: commands.Upward pointing arrows: results/reports.

A somewhat more realistic picture

The focus is on interaction among participants in the organization.

David Sloan Wilson,Evolution for Everyone

Everything is both an entity and a group.

Team TacticsMission

Theatre

Network Enabled

Campaign

EngagementIndividual Tactics

Teams of Teams Tactics

“Organisations” of Teams Tactics

Representation of Operations

JAIME CODExJAIME CODEx

OPERATIONALOPERATIONALCONCEPTSCONCEPTSTACTICSTACTICS

DEVELOPMENTDEVELOPMENT

PREPAREDNESSPREPAREDNESSSYSTEMSYSTEM

INTEGRATIONINTEGRATION

TRAININGTRAINING

REQUIREMENTSREQUIREMENTSDETERMINATIONDETERMINATION

CAPABILITYCAPABILITYOPTIONSOPTIONS

DESIGNDESIGNREVIEWREVIEW

SYSTEM SYSTEM TESTTEST

CAPABILITYCAPABILITYGAPSGAPS

JOINT/COALITIONJOINT/COALITIONWARFIGHTINGWARFIGHTING

CONCEPTSCONCEPTS

CONCEPTCONCEPTDEVELOPMENTDEVELOPMENT

CAPABILITYCAPABILITYDEVELOPMENTDEVELOPMENT

ININ--SERVICESERVICESUPPORTSUPPORT

CAPABILITYCAPABILITYACQUISITIONACQUISITION

WHO

LE

WHO

LE-- O

FO

F-- L

IFE

LIF

EW

HO

LE

WHO

LE-- O

FO

F-- C

APABIL

ITY

CAPABIL

ITY

DOCTRINEDOCTRINE

INTRODUCTION TOINTRODUCTION TOSERVICESERVICE

A concept is “a thought, idea or notion, A concept is “a thought, idea or notion, often one derived from a generalised mental operation” often one derived from a generalised mental operation”

ADF capability managementADF capability managementThe ADF is to be concept lead, capability based Joint Vision 2020, ADDP-D.2The ADF is to be concept lead, capability based Joint Vision 2020, ADDP-D.2

Technology/Systems Readiness Levels

1 Basic principle observed and reported.

Studies or initial investigations undertaken.

2 Technology concept and/or application formulated.

Potential applications have been identified.

3 Analytical and experimental critical function and/or characteristic proof of concept. R&D has been initiated, work towards validating the concept done.

4 Component and/or breadboard validation in lab environment. The basic elements of the system/ product have been integrated to show they will work.

5 Components and/or breadboard validation in relevant environment.

A higher fidelity validation of the system/ product in a realistic environment.

6 System/sub-system model or prototypeSystem/sub-system model or prototype demonstration in a relevantdemonstration in a relevant and or realistic environment.

7 System prototype demonstration in operational environmentSystem prototype demonstration in operational environment. Prototype demonstrated. Productionisation can commence.

8 Actual system completed and mission qualified through test and demonstration.

Actual system/ product has been successfully tested/ qualified.Actual system/ product has been successfully tested/ qualified.

9 Actual system proven through successful mission operations.

Actual system/ product has been successfully fielded.Actual system/ product has been successfully fielded.

CAPABILITYCAPABILITY IDENTIFIEDIDENTIFIED

ACQUISITIONACQUISITIONINTRODUCTION INTRODUCTION

INTO ADF SERVICEINTO ADF SERVICECAPABILITY MAINTAINEDCAPABILITY MAINTAINED

CONCEPT CONCEPT OPS REQT DEFNOPS REQT DEFN

00

CONOPS REQTCONOPS REQTACHIEVED &ACHIEVED &MAINTAINEDMAINTAINED

9

OT&EOT&E6

AT&EAT&E5

INTRO INTOINTRO INTOSERVICESERVICE

77

PWDPWD1010

VEH & SYST SPOVEH & SYST SPOACQUISITION &ACQUISITION &SYSTEMS ENGSYSTEMS ENG

(ASC SIM SURVEY)(ASC SIM SURVEY)

22

CAPMGRCAPMGRORD MET?ORD MET?

Y / NY / N

Complex Capability Management ModelComplex Capability Management Model- Lust to Dust Methodology- Lust to Dust Methodology

LUSTLUST WOMB WOMB FISCAL REALITIESFISCAL REALITIES! ! CLEARANCE ACTIVITY CLEARANCE ACTIVITY CERTIFICATION ACTIVITY CERTIFICATION ACTIVITY ADOLESCENCE TOMB DUSTADOLESCENCE TOMB DUST

EXPERIMENT’NEXPERIMENT’N1

QUALIFICATIONQUALIFICATION4

TTP EXPERIMENTATIONTTP EXPERIMENTATION8

DT&EDT&E3

Hierarchy of needs• In order for a design to be successful, it must meet

people’s basic needs before it can attempt to satisfy higher level needs

Creativity

Proficiency

Usability

Reliability

Functionality

Highest Value - Allows users to create and explore areas beyond the original design

High Value - Empowers users to do more than they could before

Moderate Value - Easy to use, tolerates mistakes

Low Value - Operation is consistent and reliable

Little Value - Meets design requirements, e.g. play, record

Level of need

• OPERATIONAL CONCEPT / REQUIREMENTOPERATIONAL CONCEPT / REQUIREMENT• PHYSICAL FITPHYSICAL FIT

• ENVIRONMENTAL & STRUCTURALENVIRONMENTAL & STRUCTURAL• FLUTTERFLUTTER

• PERFORMANCE & HANDLING QUALITIES PERFORMANCE & HANDLING QUALITIES • STORE EMPLOYMENT & JETTISONSTORE EMPLOYMENT & JETTISON

• FUNCTION / BALLISTICS & OFP FUNCTION / BALLISTICS & OFP VERIFICATIONVERIFICATION

• SAFE ESCAPE & TEMPLATESSAFE ESCAPE & TEMPLATES• SYSTEM SAFETY & TESTING SMARTSYSTEM SAFETY & TESTING SMART

• -> OPERATING CONFIGS / LIMITS & PUBS-> OPERATING CONFIGS / LIMITS & PUBS

• OPERATIONAL CONCEPT / REQUIREMENTOPERATIONAL CONCEPT / REQUIREMENT• PHYSICAL FITPHYSICAL FIT

• ENVIRONMENTAL & STRUCTURALENVIRONMENTAL & STRUCTURAL• FLUTTERFLUTTER

• PERFORMANCE & HANDLING QUALITIES PERFORMANCE & HANDLING QUALITIES • STORE EMPLOYMENT & JETTISONSTORE EMPLOYMENT & JETTISON

• FUNCTION / BALLISTICS & OFP FUNCTION / BALLISTICS & OFP VERIFICATIONVERIFICATION

• SAFE ESCAPE & TEMPLATESSAFE ESCAPE & TEMPLATES• SYSTEM SAFETY & TESTING SMARTSYSTEM SAFETY & TESTING SMART

• -> OPERATING CONFIGS / LIMITS & PUBS-> OPERATING CONFIGS / LIMITS & PUBS

Aircraft Mission Systems CompatibilityAircraft Mission Systems Compatibility

Significant Changes• Handing / Flying Qualities• Aerodynamic Shape,• Primary Aircraft or Store

Structural Characteristics,• Aeroelastic or Mass Distribution,• C of G Shift of 12.7 mm,• Store Weight Change of 5%,• Pitch, Roll or Yaw Moments of

10%, Interface Connections,• Suspension Lug Location,• Environmental Tolerances,• EMI/EMC Degradation,• HERO Degradation

• Role• Functional Concept or

Delivery Modes,• Any OFP or SMS change,• Fuze, Safe/Arm • Any MACS Change,• Aircraft Thrust, Store

Ballistics and/or Propulsion,

• Explosive or Fragmentation Performance,

• New or Amended Nomenclature.

MIL-HDBK-1763• Ground Test Procedures - Appendix A • 100 Fit and Function Tests

– 101 Fit Test– 102 Function Test

• 110 Static Ejection Test• 120 Aeroelastic Ground Vibration Test (GVT)• 130 Structural Integrity Tests

– 131 Aircraft-Store/Suspension Equipment Structural Integrity Ground Test– 132 Carrier Suitability Test

• 140 Wind Tunnel Tests – 141 Effects of Aircraft on Captive Stores/Suspension Equipment – 142 Effect of Captive Stores/Suspension Equipment on Aircraft – 143 Aeroelastic Effects Test – 144 Separation Tests

• 150 Environmental Tests – 151 Vibration Test – 152 Aeroacoustic Test – 153 Hazards of Electromagnetic Radiation to Ordnance (HERO) Test – 154 Electromagnetic Compatibility / EM Interference (EMC/EMI) Test – 155 Temperature Extremes and Thermal Tests

• 160 Gun/Rocket/Missile Firing Tests – 161 Gun Firing Test – 162 Rocket/Missile Firing Test

MIL-HDBK-1763• Flight Test Procedures - Appendix B • 200 Inflight Loads Test • 210 Flutter Test • 220 Environmental Tests

– 221 Vibration Test – 222 Aeroacoustic Test – 223 Thermal Test – 224 EMC/EMI Test

• 230 Flying Qualities Test • 240 Performance and Drag Tests • 250 Captive Flight Profile (CFP) Tests

– 251 Handling Qualities Test – 252 Structural Integrity Test – 253 Endurance Test

• 260 Carrier Suitability Test • 270 Employment Tests

– 271 Release Test – 272 Launch Test or Weapons Survey and Demonstrations (for the Army) – 273 Gun Firing Test – 274 Dispense Test

• 280 Jettison Test • 290 Ballistics Tests

– 291 Weapon Freestream Ballistics Test – 292 OFP Ballistics Evaluation Test – 293 Separation Effects Derivation Test – 294 OFP Ballistics Verification Test

MIL-HDBK-1763 Tests 253 & 271

DT&E - F-111G with SSB / GBU-39 CFP

DT&E - F-111G with PLOCAAS – Active Suppression

Test Planning & ReportingDescribe (see also MIL-HDBK-1763 & MIL-STD-831):• Items Under Test – expected and actual• Outline Test Objectives (the COIs and CTPs)• Method of Test• Sequence of tests and fall-back plans – test cards set-up for each

test come later• Review of Training Currency and Needs for Test Team for MoT• Review of Test Locations – before flights, T/O, transit, tests

themselves, termination systems, RTB, landing & recovery, after flight

• Review of Instrumentation & Analyses (has it actually been done this way before and using the Team?)

• Review of Safety and risks – personnel, test items, RF, facilities • Write down everything objectively in third person as to what needs

to be done and what happened• Recommendations & Conclusions must address Test Objectives

and all findings – the Test Report needs to provide details so others can move on

AcknowledgementsElements of this presentation is based on:– Universal Principles of Design (2003)

William Lidwell, Kritina Holden, Jill ButlerISBN1-59253-007-9

– MIL-HDBK-1763 / 244 / AAP 7001.067, Air Armament

– MIL-STD-3014

– Related Works:• Power to the Edge (2005)

Alberts & Hayes, www.dodccrp.org• Critical Mass (2004)

Philip Ball, ISBN 0-09-945786-5• Why most things fail (2005)

Paul Ormerod, ISBN 0-571-22013-4• The Wisdom of crowds (2004)

James Surowiecki, ISBN 0-349-11605-9• Australian aircraft stores capabilities in a network enabled world (2005)

Malcolm Tutty, University of South Australia, January 2005– Defence Capability Development Manual (2008)

http://www.defence.gov.au/capability/

UN Survey for Mal• The question: “Please give your honest opinion about the shortage of

experimentation in complex systems in the rest of the world?“The survey was a HUGE failure because:1. In Eastern Europe they didn't know what "honest" meant.2. In Western Europe they didn't know what "shortage" meant.3. In Africa they didn't know what “experimentation" meant.4. In China they didn't know what "opinion" meant5. In the Middle East they didn't know what "solution" meant6. In South America they didn't know what "please" meant7. In the USA they didn't know what "the rest of the world" meant8. In Australia they hung up as soon as they heard the Indian accent.