Symposium on "The Effectiveness of Modelling and Simulation” MSG-031 Briefing on Progress so farPoland, 13 October 2005 Progress to date on NMSG 031 The

Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation” Symposium on "The Effectiveness of Modelling and Simulation”

MSG-031 Briefing on Progress so far Poland, 13 October 2005

Progress to date on NMSG 031 The Cost Effectiveness of Modelling

and Simulation

A Presentation by Gary Summons (QinetiQ) and Lt Cdr Alastair Macdonald (MOD)



Introduction/Background:

• The use of modelling and simulation (M&S) has grown rapidly

• Continued growth will depend on M&S being seen as cost effective and adding value.

• Programmes generally make a specific business case for any M&S outlay and there is a need for generic business case guidance.

• By identifying non cost effective application areas, the barriers preventing the proliferation of M&S will be identified providing an indication of future research priorities.



Aims

• Provide a framework to collect evidence associated with the cost effective application of M&S based upon the NATO procurement process:

– Look at how the use of M&S can be categorised and mapped to the NATO process and what measures of effectiveness should be documented.

– Produce a report discussing the cost effectiveness associated with the exploitation of M&S and providing evidence to support these arguments where it exists.

• This presentation will provide an insight in the work to date that the member countries have undertaken.



• Members of the NMSG– Canada– France – Germany– Netherlands– UK– US



NATO Procurement Process• In order to place information gathered into a useful context it was decided to use the

procurement process as a framework. – Allows end users to see both the applicability of M&S as well as its Cost Effectiveness and relate it to

their particular issues

• Each Nation has it’s own “flavour” of the procurement process it was therefore agreed to adopt the NATO instantiation and map each individual country’s to it– Each member country produced it’s own mapping following consultation with appropriate home nation

procurement specialist

– While it did not provide a “one to one” mapping for all countries there was a broad alignment

– Key to this activity was an understanding of decision points and outputs required at each stage



NATO Procurement Process• Mission need evaluation

• Pre-Feasibility

• Feasibility

• Project definition

• Design

• Production

• In-Service

Mis

sio

n

ne

ed

e

valu

atio

n

Pre

-fe

asi

bili

ty

Fe

asi

bili

ty

Pro

ject

d

efin

itio

n

De

sig

n

Pro

du

ctio

n

In s

erv

ice



NATO Procurement Process• Example showing the NATO process with UK’s mapped

Mis

sio

n

ne

ed

e

valu

atio

n

Pre

-fe

asi

bili

ty

Fe

asi

bili

ty

Pro

ject

d

efin

itio

n

De

sig

n

Pro

du

ctio

n

In s

erv

ice

Pre

-co

nce

pt

Ma

nu

fact

ure

De

mo

nst

ratio

n

Ass

ess

me

nt

Co

nce

pt

UK

In s

erv

ice

Dis

po

sal



Modelling and Simulation (M&S) Classification• A way to provide common classification of M&S use is required• 3 separate views of this were investigated

– Application– Category– Generic use

• These classifications were then combined in a matrix with the NATO procurement process to provide an indication of M&S applicability



Defence Applications for M&S

• Concept Development– Requirements gathering– Doctrine development (concepts)

• Procurement (Capability) Development– Supporting procurement process– Test and evaluation

• Warfighting Development– Mission rehearsal– Training and education

• Research



M&S Categories• Component level modelling • Individual/team virtual environment• Team/collective enviroments• Constructive platform level interactive• Constructive aggregate level interactive• Constructive closed analytical model• LVC (Virtual Simulation with constructive, platform) • Pure visualisation• Infrastructure and servicesThe group adopted this series of M&S categories, in doing so it recognises that there are many other ways this could have been “sliced”



NLNL

Mission need evaluation

Pre feasibility Feasibility Project definition Design and development

Production In-service

Component level modelling

Individual/team virtual environment

Team/collective enviroments

Constructive platform level interactive

Costructive aggregate level interactive

Constructive closed analytical model

Virtual Simulation with constructive, platform

Pure visualisation

Infrastructure and services

Example of Procurement Process vs M&S Categories

Key Used often Used occasionally Used infrequently



Generic Uses

• The group also elected to look at generic uses of M&S• This approach allowed non M&S experts to evaluate the benefits of M&S to the procurement process• It is intended that both this and the M&S categories will point at the same underpinning evidence but will present it in different ways



Generic Uses• Analysis (of alternative) cost performance trades• (Systems) Engineering• Experimentation and Concept Development• Training • Test• Exercise• Operation



Mission need evaluation

Pre feasibility Feasibility Project definition Design and development

Production In-service

Analysis (of alternative) cost performance trades

(systems) engineering

Experimentation and Concept Development

Training

Test

Exercise

Operations

Key Good cost savingSome costing saving

Little or no cost saving

Procurement Process vs Generic Uses



Establishing the case for M&S• We can determine cost effectiveness by weighing the costs associated with M&S against the benefit accrued• However determining the contributing factors to cost is not quite as simple• Cost needs to be considered with the following in mind:

– Sunk costs, those which have already been invested– Costs incurred whether M&S is employed or whether a more traditional approach is used– Costs not incurred because M&S is used– Costs only incurred because M&S is employed

• In some cases the use of M&S will need consideration under one or more elements i.e. the re-use of a model will have an element of sunk but most likely incur a cost associated with it’s new use



Establishing the case for M&S• Benefits are more easily identified and can be classified as:

– Cost avoidance– Risk management– Quality of end product– Reduction in manpower costs– Reduction in Time– Reduction of Environmental impact– Increased capability– Reduction in High value asset dependency



Establishing the case for M&S• Once the evidence for cost and benefit has been collected it will be associated with the particular category of M&S, Generic use and Procurement phase• We will inform the factors associated with the cost effective application of M&S• We will make available the evidence gathered in a coherent fashion so that decision makers can make evidence based judgements on the applicability of M&S to their particular situation• Based on this evidence it will be possible to identify areas where the application of M&S is cost effective and those areas it is not



Evidence tool• To be useful all the evidence gathered needs to be readily accessible to end users and decision makers• A simple tool is required which will both present the existing evidence and aid the collection of new evidence• The tool must be able to present the information from the different views previously described:

– Stages of procurement process

– Different categories of M&S

– Different Generic Uses



Example of Query in

Evidence tool



Anticipated Results• Framework for cost effectiveness evaluation and comparison across projects/nations• More qualitative than quantitative evidence• Populated evidence schema• No general rule (one size does not fit all) difficult to generalise from specific examples



Early Conclusions• Difficulty in collecting and analyzing data

• Quantifiable evidence hard to source

• No magic bullet solution for Business Case

• This work provides a framework for cost effect analysis

• Template for the continued gathering of evidence

• Allows comparison of cost effectiveness across different projects/national boundaries

• Problems identifying ‘sunk’ costs

• Cost of non-M&S approach vs M&S approach i.e. costs are still incurred if M&S not used



Anticipated Recommendations• Develop the Evidence Tool schema as a web based solution on the NATO web• Nations to continue populating schema as evidence becomes available• Further work needed on “sunk” vs specific costs• Further work required on Metadata



NMSG 031 The Cost Effectiveness of Modelling and Simulation

Gary Summons [email protected] Cdr Alastair Macdonald [email protected]

Documents

Symposium on "The Effectiveness of Modelling and Simulation” MSG-031 Briefing on Progress so farPoland, 13 October 2005 Progress to date on NMSG 031 The