I n t e g r i t y - S e r v i c e - E x c e l l e n c e

Air Force Institute of Technology

Major Richard ‘Kelly’ BullockDr Richard F. Deckro

Department of Operational SciencesAir Force Institute of TechnologyWright-Patterson AFB, OHUSA

Foundations forMeasuring Effectiveness

30 August 2005

I n t e g r i t y - S e r v i c e - E x c e l l e n c e 2

Research Motivation Accurately predicting outcomes and assessing progress has challenged

leaders since the earliest use of national power to achieve political aims Especially true of military power

Many theories on how to employ national power to achieve desired end-states

Douhet, Jomini, Trenchard, Mitchell, Schelling,…

Effects-based Operations Not new, but efforts to ‘institutionalize’ EBO concepts are

US Joint doctrine and service doctrine, particularly USAF doctrine, has undergone change to reflect EBO concepts

History has shown theory is of little value if not supported by an empirically feasible measurement method

One accurate measurement is worth a thousand expert opinions.– ADMIRAL GRACE HOPPER, 1906 – 1992

I n t e g r i t y - S e r v i c e - E x c e l l e n c e 3

Patton on MOEs…

“How many officers did you lose today?” asked Patton. “We were fortunate,” Ward replied. “We didn’t lose any officers.” “Goddamit, Ward, that’s not fortunate! That’s bad for the morale of the enlisted men. I want you to get more officers killed.” A brief pause followed before Ward said, “You’re not serious, are you?” “Yes, goddamit, I’m serious! I want you to put some officers out as observers,” said Patton. “Keep them well up front until a couple get killed. It’s good for enlisted morale.”

Discussion between General George Patton and General Orlando Ward during WWII:

I n t e g r i t y - S e r v i c e - E x c e l l e n c e 4

Overview

Background Measurement Concepts Measurement Theory Application of Measurement Effects-based Operations Effects

Effectiveness Measurement Foundations Definitions Concepts Framework

Takeaways

I n t e g r i t y - S e r v i c e - E x c e l l e n c e 5

BackgroundMeasurement Concepts

Measurement is the objective representation of real-world objects, processes, and phenomenon

Measurement captures information through attributes Attributes may not be directly observable

A real-world system is defined by the attributes chosen to represent it Measurement is an abstraction

Measurement is the process of assigning symbols to a system attribute such that the assigned symbols reflect the underlying nature of the attributes

Set X Set Y

A

B

C

D

E 1.2

3.7

3.12.5

6.2

5.48.9

measurement)(XfY

Set X Set Y

A

B

C

D

E 1.2

3.7

3.12.5

6.2

5.48.9

measurement)(XfY

I n t e g r i t y - S e r v i c e - E x c e l l e n c e 6

BackgroundMeasurement Concepts

To measure is to know.– LORD KELVIN, 1824 – 1907

validity – how well a measure reflects what it was supposed to represent reliability – how consistent or repeatable the measurement process isamplitude – how well a measure represents higher order constructs

random – ‘noise’ variation from any source impacting the system systemic – (bias) derives from construction of the measure or definition of the measurement processobservational – oversight of a key system attribute or using the wrong measure for a system attribute Statistical Theory – concerns making inferences from data Measurement Theory – link between data and real-world

Empirical

System

Object,Process, or

Phenomenon

Abstract Attributes

Measure

Numbers

Numerals

Assessment

Reasoning &Mathematics

Insights &Information

= source for potential error

I n t e g r i t y - S e r v i c e - E x c e l l e n c e 7

BackgroundMeasurement Theory

Measurement Theory is a branch of applied mathematics that attempts to: Describe, categorize, and evaluate the quality of measurements Improve the usefulness, accuracy, and meaningfulness of measurements

Representational view of measurement built upon 3 theorems

Representation – Proves the measure f preserves the relationships in X after mapping into Y (i.e. model validation)

Uniqueness – Characterizes the family of measures that preserve the real-world relationships (i.e. scale type)

Meaningfulness – Shows resultant is invariant to scale change for measures meeting the uniqueness condition

AB

C

D

E

Set X

1.2

3.7

3.12.5

6.2

5.48.9

Set Y

measurement)(XfY

NominalOrdinal

Interval

Ratio

Absolute

I n t e g r i t y - S e r v i c e - E x c e l l e n c e 8

BackgroundApplication of Measurement

Application of measurement is… an art the mechanism for extracting information from empirical

observation within a specified context the translation of behavior into a set of ‘vital signs’ indicating

variations in behavior Application guidelines

Frameworks Vertical Horizontal

Effectiveness vs. Efficiency Measure Types & Characteristics

...things are to you such as they appear to you and to me such as they appear to me... – PROTAGORAS, 485 – 421 B.C.

I n t e g r i t y - S e r v i c e - E x c e l l e n c e 9

BackgroundFrameworks

Vertical (or Hierarchical) Structure

Horizontal Structure

FundamentalObjective

Sub-Objective

O1

Sub-Objective

O2

Sub-Objective

OX

ValueV1

ValueV2

ValueVY

Attribute1 Attribute2 AttributeZ

Objective11

Objective12

Objective1M

Objective21

Objective22

Objective23

Objective24

Objective2N

Objective31

Objective32

Objective33

Objective34

Objective35

Objective36

Objective3P

What?

Why?

How?

Effectiveness

Efficiency

We must make the important measurable, not the measurable important. – ROBERT MCNAMARA, 1916 –

Measure of Effectiveness (MOE)

Measure of Performance (MOP)

I n t e g r i t y - S e r v i c e - E x c e l l e n c e 10

BackgroundMeasure Types & Characteristics

Natural Constructed

Direct- Commonly understood measures directly linked to

fundamental objective- Example: Profit

- Measures directly linked to the fundamental objective but developed for a specific purpose

- Example: Gymnastics scoring

Proxy- In general use measures focused on an objective

correlated with the fundamental objective- Example: GNP (economic well being)

- Measures developed for a specific purpose focused on an objective correlated to the strategic objective

- Example: Student grades

Strategically-linked – traceable to fundamental objective; responsive to change and provides indication of how much change can be attributed to an input

Timely – able to be collected and processed in a timeframe needed to be relevant within the context

Objective – 1) Collection: easy to understand, same regardless of the assessor, same under similar circumstances; ‘face-value’ or whether the measure logically represents what it is supposed to represent 2) Interpretation: unambiguous interpretation; distinguish between desired and undesired consequences

Economical – Collection and processing should provide benefits that off-set the burden of measurement activities

Complete – Measures should address all areas of concern in enough detail to discern reasons for differences in actual and expected system results

Measurable – hold for Measurement Theory conditions; within a given context if the measure can be feasibly obtained with available resources (i.e. operational )

I n t e g r i t y - S e r v i c e - E x c e l l e n c e 11

BackgroundEBO Overview

Effects-based Operations (EBO) Theory for the employment of capabilities in dynamic and uncertain

environments in a manner to best attain objectives Provides a conceptual framework for determining the integration and

application of capabilities to achieve specific effects to influence the environment of interest yielding desired outcomes

Key tenets Focus on end outcomes Reduced emphasis on weapon systems De-emphasis on destruction as a sole means of achieving effects

“You begin with the objectives that indicate what you really care about. Then you follow simple logical reasoning processes to identify the mechanisms by which the objectives can be achieved. Finally, for each mechanism, you create alternatives by asking what control you have over that mechanism.”

Keeney, Ralph, Value Focused Thinking: A Path to Creative Decision Making, Harvard University Press, Cambridge, 1992.

Not everything that can be counted counts, and not everything that counts can be counted. – ALBERT EINSTEIN, 1879 – 1955

I n t e g r i t y - S e r v i c e - E x c e l l e n c e 12

Planning – focus on desired end-state; develop strategy (who, what, where, why, when, and how) Employment – efficient employment of capabilities; increased emphasis on non-lethal means Assessment – determining if the intended effects were achieved and if they are shaping the desired outcomes

BackgroundEBO Overview

Empl

oym

ent

Ass

essm

ent

Plan

ning

Effects-based Operationsmeasures

measures

measures

measures

measures

measures

measures

measures

measures

measures

measures

measures

Measures of Effectiveness – tie pillars together; provide feedback on strategy progress towards desired end-state

I n t e g r i t y - S e r v i c e - E x c e l l e n c e 13

BackgroundEffects Overview

Effect Consequence or result of a particular action

TacticalOperational

StrategicLevel

PhysicalFunctionalSystemic

Psychological

Domain

PermanentNon-temporalPersistence

PositiveNegativeResult

IntendedUn-intended (Collateral)Intent

CascadingCumulativeImpact

ParallelSequentialTiming

Direct (First-order)Indirect (Higher-order)Order

TypesAttribute

TacticalOperational

StrategicLevel

PhysicalFunctionalSystemic

Psychological

Domain

PermanentNon-temporalPersistence

PositiveNegativeResult

IntendedUn-intended (Collateral)Intent

CascadingCumulativeImpact

ParallelSequentialTiming

Direct (First-order)Indirect (Higher-order)Order

TypesAttribute Effect System state change Attribute Types

Order Direct (First-order)Indirect (Higher-order)

Timing ParallelSequential

Impact CascadingCumulative

Intent IntendedUn-intended (Collateral)

Result PositiveNegative

Persistence PermanentTemporal

Domain

PhysicalFunctionalSystemic

Psychological

LevelTactical

OperationalStrategic

System

SystemBoundary

Input

Input

Input

Input

Input

Input

Input

Input

Input

Input

Mechanism

Effect

DirectParallelIntendedPositivePhysical

Effect

DirectParallelIntendedPositiveFunctional

Effect

DirectParallelIntendedNegativePsychological

Effect

DirectParallelIntendedPositivePsychological

Effect

DirectSequentialIntendedPositiveFunctional

Tactical

Effect

IndirectParallelIntendedPositiveFunctional

Effect

IndirectUnintendedNegativePsychological

Effect

IndirectParallelIntendedPositivePsychological

Effect

IndirectSequentialCascadingIntendedPositiveFunctional

Effect

IndirectUnintendedPositiveFunctional

Operational

Effect

IndirectSequentialCascadingIntendedPositiveFunctional

Effect

IndirectSequentialCascadingUnintendedPositiveFunctional

Effect

IndirectCascadingUnintendedNegativePsychological

Effect

IndirectSequentialCumulativeIntendedPositiveSystemic

Effect

IndirectUnintendedNegativePsychological

Strategic

Input

Input

Input

Behavior

Effect

IndirectParallelIntendedPositiveFunctional

Effect

IndirectUnintendedNegativePsychological

Effect

IndirectParallelCumulativeIntendedPositiveSystemic

Input

I n t e g r i t y - S e r v i c e - E x c e l l e n c e 14

ApproachDefinitions

The beginning of wisdom is calling things by their right names.– CONFUCIUS, 551 – 479 B.C.

DEFINITION 1: A SYSTEM is a set of elements where relationships exist between the elements and the SYSTEM has a purpose or normative behavior.

DEFINITION 2: A system ELEMENT, or SUBSYSTEM, is a system providing functionality or support to a parent system.

DEFINITION 3: A MODEL is a formal image of an empirical structure.

DEFINITION 4: An ATTRIBUTE, or NODE, is a characteristic, feature, or property of a system that is directly or indirectly observable.

DEFINITION 5: A MEASURE is a model of an attribute.

DEFINITION 6: A MEASUREMENT, or observation, is a particular manifestation, or instantiation, of an attribute.

DEFINITION 7: A system STATE is a particular instantiation of all system attributes, or state variables, at a particular point in time.

DEFINITION 8: An EFFECT is a system state change.

DEFINITION 9: EFFECTIVENESS gauges the magnitude of a system state change.

DEFINITION 10:An END-STATE characterizes the desired measurements for all system attributes, or state variables.

I n t e g r i t y - S e r v i c e - E x c e l l e n c e 15

ApproachProduct Structure

m

i 1

1. System Identification• Determines system boundary• Covers all pertinent aspects of desired end-state

2. Sub-system Identification• Decomposes into smaller parts• Only relevant sub-systems are needed

3. Sub-system Importance• Discerns relative importance among selected sub-systems• Amounts to weighting each sub-system

4. Attribute (Node) Identification• Characterizes each sub-system by its salient features• Only attributes relevant to context are needed

5. Attribute (Node) Importance• Discerns relative importance among selected attributes (nodes)• Amounts to weighting each attribute (node)

6. Measure Development• Quantification of attributes (nodes)• Constructed measures may be needed• Measures should hold for metric properties

Product Structure Development Process Empirical System

Formal System(model or product structure)

System

Sub-system1 Sub-systemn

n

i 1

sub-systemweight = 1

Attribute1 Attributemattributeweight = 1

Measure1 MeasuremMeasureq

q

i 1measure = 1

I n t e g r i t y - S e r v i c e - E x c e l l e n c e 16

ApproachConcepts

EmpiricalSystem

A

FormalSystem

(or Model)xA

xA

EmpiricalSystem

A = < a1,…, an >

Element(Subsystem)

ai

FormalSystem

(or Model)xA = < x1, x2,…, xn-1, xn >

xA

x1 xnxn-1x2

System

Elements(Subsystems)

Attribute(Node)

Link

1 mn-12

Attributes(Nodes)

ά1 άmάn-1ά2Measures

Measurements(Observations)

Timet t = 0

SystemMeasurement xA = < 0, 0,…, 0, 0 >

Input

t = T

Mechanism

xA

x1 xnxn-1x2

1 mn-12

ά1 άmάn-1ά2

Measurements(Observations)

xA = < .2, .7,…, .3, .5 >

Effect(of inputt = 0 at t = T)

Effectiveness(of inputt = 0 at t = T)

t = TEnd-state

xA

x1 xnxn-1x2

1 mn-12

ά1 άmάn-1ά2

Measurements(Observations)

xA = < 1, 1,…, 1, 1 >

Count what is countable, measure what is measurable, and what is not measurable, make measurable...– GALILEO GALILEI, 1564 – 1642

I n t e g r i t y - S e r v i c e - E x c e l l e n c e 17

Takeaways Measurement turns real-world system behavior into a set of ‘vital signs’ for the

purpose of monitoring the system Measurement Theory provides a robust basis for turning qualitative observations

into quantitative data Effectiveness vs. Efficiency

Measure of Effectiveness (MOE) External measure Invariant to means of achievement Are we doing the right things?

Measure of Performance (MOP) Internal measure Coupled to means of achievement Are we doing things right?

EBO is an instantiation of an existing, structured, decision-theoretic framework (VFT) for approaching problems with abstract or ill-defined objectives

An effect is a system state change Effectiveness gauges the magnitude of the change

I n t e g r i t y - S e r v i c e - E x c e l l e n c e 18

Questions&

DiscussionA prudent question is one-half of wisdom.

– SIR FRANCIS BACON, 1561 – 1626