Methods and Metrics for Analysis of Sensemaking

Dr Karen Carr & Mr Barry McGuinness

BAE SYSTEMS

Advanced Technology Centre

Our objectives

• For this meeting:

– Contribute what makes sense to us, in our given context and with our goals

• In our work:

– Develop the ability to supply C3I ‘capabilities’ (in partnership)– Systems Engineering of socio-technical systems– Driven by need to deliver usable and demonstrable results – Science as well as engineering and domain expertise

NB We want to ensure that human issues drive the developments - but we don’t want to forget that we have to

inform technology (as well as organization, process)

What we mean by sensemaking

Why we want to use this concept to try and answer the questions we need to answer

• Our question:

“How can we develop technology, design and manage systems which support/ enhance the human roles in defence operations?”

– Significant human role is ability to adapt, respond to unexpected, creativity, play mind games, etc. Need to preserve & enhance that - not interfere.

– Support is needed for dealing with the unexpected, the unknown, as well as recognisable situations

– Include broad System of Systems issues, developers, rapid change

Sensemaking (& Situational Awareness) is a working concept to enable us to start manipulating, analysing, and measuring context, goals and human performance

Why we want methods and metrics for studying sensemaking

• Need to attribute effects - predict - in order to provide support.

• Move from concepts to metrics to analysis to (testable) models.

• Reduce subjective bias (influence of our own sensemaking, interpretation)

• No existing clear metrics we can use - no absolutes

1. Understand how human performs, and what conditions facilitate ‘good’ performance (what hinders)

2. Identify the properties of organisation, process, technology, training, etc which are important for success

3. Develop design and management methods and tools to enable implementation

• NB not necessarily numbers - could be properties

Range of methods

• observation (non intrusive)

• subjective investigation (e.g. ethnographic, knowledge elicitation)

• storytelling/anecdotes (knowledge building)

• metaphor (pattern matching)

• scientific method (empirical hypothesis testing)

• mathematical analysis (baseline)

Methods and Metrics

• Concepts

• Metrics

• Some analyses

• Implications for sensemaking

Concepts

• Orientation

– complex, uncertain situations– SA determines capacity to decide and act– sensemaking determines SA– cognitive processes are intrinsically goal-directed– people form nested hierarchy of processes & outcomes

• Objectives

– Understand SA and sensemaking– Feed into design & development of information systems and

human systems– Applied research -- theory into practice

What is a Situation?

• A situation is a pattern in state space, especially one which appears to deviate from a “normal” intended or expected pattern.

• Example:- aircraft fuel x time into flight

Aircraftfuellevel

Time intoFlight

Unexpected rate-- we have a situation!

Normal takeoff

Normal cruise

Unrecognized Patterns

• An unrecognized pattern demands attention.

Attention!

Perceivedpattern

???

Knownpatterns?

Unknown pattern?

perception comprehension

Definitions

Knowledge:

= capacity for “action”

Situational Awareness:

= dynamic “situated” knowledge, i.e. capacity to act effectively here & now in a given specific situation

Sensemaking:

= process of creating effective SA in situations of uncertainty

“Knowing what’s going onso you can figure out what to do.”

• doing• saying• thinking

Situational awareness

Dynamic mental representation of the current and future state of one’s domain of action

– includes awareness of • environment• entities • events • processes • actions • others’ perceptions & intentions

– insofar as these are relevant to • performing an action, or • choosing a course of action

Through a continuous process of situation assessment

Situational awareness

SA is based on ...

• prior KNOWLEDGE– SCHEMAS: generalized patterns representing typical situations– based on experience, training, culture

• recent INFORMATION– direct perception of the environment– perception of instruments and displays – received communications

SA

INFORMATION

KNOWLEDGE

PHYSICAL DOMAIN

instruments communications

Central role of SA

SA

Decision-making

Actionperformance

PHYSICAL DOMAIN

Informationacquisition

Sense- making

• SA both informs and is informed by • sense-making• decision-making

COGNITIVE DOMAIN

Inside SA: Cutting up the cake

Concrete(situation-specific)

Abstract(generalized patterns)

Observed Implied

Information

specific propositionse.g., “rate of fuel

loss is high”

Intentions

selected actions afforded by situation

e.g., “Contact ATCand inform”

Models

situational schemase.g., “Fuel leak?”“Faulty sensor?”

Projections

mental simulationse.g., “Risk of not

reaching destination”

Processes involved in SA

PERCEPTION Acquisition of information about

the given situation

COMPREHENSION Diagnostic interpretation of

the given situation

PROJECTION Prognostic simulation of future situations

and their possible outcomes

RESOLUTION Selection of actions to direct the given

situation towards the desired outcome

… All serving to support dynamically effective action

information

models

intentions

projections

Sensemaking and SA

Information

Models

Intentions

Projections

PERCEPTION

COMPREHENSION PROJECTION

RESOLUTION

Decisionmaking

Sensemaking

Sensing Acting

Sense-making: when comprehension is uncertainDecision-making: when resolution is uncertain

Metacognition

• Defined as:

– “Thinking about thinking” or “knowledge about knowledge”– i.e. “Awareness of your own SA”

• noticing uncertainties, gaps, conflicts in your mental reps

• identifying information needs

• employing strategies for sensemaking & decision-making

SA

“It’s like looking over your own

shoulder.”

?

Gives a subjectivesense of SA

SA and metacognition

• Four possible states:

AppropriateConfidence

(ideal state)

InappropriateConfidence(danger state)

InappropriateSensemaking

AppropriateSensemaking

True SA False SA

Confident in SA

Not confident in SA

Actual awareness:

Need for sensemaking

Subjective attitude

Team SA and shared SA

• Not the same thing

• Team SA = sum of current knowledge held across a team, irrespective of who has it

• Shared SA = those parts of the team SA that are common between team members

Shared SA

Team SA

Personal SA

What to share, with whom?

• The nature of SA in groups is dictated by goals

• Goals are hierarchic

• Top-level goals are shared by all members

– therefore need shared SA with respect to that objective

• Lower-level goals are specific to individuals

– therefore need personal SA with respect to own task

• Sharing one’s SA is necessary only to the extent that the knowledge has bearing on the goals of others

Team SA and shared SA

• Shared SA elements can be differentially allocated

perception

perception

perception

comprehensionprojection

resolution

perceptionresolution

perceptionresolution

perceptionresolution

projection

comprehension

Distributed SA in the C2 HQ

Ops

Signal Commander

Intel

Intentions(RESOLUTION)

Information(PERCEPTION)

Models(COMPREHENSION)

Metacognition?

Projections(PROJECTION)

So...

• Explicit sensemaking processes are needed when comprehension cannot easily occur

• Sensemaking requires metacognitive awareness of own knowledge -- uncertainties, gaps

• Metacognitive assessments can be wrong and lead to inappropriate subjective attitude -- and inappropriate behaviour

Measuring SA

• COGNITIVE approach

– queries about the situation – Reveals mental reps

• Multiple choice (SAGAT)• True/False (QUASA)• Sit Reps

• SUBJECTIVE approach

– self-ratings of SA – Reveals metacognitive state

• Unidimensional (SARS)• Multidimensional (SART)• Multidimensional and intelligible! (CARS)

• OBJECTIVE approach– behavioural & physiological correlates – Reveals changes in metacognitive state

• EEG, fMRI• Eye pointing

As a rule, take both cognitive & subjective measures together.

CARS

• Crew Awareness Rating Scale

• a subjective tool to elicit operator’s subjective sense of SA

• multi-dimensional

• generic, adaptable, easy to use

Dimensions

Knowledge Processing

Perception

Comprehension

Projection

Resolution

Eight CARS questions

1. the most recent information2. what is really going on here3. what could happen4. what actions should be taken

1. monitor the flow of information2. understand the big picture3. predict how it is likely to evolve4. decide what actions to take

Would you say you have a good sense of …

Would you say it is easyfor you to …

knowledge

processing

Six possible responses

YES

NO

Certain Uncertain

Definitely

Definitely not Think not

Think so

For sure?

Don’t know

Don’t need it

Do I ?

CARS results

CONTENT• Perception ||| ||| | |• Comprehension | |||| || |• Projection || |||| | |• Resolution | ||| ||| |

PROCESSING• Perception |||||| ||• Comprehension ||| ||| ||• Projection || || ||| |• Resolution || |||| ||

Def Prob Prob not Def not DK NA

CARS results

0

40

80

20

60

% o

f ra

tings

100

Run 1 Run 2 Run 3 Run 4

1. Definitely

2. Probably

3. Probably NOT

4. Definitely NOT

1. Definitely

2. Probably

3. Probably NOT

4. Definitely NOT

Comprehension knowledge over time

QUASA

• Quantitative Assessment of Situational Awareness

• a probe tool to elicit operator’s actual SA

• mathematical : based on SDT

• still under development, but promising

QUASA

• Signal Detection TheoryYES!

perception discrimination

Square?

• Targets vs non-targets• Hits, False Alarms, Good Misses, False Rejections• Also applies to internal (mental) representations

QUASA

• “Is this item true?”

– Confidence in perceived truth value of items varies

Confidence in truth value of items

Num

ber

of it

ems

Max SENSITIVITY= ideal SA

Weak Strong

TRUEitems

FALSEitems

QUASA

Confidence in truth of items

Num

ber

of it

ems

No sensitivity,poor SA

Weak Strong

Literally can’t tell the difference between true & false items…

They have similar-strength levels of confidence

QUASA

Confidence in truth value of items

Num

ber

of it

ems

Max NEGATIVE sensitivity= worst case SA

Weak Strong

FALSEitems

TRUEitems

Deception

QUASA

Confidence in truth value of items

Num

ber

of it

ems

SA’

IB’’Weak Strong

Some positive sensitivityLow positive bias

(acceptance threshold)

Bad acceptancesBad rejections

Good rejectionsGood acceptances

QUASA

Example probe:“ The tanks adjacent to bridge are enemy ”

Response: YES (accept as true) or NO (reject as false)

Evaluate:Sensitivity (discrimination of true/false situations) = SA’Bias (probability of item acceptance/rejection) = IB’’

QUASA

Maximumpositive

sensitivity:ideal SA

Maximumnegative

sensitivity:the wrong situation!

Maximumpositive

bias:too rash

Maximumnegative

bias:too cautious

TYPICAL

+100

0

-100

QUASA

Perception: information

Comprehension: model of situation

Projection:Future

developments

Resolution:CoA intention

+100

0

-100

Dynamic SA - D

-100

-80

-60

-40

-20

0

20

40

60

80

100

4 5 6 7 8 9 10 11 12 13 14 15 16 17

Last turn of block

SA' score

Info bias (IB'')

QUASA

• Mathematical assessment of SA

• Needs the truth!

• SA, bias, components, temporal

• ? Team & shared SA

Behavioural correlate of SA

Tracking eye-point-of-gaze (EPOG)

Do EPOG patternscorrelate with SA?

CRT CRT CRTCRT

ECAM(engines)ND & EFISPFD

FMS

throttles

touch panel(optional for electroniccharts, datalink)

touch panel(systems control display)

COCKPIT LAYOUTforward view

displays and controls

ND & EFIS PFD

DisplayControl

Unit

Navcontrolssidestick

priorityPR ALT

SPDALT

HDG

VSA/P

on/off

mode control panel(Flight Control Unit)

spoilers & flaps indicators

LDG GEAR controls

flapsspoilers

sidestick

Commscontrol

unit

ALT autopilot altitude controls & indicatorA/P on/off autopilot master on/off toggleCRT Cathode ray tubeECAM Electronic Centralised Aircraft MonitoringEFIS Electronic Flight Instrumentation SystemFMS Flight Management SystemHDG autopilot heading controls & indicatorLG GEAR landing gearND Nav DisplayPFD Primary Flight DisplayPR ALT pressure (altimeter) setting & indicatorSPD speedVS autopilot vertical speed controls & indicator

ECAM(systems)

EPOG research

‘Entropy’ = known loss of SA

SA and flightdeck automation

Collision avoidance system

+01

Heathrow control this is Speedbird five five, descending now to flight level one four.

Speedbird five five, Heathrow control, roger, descending to flight level one four.

Heathrown control this is Delta four zero four, flight level two zero, request descent clearance.

Delta four zero four, Heathrow control, what is your present altitude?...

Radio “party line”

SA and flightdeck automation

Traffic Situation Reporting

Detectableaircraft

Non-Detectableaircraft

50%

25%

0%

With automation

Conventional

Reportedaircraft

SA and C2 digitization - ISTAR

Own force positionsEnemy positions

BGHQ crewstationCommon Operational Picture

SA and C2 digitization - ISTAR

Battlespace digitization demonstratorSynthetic

environment

SA and C2 digitization - ISTAR

•2-hr ISTAR recce operation

•Performed with voice AND digital C2 systems

Measures taken of mental workload & situational awareness

SA and C2 digitization - ISTAR

rating

voicedigital

aspects of SA (knowledge of enemy)

PERCEPTION

DEF

PROB NOT

DEF NOT

PROB

PROJECTION RESOLUTIONCOMPREHENSION

Some implications

Both actual SA and subjective sense of SA affect decision-making & performance

Technology can affect SA for better or worse

Analyses with metrics provide specific insights

Other work

• DS1 trials

• BattleLab trials

• Cognitive modelling

– COGNET in C2 environment– Ideal Decision Maker

• ? Can be used to predict dips in SA and sensemaking needs

Building Industry-MoD partnership

Implications for sensemaking

• Thinking about thinking

• Concepts : sensemaking as processes supporting SA

• Role of metacognition : group context

• Metrics of SA : can be used to evaluate sensemaking solutions

• Data can feed development of predictive models

• Knowing what’s going on so we can figure out what to do!