MIT ICATMIT ICAT

Supporting Pilots’ Time-Dependent

Information NeedsDuring Operations in Adverse Weather

Laurence Vigeant-Langlois R. John Hansman, Jr.langlois@mit.edu rjhans@mit.edu

MIT ICATMIT ICAT

Motivation

• Adverse Weather Significantly Impacts Flight Operations Safety -- 22. 5% All US Accidents Efficiency -- 17% / $1.7B per year Avoidable Weather Delays (Source: FAA)

• Multiple efforts to develop new weather information tools Cockpit weather datalink

• In order to develop safe and effective decision-support tools, it is important to: Understand users’ information needs and cognitive tasks Understand the implications and limitations of supporting their tasks

TurbulenceConvective Weather Icing

(Courtesy of NASA)

Ceiling & Visibility

MIT ICATMIT ICAT

Methodology

• Cognitive Analysis of Pilots Decisions

• Temporal Representation Framework Deterministic Regime Stochastic Regime

MIT ICATMIT ICAT

Human-Centered ApproachClosed Loop Feedback Process

Aircraft

AircraftSensors

NumericalWx Models

TrajectoryControl

Direct Weather Observation

SituationDynamics

PilotInteraction

Weather

AircraftState

Displays

Direct Aircraft State Observation

Sensors ModelHuman

OperatorDisplays

Aviation Weather Information

RemoteWx Sensors

SensorReadings

Reports

Fore-castsVoice

Datalink

In-SituWx Sensors

Dissemination

MIT ICATMIT ICAT

3 Projection DECISION

Monitor

Evaluate

Adjust

Select

Project UnderstandPerceive

Weather State

Aircraft State

Interaction

SITUATIONAL AWARENESS

Weather Phenomenology

Aircraft Enveloppe

FutureExposure

Weather Forecast

Aircraft Trajectory

PLAN

Formulate Nominal Plan

Formulate Alternative Plans

PERFORMANCE OF ACTIONS

Implement

Aircraft

SituationDynamics

Interaction

Weather

INFORMATION

Sensors

AircraftTrajectory

Control

Displays

WORKING MENTAL MODEL

Pilot

MIT ICATMIT ICAT

Route Selection

Go/No-Go

Tactical Avoidance

Escape

Situation Monitoring

Diversion

Non-Weather-RelatedReplanning

Aircraft Systems Management

Request for Weather Information

Use of Emergency Authority

Reactive Tactical Strategic

Route Planning

Planning Tasks of PilotsWeather-Related Decisions

Time

Short-term planning where safety goals

dominate the decisions

Planning for the remainder of the flight considering the main goals: safety, legality, efficiency, satisfactory level of comfort and service

Planning based on the main goals without

considering the entire remainder of the flight

MIT ICATMIT ICAT

Methodology

• Cognitive Analysis of Pilots Decisions

• Temporal Representation Framework Deterministic Regime Stochastic Regime

MIT ICATMIT ICATWEATHER REPRESENTATION

Historical Constant Stochastic

?

TIME

Temporal Regimes of Cognitive Processes

Time of Information Production

Reactive Tactical Strategic

FLIGHT PLANNING

Time of Information Use

Deterministic

MIT ICATMIT ICAT

Methodology

• Cognitive Analysis of Pilots Decisions

• Temporal Representation Framework Deterministic Regime

Stochastic Regime

MIT ICATMIT ICAT

Pilots’ Perception of Forecast Quality

Protected Aircraft HypertubeHazardous

WeatherHypervolume

Prediction of 4-D Intersection*

Y N

Occurrence of 4-D Intersection*

Y Hit Miss

NFalse Alarm

Correct Rejection

*Between Protected Aircraft Hypertube and Hazardous

Weather Hypervolume

CONTINGENCY MATRIX

MIT ICATMIT ICAT

Traditional Forecast VerificationMethodology for Hours

ForecastArea

ActualAdverseWeather

Miss

Hit

False Alarm

Correct Rejection

Scores Based on ContingenciesCritical Success Index 73%Signal Detection Theory Hit Rate 89%Signal Detection Theory False Alarm Rate 17%Mean Square Error 14%

MIT ICATMIT ICAT

Sensitivity of 4-D Intersection Test to Timing (e.g., ETD)

Forecast

Estimated Departure Time (Hrs)

4-D Intersectionbetween Aircraft Hypertubeand Weather Hypervolume

No

Yes

0 4

CorrectDetection (A)

CorrectRejectionFalse

Alarm (A)

Forecast Area

Aircraft A

Aircraft B

CorrectDetection

(B)

FalseAlarm (B)

2

MIT ICATMIT ICAT

Trajectory-Based Weather Forecasting

• Opportunity Route Availability Planning Tool (RAPT) with MIT Lincoln Lab

Uses convective weather forecast + airline schedulePredicts clear/blocked/impacted status of departure routes

Essentially 4-D intersection test

Key QuestionsWhat interface to provide to ATC users?How to manage/communicate uncertainty?

Weather growth/decayAircraft trajectoryEncounterTiming

• Analytical approach Understand and model uncertainty in situation dynamics Characterize the geometry/kinematics of encounter Identify

Implications for available decisionsOpportunities for support the identification of options

Provide recommendations for RAPT team & other tool development efforts

MIT ICATMIT ICAT

Summary of Analysis

• Relative Aircraft Track Angle Upstream: Aircraft trajectory toward boundary line Downstream: Aircraft trajectory away from boundary line

• Aircraft-Weather Encounter Situation Approaching weather boundary Receding weather boundary

• Error (Trajectory-Based) False alarms (FA)

Adverse weather is forecast to impact aircraft trajectory but does not Missed detections (MD)

Adverse weather is not forecast not to impact aircraft trajectory but does

• 8 Scenarios Potential causes Characterization Implications Adjustment options

Relative AircraftTrack Angle

Errors

Aircraft-WeatherEncounter Situation

Upstreamaircraft Downstream

aircraft

Approachingweather

Recedingweather

MIT ICATMIT ICAT

ExampleWorse Case

• Missed detection of approaching weather boundary for upstream aircraft trajectory

• Potential causes Front velocity underestimation Initiation/growth underestimation Aircraft velocity overestimation

• Characterization Transition to reduced capacity along route Associated with too late scenario Aircraft launch decision made in stochastic regime of weather representation

• Implications - Strategic Aircraft already in flight may have to divert Aircraft not already launched may have to delay departure for a long time

• Adjustment options - Tactical ATC: Identify aircraft trajectories around or above front Pilots: Identify gaps in front line

• Adjustment support ATC: Pre-negotiation between controllers for deviating aircraft flows Pilots: Support weather information update

MIT ICATMIT ICAT

Conclusions

• Weather decision-support requires addressing users’ time-dependent information needs Planning tasks Weather representation

• Under deterministic regime, trajectory-based weather forecasting has the potential to help support key decisions Matches users’ perspective and information needs for dynamics assessment

PilotsATC

Aircraft-weather encounter analysis points to key risk adjustment strategies

MIT ICATMIT ICAT

MIT ICATMIT ICAT

MIT ICATMIT ICAT

Backup

MIT ICATMIT ICAT

MotivationPilots’ information needs change over time

Decision-Maker

SituationDynamics

Information

Tt

Time of

InformationApplicability

Time of SituationDynamics

Time of Planning

Weather is chaotic

Forecast accuracy decreases with forecast horizonInformation is not sampled and provided continuously

Pilots’ functionsvary over time

Human Decision-Maker Controlled System

Environment

Information

Information

tT2

1

2

T1

Sensors

Sensors

Displays

Displays

Actuators

MIT ICATMIT ICAT

Uncertainty Growth with Forecast Horizon

http://www.msc-smc.ec.gc.ca/cmc/verification/performance/index_e.html

RMS Error inGeopotential Height

Error

ForecastTime

Stochastic

Deterministic

Constant

E(t)

Weather representation

based on observation over

a time period where conditions

do not significantly

change

Weather representation at

time in future beyond

“predictability limit”

Weather representation

based on deterministic forecast of

“acceptable” accuracy

MIT ICATMIT ICAT

Predictability Horizon Linked to Weather Phenomenon Lifetime

.01

.1

1

10

100

1,000

10,000

1 sec. 1 min. 1 hour 1 day 1 week

GustBuilding Wake

Dust Devil

ThermalCAT

Shearing Gravity WaveCumulusDownburst

RotorLee WaveCumulonimbus

Severe Thunderstorm

Gust Front

Land/Sea BreezeChinookSquall Line

HurricaneFront

CycloneJet Stream

MonsoonPlanetary Circulation

Reference: P. F. Lester, Turbulence, A New Perspective for Pilots

Ho

rizo

nta

l D

imen

sio

n (

n.m

.)

Lifetime

Mic

rosc

ale

Mes

osca

leM

acro

sca

le

MIT ICATMIT ICAT

Matrix of Temporal Regimes of Cognitive Processes

Temporal Regimes of Flight Planning

StrategicTacticalReactive

tt0

t0

Te

mp

ora

l Re

gim

es

of

We

ath

er

Re

pre

se

nta

tio

n

Historical

Constant

Deterministic

Stochastic

0

t0: time of information use

0: time of information production

NOW

re-routing around convective weather

MIT ICATMIT ICAT

Angle between weather front line and aircraft track

VF

VF Front velocity perpendicular to the front line

Aircraft trajectory segment

Scenario Analysis Model

VA

VA Aircraft velocityStorm line

MIT ICATMIT ICAT

Trajectory-Based Forecast Errors

Front Departure

Error

False Alarm Fly Faster

More options

Take-off ASAP

More options

Missed Detection

Hold in air

More options shortly

Hold on ground

More options shortly

Front ArrivalError

False Alarm

Take-off ASAP

No other options shortly

Fly faster

No other options shortly

Missed Detection

Hold on ground

No other options

Divert

No other options

Away from Front

Toward Front

N E S W

Away from Front Toward Front

Front ArrivalErrors Affect Origin Airport

Front Departure Errors Affect Destination

Airport

RelativeAircraft

Track Angle

Front Departure Errors Affect Origin Airport

Front Arrival Errors Affect

DepartureAirport

AircraftDeparture

Time

12

34

5 6

7 8

StochasticRegime

Departure ofFront from

Critical Point

Arrival of Front at

Critical Point

MIT ICATMIT ICAT

Trajectory-Based Forecast ErrorsTypes and Implications

N E S W

N

E

S

W

Toward Front Away from Front

Front Departure

Error

False Alarm

Take-off ASAP

More optionsFly Faster

More options

Missed Detection

Hold on ground

More options shortly

Hold in air

More options shortly

Front ArrivalError

False Alarm Fly faster

No other optionsshortly

Take-off ASAP

No other options shortly

Missed Detection

Divert

No other options

Hold on ground

No other options

Toward Front Away from Front

Front ArrivalErrors Affect Destination

Airport

Front Departure Errors Affect Origin Airport

AircraftTrack Angle

Front Departure Errors Affect

Destination Airport

Front Arrival Errors Affect

OriginAirport

AircraftDeparture

Time

Accurate forecast has more value for longer impact time of adverse weather on aircraft route!

StochasticRegime

Arrival of Frontat Critical Point

Departure ofFront from

Critical Point

MIT ICATMIT ICAT

Predictability Horizon Linked to Weather Phenomenon Lifetime

.01

.1

1

10

100

1,000

10,000

1 sec. 1 min. 1 hour 1 day 1 week

GustBuilding Wake

Dust Devil

ThermalCAT

Shearing Gravity WaveCumulusDownburst

RotorLee WaveCumulonimbus

Severe Thunderstorm

Gust Front

Land/Sea BreezeChinookSquall Line

HurricaneFront

CycloneJet Stream

MonsoonPlanetary Circulation

Reference: P. F. Lester, Turbulence, A New Perspective for Pilots

Ho

rizo

nta

l D

imen

sio

n (

n.m

.)

Lifetime

Mic

rosc

ale

Mes

osca

leM

acro

sca

le

Multi-Cell

Convective Line

MIT ICATMIT ICAT

Key Results

MIT ICATMIT ICAT

Area-based forecast assessment (top) vs.Trajectory-based forecast assessment (bottom)

MD

CD

FA

CR

4- DTrajectory

Missed Detection (MD)

Correct Detection (CD) False Alarm (FA)

Correct Rejection (CR)

Observation

Forecast

MIT ICATMIT ICAT

Aircraft Trajectory Assessmentfor ATC Applications

• Aircraft Tracks inside Providence Sector

• 24 Hours – 1 mile grid

Courtesy of Jonathan Histon

MIT ICATMIT ICAT

Weather Hypervolume Assessment

MIT ICATMIT ICAT

Metrics of Interest

MIT ICATMIT ICAT

Along Weather-Track Errors

N

E

S

W

Correct Detection False AlarmMissed Detection

MIT ICATMIT ICAT

Along Weather-Track Errors

Correct Detection False AlarmMissed Detection

MIT ICATMIT ICAT

Across Weather-Track Errors

N

E

S

W

Correct Detection False AlarmMissed Detection

MIT ICATMIT ICAT

Across Weather-Track Errors

Correct Detection False AlarmMissed Detection

MIT ICATMIT ICAT

Sensitivity of Trajectory-Based to Area-Based Performance Assessment

Dep

art

ure

Tim

e (H

ou

rs)

Dep

art

ure

Tim

e (H

ou

rs)

Trajectory Angle (Degrees from North) Trajectory Angle (Degrees from North)

Along Weather-Track Errors Across Weather-Track Errors

Tra

jec

tory

-Bas

ed

% P

OD

/FA

R

Tra

jec

tory

-Bas

ed

% P

OD

/FA

R

20% Area-Based

FAR and POD

20% Area-Based

FAR and POD

MIT ICATMIT ICAT

Sensitivity AnalysisSummary of Results

• Trajectory-based MD and FA correspond to 4-D intersections between 4-D aircraft hypertubes and FA/ MD hypervolumes

Except when there are 4-D intersections between aircraft and CD hypervolumes, since they correspond to CD

•

How should one measure ratios of POD or FAR? For fixed 4-D trajectories over departure time intervals?

What time intervals are relevant?

For 2 simple cases investigated, the trajectory-based scoring will either look equal or better than the area-based statistics over the full time interval

Many exceptions

N

E

S

W

N

E

S

W

MIT ICATMIT ICAT

Sensitivity AnalysisSummary of Results

• Trajectory-based errors depend on the geometry/kinematics Along weather-track errors are observed

At departure times when the weather is overhead the critical point FAR is worse for certain time intervals

E.g., for trajectories with relative velocity (Va-Vw) perpendicular to the weather

track Linear relationship to timing error

Across weather-track errors are observed For trajectories that intersect the FA and MD hypervolumes

FAR and POD Highly dependent on the FA and MD hypervolume trajectories w.r.t. critical point

N

E

S

W

N

E

S

W

MIT ICATMIT ICAT

Angle between weather front line and aircraft track

VF

VF Front velocity perpendicular to the front line

Aircraft trajectory segment

Scenario Analysis Model

VA

VA Aircraft velocity

MIT ICATMIT ICAT

Correct DetectionForecast Error

Angle between weather front line and aircraft track

VF

VF Front velocity perpendicular to the front line

Aircraft trajectory segment

Scenario Analysis

VA

VA Aircraft velocity

MIT ICATMIT ICAT

Summary & Implications

• Types of errors Front arrival

Marks transition to “blocked trajectory” state Front departure

Marks transition to “clear trajectory” state False Alarms

Missed opportunity if tactical response not supported Missed detections

Can be very disruptive When related to front arrival errors, mean long ground hold or even diversion.

• Error type comparison Front arrival errors worse than front departure errors Missed detection worse than false alarm Worse when affecting destination airport

Stochastic regime Affected by:

Front arrival errors when flying toward front Front departure errors when flying away from front

• Value of finding “pores”/gaps: Greatest for front arrival forecasts Greatest for destination airports of aircraft

MIT ICATMIT ICAT

Opportunity for Applications

MIT ICATMIT ICAT

Methodology

• Cognitive Analysis of Pilots Decisions Feedback control loop Decision-making models Task analysis

• Temporal Representation Framework Deterministic Regime

Stochastic RegimeBeyond the “deterministic predictability limit” Importance of “options”How to support “options” identification in planning?

MIT ICATMIT ICAT

MIT ICATMIT ICAT

Uneventful Flight

Hazardous Encounter with Wx

Loss of Flight Opportunity

Chance of Uneventful Flight

Chance of Hazardous Encounter with Wx

Go

No-Go

Gain Outcome

Loss Outcome

Sure Outcome

Chance of Gain

Chance of Loss

Risk-Tolerant Action

Sure Action

The “gain” option identification is important in potentially hazardous weather

If pilots frame the “no-go” decision as a loss, they will opt for the risk-tolerant “go” decision

MIT ICATMIT ICAT

Uneventful Flight

Hazardous Wx Encounter

Loss of Flight Opportunity

Chance that hazardous wx conditions are not

present along the planned route

Chance that adverse wx conditions are present along the

planned route

Go

No-Go

Abort

Chance that safe options are

not available

Chance that safe options are

available

Successful Avoidance

Unsuccessful Avoidance

Gain Outcome

Loss Outcome

Sure Outcome

Chance of Gain

Chance of Loss

Risk-Tolerant Action

Sure Action

The “gain” option identification is important in potentially hazardous weather

MIT ICATMIT ICAT

Implications

• The “options” identification is a key element of risk assessment

• Supporting the identification of “options” explicitely may add value to the decision-support information

MIT ICATMIT ICAT

Conclusions

• How to improve time-varying decision-support Improve the “deterministic predictability limit” Understand how the deterministic regime supports tactical vs. strategic

planning Understand how to support decision-making under the deterministic and

stochastic regimes

• Under deterministic representation Decision-makers wish to have information that supports their assessment of

the situation dynamics

• Under the stochastic representation Decision-makers wish to have information that supports their assessment of

the availability of options

MIT ICATMIT ICAT

Decision-Maker

SituationDynamics

Information

Tt

Time of

InformationApplicability

Time of SituationDynamics

Time of Planning

MIT ICATMIT ICAT

Back Up Slides