This may be the author’s version of a work that was submitted/acceptedfor publication in the following source:

Karanikas, Nektarios & Nederend, Jeffrey(2017)Controllability of events and associated factors: Lessons learned from avi-ation. InCentre for Transport Studies Seminar, 2017-02-22 - 2017-02-22. (Unpub-lished)

This file was downloaded from: https://eprints.qut.edu.au/130955/

c© 2017 the Author(s)

This work is covered by copyright. Unless the document is being made available under aCreative Commons Licence, you must assume that re-use is limited to personal use andthat permission from the copyright owner must be obtained for all other uses. If the docu-ment is available under a Creative Commons License (or other specified license) then referto the Licence for details of permitted re-use. It is a condition of access that users recog-nise and abide by the legal requirements associated with these rights. If you believe thatthis work infringes copyright please provide details by email to qut.copyright@qut.edu.au

Notice: Please note that this document may not be the Version of Record(i.e. published version) of the work. Author manuscript versions (as Sub-mitted for peer review or as Accepted for publication after peer review) canbe identified by an absence of publisher branding and/or typeset appear-ance. If there is any doubt, please refer to the published source.

1

CONTROLLABILITY OF EVENTS AND ASSOCIATED FACTORS:LESSONS LEARNED FROM AVIATION

Dr. Nektarios KaranikasJeffrey Nederend

Aviation Academy

Centre for Transport Studies Seminar, 22 February 2017, Imperial College, London

RESEARCH BACKGROUND

BACKGROUND

• Safety performance:• Safety performance in the industry is measured based on the

severity of events (incidents, serious incidents and accidents) through respective indicators (e.g., number of accidents / unit of exposure).

• Aviation safety performance has been continuously increased over the last half-century.

• Authorities and organizations try to improve safety levels through:• Proactive methods (i.e. before an unwanted event occurs), such

as audits, analysis of hazard reports etc.• Reactive methods (i.e. after an unwanted event happens)

activities, mainly safety investigations.3

BACKGROUND• Results from safety investigations target to:

• Reveal causes of safety events• Make recommendations about prevention of future safety events

• According to ICAO (2010) and EC (2010), safety investigations are mandatory for accidents and serious incidents.

• Although safety incidents can comprise valuable sources for improving safety levels further, limited resources have led:• Authorities to not investigate all incidents. • Some authorities monitor trends over time and decide to investigate

causes for a set of incidents.• Organizations to investigate a limited number of incidents without

any criteria for the priority of events that should be investigated. Trends over time can be also consulted.

4

BACKGROUND• The emphasis on actual event severity does not provide

information about the extent to which outcomes were a matter of control.

• Therefore we do not:• Consider the potential of an event to escalate to a category of

higher severity• Obtain a picture of the degree to which our systems are under

control• The focus on actual outcomes might lead to stronger

biases: the higher the severity, the higher the importance of the causal factors revealed from the investigation.

5

BACKGROUND• Contemporary safety theory and

research suggest that:• Safety events do not occur merely due

to unreliable system components (i.e. individual causes revealed from investigations).

• Complexity of modern socio-technical systems necessitates also the consideration of interlinks and dependencies.

• Efforts must be first directed to the control of systems even when behaviour of components is as expected.

6Leveson, N. (2011), Engineering a Safer World. MIT Press, Cambridge, USA

BACKGROUND

7

EVENT CONTROL CLASSIFICATION USER REACTION CLASSIFICATION

CONTROLLED: The user attempted to control the event

march.

POSITIVE: User’s actions did not worsen the outcome; the

event outcome was managed successfully; no errors or

violations were noticed during the control attempt.

NEGATIVE: User’s actions following the safety event initiation

resulted in adverse outcomes due to human errors or

violations.

UNCONTROLLED: Safety event’s consequences were

developed without control; there had been no intervention

until the time the outcomes were noticed.

NONE

NEUTRAL: Inevitable application of normal procedures;

standard reactions to identified problem.

AS EXPECTED BY PRESCRIBED PROCEDURES.

Karanikas, N. (2015), An Introduction of Accidents’ Classification Based on their Outcome Control, Safety Science, 72, pp. 182‐189.  DOI: 10.1016/j.ssci.2014.09.006

RESEARCH OBJECTIVES AND QUESTIONS

RESEARCH OBJECTIVES

• Examine whether the current classification of events based on their controllability needs improvement.

• Explore how much aviation safety events have been controlled.

• Assess the association between severity & controllability and the introduction of the latter as a metric of safety performance prior to the consideration of actual severities.

• Identify factors/variables that might be linked to controllability of events.

9

RESEARCH OBJECTIVES

• Provide information to the aviation industry for current levels of controllability of safety events and associated factors.

• Recommend how controllability, regardless industry sector, might be used:• As criterion for prioritizing investigation of safety events• Reactively and/or proactively in safety management• As metric for safety performance

10

RESEARCH QUESTIONS

• Is the current classification of safety events based on the degree of their controllability sufficiently reliable?

• To what extent have aviation safety events been controlled?

• What factors are associated with the controllability of safety events in aviation?

11

RESEARCH METHODOLOGY

IMPROVEMENT OF CLASSIFICATIONRecruitment of 4 graduate students (Aviation Studies)

Familiarization with published 

classification

Application of classification to 4 

reports

Calculation of inter‐rater agreement

(ICC < 0.75)

Refinement of definitions based on comments from 

analysts

Application of refined classification 

to 20 reports

Calculation of inter‐rater agreement

(ICC > 0.75)

Further adjustment of classification 

based on comments 13

APPLICATION OF CLASSIFICATION

• Sample of 297 safety investigation reports published by:• Air Accident Investigation Board of the United Kingdom (AAIB)• Australian Transport Safety Bureau (ATSB)• Dutch Safety Board (DSB)• National Transport Safety Board of the United States (NTSB)• Transport Safety Board of Canada (TSB)

• Reports randomly selected based on:• Availability online• Publication in English language• Sample used in concurrent studies performed in the Aviation

Academy• Time constraints

• Time span of safety events: 1990 - 201414

INDEPENDENT VARIABLES & STATISTICS

15

Groups of Independent Variables

Dependent VariablesControllability Reaction level Outcome

Origin of aircraftregistration

X X X

Temporal and seasonalfactors

X X X

Aircraft characteristics X X XOperationalcharacteristics

X X X

Event characteristics X X XCrew/ground stafffatigue

X

Statistics:• Chi-square or Fisher’s Exact tests of independency (depending on sample

distribution across variable cases).• Significance level: 0,05

RESULTS & DISCUSSION

REFINED CLASSIFICATION

17

CONTROLLABILITY LEVEL OF USER’S REACTION OUTCOME FROM REACTIONSCONTROLLED: The user attempted tocontrol the accident march.

SKILL – RULE BASED (SRB): Reaction toa single event and standard applicationof procedures described in check‐listsand manuals.

POSITIVE: User’s actions did notworsen the outcome or theaccident outcome was managedsuccessfully. No errors orviolations were noticed duringthe control attempt in referenceto established procedures and/ortraining provided.

NEGATIVE: User’s actionsfollowing the safety eventinitiation resulted in adverseoutcomes due to human errors orviolations in reference toestablished procedures and/ortraining provided.

RULE – KNOWLEDGE BASED (RKB):Reaction to complex situations bycombining existing rules, partially/notdescribed in check‐lists and manuals,and generally acquired knowledge.

UNCONTROLLED: Safety event’sconsequences were developed withoutcontrol; there had been no plannedintervention with the scope to preventthe event until the time the outcomeswere noticed.

Not applicable Not applicable

EXAMPLES OF APPLICATION

18

CASE REACTION CONTOLLABILITY OUTCOMEACTION CATEGORY

One tyre of the mainlanding gear systemblew out duringtaxiing.

The pilot stoppedthe aircraft

SRB Controlled Positive

Engine flame outduring landingapproach underadverse weatherconditions.

The crew landed theaircraft safely

RKB Controlled Positive

Aircraft malfunctionleads to an unstablestate while in finalapproach.

Incorrect techniquesresulted to a runwayexcursion

RKB Controlled Negative

Important impactson the enginecompressor bladesdue to ForeignObject Damagesobserved duringAfter FlightInspection.

Not applicable Not applicable Uncontrolled Not applicable

DEGREE OF CONTROLLABILITY

19

75,70%

24,30%

Safety events

Controlled Uncontrolled

USER REACTIONS & OUTCOMES (CONTROLLED EVENTS)

20

53,80%

24,60%

21,60%

0,00%

10,00%

20,00%

30,00%

40,00%

50,00%

60,00%

70,00%

80,00%

Rule‐Knowledge Based Skill‐Rule Based

Positive Outcome Negative Outcome

CONTROLLABILITY & SEVERITY

21

70%80,30%

93,80%

30%19,70%

6,20%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Accidents Serious Incidents Incidents

Controlled Uncontrolled

REACTION & SEVERITY(CONTROLLED EVENTS)

22

52,10%37,90%

13,80%

47,90%62,10%

86,20%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Incidents Serious Incidents Accidents

Skill‐Rule Based Rule‐Knowledge Based

OUTCOMES & SEVERITY(CONTROLLED EVENTS)

93,30% 90,60%

57%

6,70% 9,40%

43%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Incidents Serious Incidents Accidents

Positive Outcome Negative Outcome 23

CONTROLLABILITY AND OUTCOMES PER COUNTRY OF REGISTRATION

24

92%

80% 78% 77%

68%64%

91%

61%

45%

88%95%

74%

AUSTRALIA OTHER USA UK NETHERLANDS CANADA

CONTROLLABLE POSITIVE OUTCOME

OTHER INDICATIVE STATISTICALLY SIGNIFICANT ASSOCIATIONS• Aircraft characteristics: newest (i.e. ≤ 6 years old), jet, and

heavy (i.e. ≥ 27tons weight) aircraft were involved in more controlled events which required a higher frequency of Skill-Rule Based (SRB) actions.

• Flight characteristics: • Commercial Air Transport and passenger operations were associated

with more controlled and SRB events.• En-route events were linked with more RKB reactions compared to

events during “other” flight phases (i.e. take-off, climb, approach and landing) and on ground.

• “Other” flight phases were associated with more negative outcomes.• Fatigue was associated with increased negative outcomes.

25

CONCLUSIONS & RECOMMENDATIONS

CONCLUSIONS (1/2)

• The application of controllability classification:• Confirmed literature suggesting the relationship system

control and unwanted outcomes.• Indicated that when end-users had the opportunity to

control a system during an unfolding adverse situation, in about 80% of the cases they successfully applied procedures and used their experience to mitigate the outcomes.

• Identified that safety incidents that were investigated as such could have escalated to serious incidents or accidents.

27

CONCLUSIONS (2/2)

• The application of controllability classification:• Showed that events of higher severity were linked to

cases with operational complexity (i.e. more Rule –Knowledge reactions).

• Revealed that the distribution of the characteristics of the controllability classification varies across regions, event, flight and aircraft characteristics.

• Indicated the effect of fatigue on human performance (i.e. more negatively controlled events).

28

RECOMMENDATIONS & REMARKS• Suggested classification of event controllability can be:

• Applied retrospectively with its current binary form in order to:• Analyse safety events, generate insights about the level and

effectiveness of control over a system and examine associated factors.

• Under limited resources, steer safety management efforts towards increasing controllability and positive outcomes of events.

• Used proactively as part of hazard reporting systems:• End-users can state the perceived degree of controllability and

level of reaction with the introduction of measurement scales.• Analysts can process respective data, monitor trends and

assess the need for further research and interventions. 29

RECOMMENDATIONS & REMARKS• Suggested classification of event controllability can be:

• Adapted as criterion for performing incident investigations under limited resources: priority to uncontrolled events.

• Added as metric of safety performance and effectiveness of safety management.

• In regard to the application of the proposed classification:• Users must become familiar with the definitions. Common

understanding might be assessed through inter-rater agreement tests.

• The reaction level is subject to interpretation of the operational complexity and its assignment depends on the context.

• The assignment of negative outcomes in relation to procedures established and training provided does not intend to blame the end-users but to indicate mismatches. 30

31

CONTROLLABILITY OF EVENTS AND ASSOCIATED FACTORS:LESSONS LEARNED FROM AVIATION

Dr. Nektarios KaranikasJeffrey Nederend

Aviation Academy

Contact: n.karanikas@hva.nl