A Hybrid Technique for Organizational Safety Risk Analysis · A Hybrid Technique for Organizational Safety Risk Analysis ... Hybrid Technique for Organizational Safety Risk BB N FT

Center for Risk and Reliability

A Hybrid Technique for Organizational

Safety Risk Analysis

Reliability Eng. ProgramMechanical Eng. Department

Presented at

International Conference on

Probabilistic Safety Assessment and Management (PSAM 9)

May18-23 2008, Hong Kong, China

Zahra Mohaghegh

Ali Mosleh


University of Maryland


Motivation

• Major system failures with significant

contributions from human & organizational

factors

– Chernobyl

– Columbia and Challenger space shuttle accidents

• In the quest to achieve 80% reduction in aviation

accidents, US FAA has recognized

“organizational factors” as one of the most

critical components


Key Questions

• What are the organizational “factors” that

affect system risk?

• How?

• To what degree?


The Research Approach

5. Key Gap… Leading to

New Research

1. Classical PRA

2. Approaches to Include

Organizational Factors

In PRA

3. Organizational

Accident Causation Theories

4. Quantitative

Organizational Safety Risk

Frameworks

6. Principles of Org. Safety Risk Analysis 7. Proposed Hybrid Technique

for

Org. Safety Risk Analysis

8. Realization Of Modeling Principles in SoTeRiA

9. An Example Application of SoTeRiA in Aviation

10. An Example Application of Hybrid technique In

Aviation


Quantitative Approaches

• Static : Variations of Influence Diagrams

(e. g., BBN), Process Models and Logic

Models – MACHINE (Embrey, 1992)

– SAM (Pate-Cornell, 1996)

– Omega Factor Model (Mosleh & Golfeiz, 1999),

– ASRM (Luxhoj, 2004)

– “Causal Modeling of Air Safety” (Roelen et al., 2003)

• Dynamic: e.g, use of “System Dynamics”– Mousang (2004)


Key Gap

• In the absence of a comprehensive theory, or at least a set of principles rooted in theory, all models look equally good, or equally poor, with very little basis to discriminate, and build confidence.

• This research focused on improving the theoretical understanding of relation between characteristics of organizations and their (system) safety outputs


Modeling Principles

Problem Definition, Metrics &

Scope

(A ) Defining the Unknown of Interest

(B ) Safety & other Org. Performance

(C ) Safety Performance & Deviation

Level of Analysis (D ) Multilevel framing

Factors / Elements (E ) Basic building block

(F ) Factor Level

(G) Factor Selection

(H) Measurement methods

(I ) Role of Perception

(J ) Factor Interdependencies

(K) Multidimensional measurement perspective

Relations • Multidimensional links

• Dynamic characteristics

Boundaries & Assumptions (N) Depth of causality and level of detail

(O) Generalizablity

Characteristics (P) Hybrid


SoTeRiA (Socio-Technical Risk Analysis )

Industrial &

Business

Environment

Social and Political

Culture and Climate

organizational

vision, strategy

and

goals

Regulatory

Environment

Safety

Culture

Organizational

Culture

Organizational

Structure&

Practices

Org. Safety

Structure &

Practices

Group

Climate

Group

Safety

Climate

SCT

Business

Outcome

System

Risk

Unit

Process

Model

Organizational

Climate

Org. Safety

Climate

Psyc.

Safety

Climate

Individual

PSFs

Emergent Process

(Leadership/supervision,

social interaction & homogeneity )

Physical Environmental Factors (e.g., Harsh Weather Condition)

Industrial &

Business

Environment

Social and Political

Culture and Climate

organizational

vision, strategy

and

goals

Regulatory

Environment

Safety

Culture

Organizational

Culture

Organizational

Structure&

Practices

Org. Safety

Structure &

Practices

Organizational

Structure&

Practices

Org. Safety

Structure &

Practices

Group

Climate

Group

Safety

Climate

Group

Climate

Group

Safety

Climate

SCT

Business

Outcome

System

Risk

Unit

Process

Model

Organizational

Climate

Org. Safety

Climate

Organizational

Climate

Org. Safety

Climate

Psyc.

Safety

Climate

Individual

PSFs

Emergent Process

(Leadership/supervision,

social interaction & homogeneity )

Physical Environmental Factors (e.g., Harsh Weather Condition)


Menu of Modeling Techniques

AND AND

0 1

1

0 1

+

3

20 1

Cut-Sets

Probabilityof Sequence

ESD

FTAND AND

0 1

1

0 1

+

3

20 1

Cut-Sets

Probabilityof Sequence

ESD

FT

Techniques for Technical SystemTechniques for Technical System(e.g., Groen et al., 2002)(e.g., Groen et al., 2002)

A1

A2

B2

B1

C1 C2 D1 D2A

B

C D

Latent Variable Measured variable

RegressionRegression--based Causal Modelingbased Causal Modeling(e.g., (e.g., BollenBollen, 1989), 1989)

Activity

Transformation Inputs

Controls/ Criteria

Resources

OutputsActivity

Transformation Inputs

Controls/ Criteria

Resources

Outputs

Process Modeling TechniquesProcess Modeling Techniques(e.g., (e.g., HeinsHeins, 1993, 1993 )

A

B1

X

B2

Y

B3

Z

Bayesian Belief Network( BBN)Bayesian Belief Network( BBN)


Menu of Modeling Techniques

A1

A2

A4

A31

A

C1

C2

C4

C3

C

A1

A2

A4

A31

A

C1

C2

C4

C3

C

Configurational approach(e.g. Meyer et al., 1993)

Quantitative

Qualitative

A

B

C

D

E

Quantitative

Qualitative

A

B

C

D

E

QQ-BBN(Wang, 2007)

Population

BirthDeath

Birth Rate Death Rate

System Dynamics(e.g. Sterman, 2000)

Human Reliability Techniques

(e.g., Chang & Mosleh , 2004)

•THERP

•IDAC

•NARA


Hybrid Technique for Organizational Safety Risk

SD

Environment

QQ-BBN

ESD/FT

MethodStock and flow

Diagram

HCL(Mosleh

et al. , 2005)

technical systems

•process modeling

•regression-based

models

•configurational

characteristics

•human model

deterministic relations

dynamic integration


Hybrid Technique for Organizational Safety Risk

BB

N

FT

ESD

SD

SMPs

Human

Action

Human

Action

TeamPerformance(e.g. Maintenanceteamperformance)

Team Perfo

rmance

Model

Failure due toother causes

System 1

Human

Action

System 1

OmegaFactor

Resources

ResourcesResources

Initiatingevent

S

S

F

Failure due to

organizational factors

(e.g. maintenance error)

System Dynamics Model of

Org.


Financial

performanceTechnical System Risk Model

Quality of Maintenance

Maintenance Unit Process Model

Regulatory Auditing System

Individual-level PSFs

Weather

Condition

Emergent

Process

Maintenance Group Safety Climate

Organizational

Safety Structure & Practices

Organizational Safety Culture

A

B

C

D

EF

G

H

I

J

K

L

M

SoTeRiA –based Aviation Maintenance Model


Technical System Risk

Engine

Flight crew action

S

S

F

Initiating

Event

Engine Failure

Internal Engine

Failure

Engine failure

Maintenance

related

Engine

mismanagement

By crew

Engine failure by

External factors


MFinancial






Weather

Condition

Emergent

Process


Organizational



A

B

C

D

EF

G

H

I

J

K

L

Implementing Implementing SoTeRiASoTeRiA in Aviation Maintenancein Aviation Maintenance


Return to Service

Inspection

Maintenance Quality

Outsource MX In-house MXThrough Flights/ Daily

ChecksLayer (1)

Direct

Activities

4thParty

CTRS Actions

Technicians

Actions

CAMP

Proc.

Tools, Equipment

Parts/m

aterial

MX.

Facility/ Base

Inspector

Actions

4thParty

CTRS Actions

RISI

Procedures

Layer (1-1)

Outsourced/

In-house

Outsource

technicians

MX.

Schedule

ADS/ CMR/

STC/SBs

Manuals

Records &

Reporting

Inspection

Program

Repair

Proc.

Locally Produced

T/E

Purchased/

Produced

Purchased

Tools, Equipment

Operator produced

Parts

Purchased Parts/

Materials

Purchased/

Produced

Layer (1-1-1)

Layer (1-1-2) Eng.

Proc.

SFAR

Proc.

Maintenance Unit Process ModelMaintenance Unit Process Model

Adapted from Eghbali (2006 )


Financial






Weather

Condition

Emergent

Process


Organizational



A

B

C

D

EF

G

H

I

J

K

L

M



“Training” in System Dynamics Environment

Total technicions Experience

Increase in experience

f rom hiring

Loss of experience

f rom attrition

Increase on the job experience

Training

Saf ty experience gained

on job

Target experience

Experience gap

Av erage technicions experience

Average technicions experience

Total Number of Quits

Hiring

Ref erence experience

Relativ e ev perience

of technicions

Average experience

of technicions hired

Time to prov ide training

Relative technicion commitment

Total technicions

Relative management

commitment to safety

Rookie Quit

Senior Quit

T raining

Based in Part on Cooke (2004)


“Hiring Model” (System Dynamics Environment)

~

Ef f ect of Relativ e technicion

Commitment on quit

Senior quit f raction

Rookies SeniorsHiring

Up to speed Senior Quit

Rookie Promotion Time

~

Effect of Relative technicion

Commitment on quit

Rookie Quit

demand

Rookie quit f raction

Gap in Number of technicions

Target technicion Demand

Technician working

hours per y ear

Total technicions

Relative management


Hiring Time


Staf f ing

Based in Part on

Cooke (2004) & Cabe(1998)


Financial






Weather

Condition

Emergent

Process


Organizational



A

B

C

D

EF

G

H

I

J

K

L

M



Human Reliability Model

(in System Dynamics Environment)

Total technicions

EPC time pressure

EPC moral

EPC experience

GTT

Time Pressure

demand Time Available

Regulatory allowed working hours

technicion error Probability

~

APOA experience

~

APOA moral

~

APOA time pressure

Relative evperience

of technicions


Teh Error Probability

input to IRIS

Human Reliability

The module is built partially based on Nuclear Action Reliability Assessment (NARA) (Kirwan, et.al., 2004)


Technician Commitment(in System Dynamics Environment)

A modified version of Cooke’s (2004) model


Financial






Weather

Condition

Emergent

Process


Organizational



A

B

C

D

EF

G

H

I

J

K

L

M

SoTeRiASoTeRiA in Aviation Maintenancein Aviation Maintenance


Management Commitment Model(in System Dynamics Environment)

• The basic modeling follows Sterman 2000

• Describes the process of groping toward a proper quantity

• Assumes an initial level of commitment which later changes according to safety and financial pressures applied to organization

Management Commitment to safety

Change in Management

Commitment to Safety

refrence technicion error

Target management


Time to change manaegment


Pressure to change management


Effect of relative

tech error

on management commitment

to safet

Relative

technicion error Probabilitytechnicion error Probability

Safety priority

exponent

effect time

Effect of financial pressure on

management commitment to safety

Z Score

FP effect time

Financial priority

exponent

Management drive to prioritize

financial situation over safety

managemnet commitment

Partially referred to Cooke (2004)


Financial






Weather

Condition

Emergent

Process


Organizational



A

B

C

D

EF

G

H

I

J

K

L

M

SoTeRiASoTeRiA in Aviation Maintenancein Aviation Maintenance


Financial Stress Model

(in System Dynamics Environment)

• “Altman's score” model has been employed As a measure of financial distress

• “Z score” is a linear combination of some financial ratios available on afirm’s balance sheet

Relative

technicion error Probability

Relative Risk

Z Score

Increase rate

in ZDecrease rate

in Z

Z Increasing Multiplier

Z Decreasing Multiplier

Increase

Decrease

Financial Pressure Module


Integration of Software Tools

FT

SD

ESD

MX

BBN

SD

BBN

FT

ESD

Tech. error

probability

Aircraft

Airworthiness

Top event

probability

Risk

STELLA IRIS


Example Causal Loop

Management

Commitment to

Safety

Training

Attrition

Average

Technician

Experience

Technician

Error

Probability

+ +

+ -

+

+

+

-

+

+

+

+


Example Causal Loop:

Safety & Profitability

Management

Commitment

To Safety

Technician

Commitment

To Safety

Technician

Error

Probability

Financial

Pressure

-

+

-

-

+

+

-

+


Typical Output: Financial Stress as a Trigger Point

2:06 PM Fri, Jul 06, 2007Page 1

0.00 4.00 8.00 12.00 16.00 20.00

Years

1:

1:

1:

2:

2:

2:

3:

3:

3:

0

1

1

0

0

1

1: Management Commitment to safety 2: technicion error Probability 3: Z Score

11 1

1

1

1

2

2

2

2

2 2

3

3

3

3

3

3


Typical Output :A period of Low-error Stability

2:16 PM Fri, Jul 06, 2007Page 1

0.00 4.00 8.00 12.00 16.00 20.00

Years

1:

1:

1:

2:

2:

2:

3:

3:

3:

0

1

1

1: Management Commitment to safety 2: technicion error Probability 3: Personal tech Commitment to Safety

1

1

1

1

1

1

22

22

2

2

3

3

3

3

3

3


Typical Output: Total Risk Over 15 Years

1.005500000000000E-08

1.006000000000000E-08

1.006500000000000E-08

1.007000000000000E-08

1.007500000000000E-08

1.008000000000000E-08

1.008500000000000E-08

1.009000000000000E-08

1.009500000000000E-08

1.010000000000000E-08

1.010500000000000E-08

1.011000000000000E-08

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Time

Risk

Series1


Concluding Remarks

� Introduced a 4-layer hybrid dynamic framework for causal modelling of organizational safety risk

� Integration of deterministic (e.g. SD ) and probabilistic (e.g., BBN, ESD, and FT ) modelling methods

� Flexible risk-informed decision making tool with explicit consideration of

� Dynamic effects, such as time lags between decisions and outcomes, and feedbacks such as the impact of incidents on worker awareness and attention to safety

� The uncertain nature of the impact of organizational factors on human performance

� The impact of human performance on the systems and evolution of

risk scenarios


Acknowledgment

� The work described in this paper was in part supported by the US Federal Aviation Administration.


Example Causal Loops:

Management

Commitment to

Safety

Training

Attrition

Average

Technician

Experience

Technician

Error

Probability

- -

+ -

-

-

+

-

+

+

Documents

A Hybrid Technique for Organizational Safety Risk Analysis · A Hybrid Technique for Organizational Safety Risk Analysis ... Hybrid Technique for Organizational Safety Risk BB N FT