Human Reliability Assessment

Human Reliability Assessment

Grace Kennedy

[email protected]

16th/19th October 200606ELD061/06ELP461

Objectives for the Sessions

• Understand the Human Reliability Assessment process

• Gain practical experience of a simple task analysis

• Gain practical experience of error identification

• Gain practical experience of representation

• Gain practical experience of quantifying error probabilities in a simple example

Human Error Example 1

• A KC-135 Aircraft was being pressurised at ground level.

• The outflow valves were capped off during a 5 year overhaul and never re-opened.

• A civilian depot technician was using a home-made gauge, and no procedure.

• The technician's gauge didn't have a max "peg" for the needle which had gone round the gauge more than once.

• The result….

Human Error Example

Human Error Example 2

Helios Crash 2005

Extract taken from BBC News Site

http://news.bbc.co.uk/1/hi/world/europe/6036507.stm?ls

• Pilot misread instruments AND misinterpreted warning signals

• Maintenance left pressure control in wrong setting

• Manufacturer did not respond adequately to previous similar incidents

Predicting errors

• Task analysis and error identification

Preventing errors

• Specifying training requirements

• Equipment design (e.g. pressure gauge)

• Detailed procedures (administrative control)

Ultimately:

• Reduce risk

• Save money

• Justify design decisions

What can be done about it?

“For every $1 spent in the early stage,

approximately $10,000 are saved (if the problem

were to be fixed later instead).” – Manprint

Understanding Human Error

Process Techniques

Accident Analyses

Prediction

Risk Assessment

Psychology

Environment HCI Design

What is HRA?

HRA = Human Reliability Assessment

HRA Process Outline

• Task analysis is used to describe and understand the human interactions with the system

• The results of the task analysis are used with an error taxonomy (classification scheme) to allow error identification

• The identified errors are analysed either qualitatively or quantitatively

• The process is repeated each time a design iteration occurs

Human Reliability Assessment Process

General HRA Process – Kirwan, 1994


• Problem Definition

• Task Analysis– Describe what is done

– Improve analyst’s knowledge

• Error Identification– Taxonomy

– Failure criteria

• Representation– Fault tree/event tree

– Risk model

• Quantification– e.g. HEART

• Impact Assessment– Effect of errors

– Risk contribution

• Error Reduction– Re-design tasks

– Add engineered features

– Procedures / training

• Quality Assurance– Appropriate techniques

– Technical checking

• Documentation

Definitions of Terms

• Process: The overall HRA process

• Method: The steps in a process

– e.g. Task Analysis, Quantification, etc

• Technique: The specific implementation of a method(s)

– e.g. HEART, THERP, etc

• Tool: A software tool to record and guide the use of a technique

– e.g. Fault Tree +

HRA Techniques

• Many HRA techniques available

• Working to different levels of detail on different concepts

• From expert judgement techniques (e.g. APJ, PC)

• Hazard identification techniques (e.g. HAZOPS, THEA)

• To quantitative techniques (e.g. HEART, THERP)

• To second generation techniques (e.g. CREAM, ATHEANA)



TASK ANALYSIS

Task Analysis

• Range of techniques to understand what humans are

required to do in order to achieve a system goal

– Collect and organise information

– Improve the analyst’s understanding

– Structured approach

– Support to design and assessment

A Guide to Task Analysis,

Barry Kirwan & Les Ainsworth (1992),

Taylor and Francis

ISBN 07484-0058-3

Hierarchical Task Analysis

• Expresses a job or function in terms of goals,

operations and plans

– Goals Objectives to be achieved

– Operations Actions required to achieve the goals

– Plans Conditions under which the actions are

carried out

Hierarchical Task Analysis Example

• Express the task of making a cup of tea using HTA

– Goals Objectives to be achieved (e.g. Make Tea)

– Operations Actions required to achieve the goals (e.g.

Boil water, Add milk / sugar)


carried out (e.g. boil the water before

adding it to the cup)

Example provided using TaskArchitect software

Making Tea - one solution (1 of 5)

Bar beneath the activity shows no further development

Stub beneath the activity shows further development has taken place

Plan describes the logic





Hierarchical Task Analysis - Practical

• Express the task of fitting an electric plug using HTA

– Goals Objectives to be achieved (e.g. Fit plug)

– Operations Actions required to achieve the goals (e.g.

Strip outer casing, Twist exposed

wires)


carried out (e.g. fit the fuse before

closing up the plug to the

cup)

Hierarchical Task Analysis - Guidance

• State an overall goal (box at the top)

• Breakdown each goal or sub-goal one at a time (i.e.

finish one box before moving on)

• Ensure all the actions under a goal are relevant and

would actually achieve the stated goal

• Keep the order and logic in the plans (make the plans

specific to the goal)

Work in Pairs - 10 minutesA solution on pink sheet

Wiring a plug – HTA one solution (1 of 4)






ERROR IDENTIFICATION

Error Identification - General

• Task Analysis describes the activities necessary to achieve a goal

• An Error Taxonomy (classification scheme) can be used to identify specific errors

• Many errors will be possible, so need to understand– Error effects (relating to the task goal)

– Failure criteria (goal failure)

• Produce a list of identified errors, which lead to goal failure

• Organise the information in a Tabular Task Analysis

Error Identification - Tabular Task Analysis

• Use the information from the HTA

• Create a Tabular Task Analysis (TTA)

• Error taxonomy (classification scheme) to identify errors

• Understand

– Error effects

– Failure criteria

• List of identified errors

Tabular Listing from HTAID Task Plan0 Wire an electric plug do in sequence 1-50.1 Collect tools0.2 Unscrew plug cover0.3 Prepare lead do in sequence 1-60.3.1 Estimate length of stripped wire required to reach earth terminal0.3.2 Strip outer casing according to estimate

0.3.3Check yellow/green wire reaches earth terminal whilst outer casing exceeds holder by 5 mm

0.3.4 Cut blue and brown wires to reach their terminals0.3.5 Strip each of the coloured leads to leave exposed wire0.3.6 Twist exposed wires on each coloured lead

0.4 Ensure correct fuse is in place

do in sequence 1-3; If mismatch between required and in-situ fuse then do ( 4)

0.4.1 Locate appropriate instructions for equipment0.4.2 Read fuse requirement0.4.3 Compare fuse requirement with given fuse0.4.4 Change the fuse do in sequence 1-30.4.4.1 Select the correct fuse0.4.4.2 Extract fuse from plug0.4.4.3 Insert correct fuse0.5 Attach plug to lead do in sequence 1-40.5.1 Thread lead through holder0.5.2 Fit each twisted wire to correct terminal0.5.2.1 Select one coloured lead0.5.2.2 Identify lead based on colour (earth, live, neutral)0.5.2.3 Locate correct terminal for selected lead0.5.2.4 Unscrew terminal0.5.2.5 Place exposed twisted wire through terminal hole0.5.2.6 Tighten terminal screw0.5.3 Secure main lead holder0.5.4 Replace plug cover

TTA Example – Selected ActivitiesID Task Plan Error

IDError Type Immediate

effects of errorDetection of error

Recovery of error

0.4 Ensure correct fuse is in place do in sequence 1-3; If mismatch between required and in-situ fuse then do ( 4)

0.4.1 Locate appropriate instructions for equipment

0.4.2 Read fuse requirement

0.4.3 Compare fuse requirement with given fuse

0.4.4 Change the fuse do in sequence 1-3

0.4.4.1 Select the correct fuse

0.4.4.2 Extract fuse from plug

0.4.4.3 Insert correct fuse

Error Taxonomy

• Classification scheme

• Generic error types

• Similar to HAZOP guidewords

• Taxonomy can be made domain specific

Error Taxonomy – SHERPA (see handout)

• Example error types for an action task

– E3 Action Omitted

– E4 Action too much

– E5 Action too little

– E9 Right action wrong object

– E10 Wrong action right object

Tabular Task Analysis Example - PracticalID Task Plan Error

IDError Type Immediate

effects of errorDetection of error

Recovery of error

0.4 Ensure correct fuse is in place do in sequence 1-3; If mismatch between required and in-0.4.1 Locate appropriate instructions for

equipmentE3 Action omitted Instructions not

obtainedUnable to confirm fuse type

Re-start task with instructions

E16 Wrong information obtained

Instructions for another device obtained

May not detect




0.4.4 Change the fuse do in sequence 1-3




Tabular Task Analysis - Guidance

• See Handout (blank TTA)

• Review each activity one at a time

• Read through the generic errors in SHERPA

• Add error types to the TTA and fill-in the remaining

columns (see example for guidance)

Work in Groups (max. 5) - 10 minutes

Tabular Task Analysis – Solution to Practical

• See handout sheet for example error types against each activity (example is not a comprehensive record)

ID Task Plan Error ID

Error Type Immediate effects of error

Detection of error

Recovery of error

0.4 Ensure correct fuse is in place

do in sequence 1-3; If mismatch between required and in-situ fuse then do ( 4)

0.4.1 Locate appropriate instructions for equipment

E3 Action omitted Instructions not obtained

Unable to confirm fuse type



Instructions for another device obtained

May not detect



E3 Action omitted Fuse type not obtained

Unable to complete step 0.4.3

Return to instructions


Incorrect fuse information used

May not detect - device could fail

Repeat task with correct information


E11 Check omitted Incorrect in-situ fuse not detected


Repeat task


Incorrect in-situ fuse not detected


Repeat task

0.4.4 Change the fuse

do in sequence 1-3


E3 Action omitted No replacement fuse

Unable to complete step 0.4.4.3

Repeat task

E9 Right action on wrong object

Incorrect fuse selected


Repeat task


E3 Action omitted Original fuse left in place

Unable to complete step 0.4.4.3

Repeat task


E3 Action omitted No fuse fitted Device does not work

Repeat task

E17 Misalign Fuse fitted incorrectly

Device does not work

Repeat task

Example TTA - Note this TTA is not comprehensive as additional error types may apply

You cannot read this, but . . .see green handout

Error Identification - Question

• Assume the fuse must be changed

• Review the tasks to achieve the goal at 0.4

• Use the error taxonomy (SHERPA) to identify :

– Example of an error leading to no fuse being fitted

– Example of an error leading to the incorrect fuse being fitted

• Remember

– Error Effects

– Failure Criteria

Work in pairs - 5 minutes

Error Identification Question - One Solution

• Errors leading to no fuse being fitted

– Step 0.4.4.3, Error E3 Action omitted

• Errors leading to the incorrect fuse being fitted

– Step 0.4.3, Error E11 Check omitted

– Step 0.4.4.1, Error E9 Right action on wrong object



Representation

Human Reliability Assessment and Risk Models

• Risk models will usually include human errors for

quantification (human as mitigation)

• Human Reliability Assessor will collaborate with the

Risk Modeller

– Further investigation may be needed in order to carry out

Human Reliability Assessment

– Additional errors may be identified for inclusion in the risk

model

– Changes to models may be necessary to represent human

error

Risk Assessment - General

• Risk = Frequency x Consequence/Severity

• Assessment of a complex system requires a structured process (Probabilistic Safety Assessment)

• Operation of the system is represented by a model (risk model)

• Risk model represents features in the system that prevent or mitigate against serious consequences (e.g. safety systems, intervention from human operators)

Risk Models – A Whirlwind Tour

• Hazard identification process used to establish a set of initiating events (what can happen to the system)

• Frequency of each initiator is assessed

• Consider the effects of each initiator on the system

• Typically use event trees to model accident sequences

• System features are ‘modelled’ as events in an Event Tree (ask success/failure questions as top events)

• Fault trees used to investigate detailed causes of equipment/system/human failure

An Event Tree

S

F

Respond to Alarm

Shut Valve

Start Pump

Success

Success (recovered)

Failure 1

Failure 2

Failure probability = x Success probability = 1 -x

x

1 - xInitiating event – system

leak

An Event Tree - Quantified

S

F

Respond to Alarm

Shut Valve

Start Pump

S1

S2

F1

F2

P(F) = F1 + F2 = (0.999 x 0.01 x 0.01) + 0.001 = 0.0011

P(S) = 1 – Failure = 1 – 0.0011 = 0.9989

0.001

0.999Initiating event – system

leak

0.01

0.01

0.99

0.99

Fault Tree+ from Isograph(http://isograph-software.com/ftpovereta.htm)

A Fault Tree

Valve Fails to Shut

Electrical signal to

valve fails

Mechanical valve failure

Operator fails to demand valve to

shut

OR

A B C

Failure probability = A + B + C – AB – AC – BC + ABC

A

CB

For OR use A U B U C

For AND use A n B n C

Fault Tree - Practical Example

Create a Fault Tree for incorrect fuse in place (i.e. 0.4)

• Two types of boolean operators

1. OR

Occurrence of ANY event below causes failure above

2. AND

Only the occurrence of ALL events below causes failure above

OR

AND

Fault Tree - Practical Example Guidance

• Use the errors identified as the branches to the trees

• Think about the HTA to give an indication of the layers required

• Think about which operator to use

Work in Groups (max. 5) - 10 minutesA solution on blue sheet

Fault Tree - A Solution

Incorrect fuse is in place

Appropriate instructions not

found

Fuse requirements not read

Fuse requirement not compared with

given fuse

Fuse not changed

Correct fuse not selected

Fuse not extracted from plug

Correct fuse not inserted

OR

OR



QUANTIFICATION

Human Error Probabilities - Quantification

• Human Reliability Assessment exists to provide quantification of the probability of human error

• Human Error Probabilities are used in Probabilistic Safety Assessment (risk models)

• Obtaining or generating Human Error Probabilities requires a range of techniques

Ways to get Quantitative Data

• Historical records

• Collected data (direct or simulated)

• Estimation techniques (constructive, comparative)

• Judgement and experience

Historical Records / Collected Data

• Number of recorded events of interest over time provides frequency of error

• Number of recorded events of interest over number of chances for event to occur provides the probability of error

• Strengths

– specific to the error of interest

– data validity (true values)

• Weaknesses

– may not have recorded all instances of error (under estimate)

– may need a lot of data to get a fair answer

– hard to identify root of some errors

– collection method may affect reliability

– collection in simulators may not be realistic for actual errors

– design changes over time may affect reliability

A Selection of Estimation Techniques

• Technique for Human Error Rate Prediction (THERP)

• Human Error Assessment and Reduction Technique (HEART)

• Success Likelihood Index Methodology (SLIM)

• Paired Comparisons (PC)

Estimation – THERP (1)

• Technique for Human Error Rate Prediction (NUREG/CR-1278, 1983)

• Collected data from civil PWRs in the USA (mainly control room actions, some manual valve actions)

• Presented as a database of Human Error Probabilities

• Flowchart of options to navigate the database

• Simple error taxonomy (omission, commission)

• Includes human error dependence model

• Construct HEPs from basic error data

Estimation - THERP (2)

• Strengths

– Powerful method with good auditability and supporting qualitative material

– Well suited to proceduralised, structured assessments

• Weaknesses

– Resource intensive (better with more experience)

– Not adaptive

– Limited error reduction information

Estimation - HEART (1)

• Human Error Assessment and Reduction Technique (Williams, 1990)

• Generic Task (GT) data in 9 categories

• List of 38 Error Producing Conditions (EPC)

• Select GT based on descriptions and examples (each task has a reliability attached)

• Modify base reliability by considering EPCs

• Advice included on possible error reduction measures

Estimation - HEART (2)

• Strengths

– Simple method, not resource intensive

– Error reduction suggestions

– Versatile (generic nature adapts to many tasks)

• Weaknesses

– Does not model dependence

– Results can vary greatly dependent on initial assessment (GT selection)

HEART – PC Demo

http://www.ewe.ch/regional/regional_2_6.html

http://www.ewe.ch/regional/regional_2_6.html

Estimation - SLIM (1)

• Success Likelihood Index Method (Embrey et al, 1984)

• Panel of assessors (including subject experts)

• Evaluate performance shaping factors (PSFs) for task of interest

• Assign weighting as to the relative importance of the PSF to each other. Assign rating based on how useful the PSF is for the task.

• SLI derived from the sum of the ratings and weightings (ranks the errors)

• Calibrate SLIs with known data to convert SLI to HEP

P4P3P1 P2

Known data points

SLI

Estimation - SLIM (2)

• Strengths

– Flexible technique, good theoretical method

– Does not need task decomposition (task analysis and error taxonomies)

• Weaknesses

– Complex method, resource intensive

– Lack of valid calibration data (known values)

Estimation - Paired Comparisons (1)

• Paired comparisons (Hunns, 1982)

• Panel of assessors (including subject experts)

• Each assessor compares all possible pairs of error descriptions (decide which of the two is more likely for each pair)

• Combine all comparisons made by all assessors to produce a relative scaling of error likelihood

• Calibrate scaling with known data to convert to HEPs

Estimation - Paired Comparisons (2)

• Strengths

– Simple method, best use of limited known data

– Provides information on the quality of data from individual assessors (internal consistency check)

• Weaknesses

– Comparisons not suited to complex tasks

– Only suited to comparisons of similar tasks

Judgement / Experience

• Expert judgement

Judgement/Experience – APJ (1)

• Absolute Probability Judgement

• Panel of experts to provide direct generation of HEPs (subject experts and HRA expert)

• Assumes assessors are capable of making such estimates of reliability

• Describe the tasks of interest

• Describe errors and estimate HEPs

– Individual estimates aggregated

– Group consensus of estimates

Judgement / Experience - APJ (2)

• Strengths

– Simple method, allows constructive qualitative discussion

– Practical error reduction measures can be discussed during the assessment

• Weaknesses

– Prone to biases, may have little face validity

– Needs experienced experts

Criteria for Quantification Technique Selection

• Availability of data

• Applicability of data

• Ease of use (time, cost, resources, information)

• Data validity (justification)

• Experience of assessor

• Level of assessment needed

HEART - Practical Example

Use HEART to estimate the probability of fitting an

electric plug to a device incorrectly

• Assume :

– Time shortage for the task

– Written procedure (<10 steps) is used

– Over the shoulder checking is carried out

HEART – See Handout

• HEART paper

• Generic task descriptions

• Failure probabilities for each task description

– (50th percentile value should be used, 5th and 95th percentiles

indicate the uncertainty/range)

• Error producing conditions and associated

multiplication factors

HEART - Practical Example Guidance

• Read the Generic Task Descriptions

• Consider the task complexity and difficulty by examining the identified errors from the TTA

• Select a Generic Task

• Review the EPCs and select the ones you believe are relevant

• Modify the Generic Task base HEP using factors for selected EPCs

– See the worked example in the HEART paper

– EPC equation developed to avoid negative probabilities

Work in Groups (max. 5) - 10 minutesA solution on yellow sheet

HEART Practical - One (simple) Solution

• Probability of fitting an electric plug to a device incorrectly

– Generic Task F 0.003

– EPC 2 time shortage, x 11

• HEP = (0.003)(11) = 3.3E-2

Quantification Summary

• A range of HRA techniques is available

• Technique selection depends on the nature of the assessment

• Human Reliability Data can be difficult to obtain

• Human Reliability Data can be uncertain (range of probabilities)

• Information from the task analysis can be organised to suit the quantification technique (e.g. describe activities with HEART generic tasks in mind)

• Quantification must be based on detailed qualitative understanding

Capability Management Process for HRA

Prospective Analysis

HRA (concept)

HRA (design)

HRA (validate)

Deliverables required

Process Acceptance

CUSTOMER

Liaison

Market watch

SAFETY TEAMOwn, Maintain, Manage HRA Processes

Retrospective Analysis PRODUCT

Benchmarking

External SourcesOther

techniques/ external

databases/ review/researc

h

certify

accident/incident

Design

HRA Data

Safety Assessment Reporting

Experience/ Lessons Learnt

Maturity

Product design &

development process

BU REPSBest

practice, peer

review

Knowledge CaptureReporting and debriefing

FeedbackHE assessors, safety team

MeasuresCriteria (e.g. time, labour

involved, etc.)

Training

Process

DatabaseHRA data, lessons learnt, assessments, techniques

HRSelect & deploy team

Recording mgt.

Data & process

Process mgt.

Refine, tailor, what

& when

System Requirements

Regulatory Authorities

Human Reliability Assessment - Summary

• Understand the actions being investigated

• Use a structured approach to investigate and represent

the actions (task analysis)

• Consider the level of detail needed (compare with

description detail of available data)

• Understand the failure criteria

• Select an appropriate technique(s)

• Represent the identified errors in a risk model

Objectives – Re-visited

• Understand the Human Reliability Assessment process

• Gain practical experience of a simple task analysis

• Gain practical experience of error identification

• Gain practical experience of risk models

• Gain practical experience of quantifying error probabilities in a simple example

Useful References

• The following books provide a comprehensive guide to

Human Reliability Assessment and Task Analysis

A Guide to Practical Human Reliability Assessment,

Barry Kirwan (1994),

Taylor and Francis

ISBN 07484-0111-3

A Guide to Task Analysis,

Barry Kirwan & Les Ainsworth (1992),

Taylor and Francis

ISBN 07484-0058-3

Documents

Human Reliability Assessment