The Impact of Interface Developments on Operator Consistency · michael.hildebrandt@hrp.no...

The Impact of Interface Developments on Operator Consistency

03.10.2013 2

Michael Hildebrandt michael.hildebrandt@hrp.no

Industrial Psychology Division Institute for Energy Technology /

OECD Halden Reactor Project, Norway

Example

03.10.2013 3

Traditional process display

03.10.2013 4

Information-rich display

03.10.2013 5

IRD for the Halden BWR LSD

03.10.2013 6

In operation since 2012, developed in cooperation with CR operators

03.10.2013 7

Institute for Energy Technology R&D for the nuclear, oil & gas industry

OECD Halden Reactor Project

03.10.2013 8

OECD Halden Reactor Project • International collaborative research project

• Fuels & Materials • Man, Technology, Organisation (MTO),120 staff

• Halden Agreement first signed in 1958 • Renewed every 3rd year • Current program period from 2012 to 2014

• HRP is jointly funded by its members: • 18 countries • > 100 nuclear organizations world wide

• Participant types • Utilities, Vendors, Licensing Authorities and R&D centers

HRP Members – Signatory Members USA US NRC - United States Nuclear Regulatory Commission Norway IFE - Institutt for energiteknikk Belgium SCK/CEN - Belgian Nuclear Research Centre Denmark Risø DTU Finland MITI - Finnish Ministry of Trade and Industry, VTT France EDF - Electricité de France Germany GRS - Gesellschaft für Anlagen- und Reaktorsicherheit Japan JNES - Japan Nuclear Energy Safety Organization Korea KAERI - Korean Atomic Energy Research Institute Spain CIEMAT - Spanish Centro de Investigaciones Energéticas, Medioambientales y Tecnológicas Sweden SSM - Swedish Radiation Safety Authority Switzerland ENSI - Swiss Federal Nuclear Safety Inspectorate UK NNL – National Nuclear Laboratory

03.10.2013 9

HBWR – Reactor hall MTO-lab Facilities

Facilities MTO-lab

HRA research areas in Halden

Improving HRA

practice

Halden HRA

database

Improving emergency and severe

accident operation

In this presentation • Operator performance, reliability and resilience • Halden interfaces

• Procedure flowchart • State-based alarm systems • Large-screen displays / IRD • Operator screens on iPads for ex-CR staff • Task-based displays • (Highly automated plants)

03.10.2013 13

Operator performance, reliability and resilience

Part 1

03.10.2013 14

Why worry about interfaces? • Information overload • Keyhole effect • Achieving / maintaining situation

awareness • Anticipation / prediction • Human reliability • Knowledge-based decisions, complexity • Resilience

03.10.2013 15

Why is it difficult to draw conclusions about the benefits of innovative interfaces on operator performance in PSA scenarios?

03.10.2013 16

(1) Many criteria for assessing innovative interfaces

• Performance on PSA scenarios / HEP effect • Workload / SA • Operator feedback • Usability • Expert evaluation • Early in the design process:

Assessment of “promisingness”

• It depends partly on your audience / customer

03.10.2013 17

(2) Project drivers • Improving performance on PSA scenarios is

not the direct driver in many of our projects • Beginning to change

• Focusing too much on PSA/HRA type of issues might not always lead to the best interface

03.10.2013 18

(3) Interfaces are part of the larger Emergency Operating System

• Difficult to isolate the effects of interfaces on performance, even in factorial experiments • Operators adapt, strategies change • Compensation, working harder

• Some measures not specific enough to the effect of interfaces

• Some measures too specific to interfaces, not taking into account the overall mission goals and work demands

03.10.2013 19

(4) Effect size • Effect of interface manipulation may not

be large enough to show statistical significance on HRA-type measures

03.10.2013 20

(5) Crew variability • Effect of interfaces can be drowned in

the variability between crews, and even between runs for the same crew

• Usually low sample size • Trade-offs in study design

03.10.2013 21

(6) Baseline performance • EOS performance (operators, procedures,

training, interfaces) is often very high • Even in relatively complex scenarios we see

many crews succeeding • Success on HFE criteria • Not necessarily good performance on other

measures

03.10.2013 22

(7) Baseline CR interface • Baseline control room EOS has proven to be

good enough for many scenarios • In our recent HRA studies, we have not seen

“HMI” show up as a main negative driver (mostly nominal)

03.10.2013 23

(8) Demand • Not clear if there is demand for HRA simulator

studies where factor such as HMI, staffing are systematically manipulated • Does an HRA analyst need / want this information?

• De facto: “Segregation” between HRA studies and interface design studies

• Attempt to change this with new HRA program • Improving emergency and severe accident operation • Test case: 2013/14 study on Resilient Procedure Use

03.10.2013 24

Measurement

Work on Integrated System Validation

03.10.2013 25

Objective/Scope 2012-14 Ø Performance criteria and acceptance criteria issue Ø Development of basis for Criterion-Referenced Approach (CRA) Ø Supplement or alternative to the benchmark approach

Ø Context issue – not sufficient NPP operation and safety relevance of many commonly recommended measures

Ø Causality issue Ø Performance effects of specific control room elements hard to

establish in integrated tests of high reliability systems (redundancy and diversity)

Ø Coherence Issue of performance measures Ø Measures reported in the literature often concentrate on one

performance dimension (for example Performance or Teamwork). Ø Coherent measures important for ISV evaluating the realistic “full”

operation.

03.10.2013 26

Framework Overview

03.10.2013 27

Cognitive Dimensions

03.10.2013 28

Situation Understanding

Plant state and Boundaries

Function / System

Component

Mission

Mission Analysis

Goals

Actions

Control and Verification

Mission Verification

Progress Monitoring

Execution control

Response Verification

Team Cognition

Leadership Decision Making Consultation Communication Back-up Workload Attention Resource Management Team Self-evaluation

Interfaces

Part 2

03.10.2013 29

resilient procedure use simulator study 2013/14

resilience is about… • the ability of a system to adapt to

unforeseen, challenging situations (including but not limited to Fukushima-scale events)

• the ability to mobilize additional resources (cognitive, physical, expertise, staff etc.) when a system approaches its margin of maneuver

• enabling smooth transfer of control

resilient procedure use requires balancing of

heads-down operation

heads-up operation and

How to support this? - Crew roles - Procedure support tools - Overview tools (e.g. LSD) - External support - Others?

Experimental design

03.10.2013 33

Run 1

Run 2

Run 3

Run 4

Shift technical advisor

✔ ✔

✗ ✗ Large-screen display, Procedure flowchart ✔

✗ ✔

✗

Swedish and US crews; four PSA scenarios with complications

new large screen display

status of important

safety systems

status of automatic systems, and other important alarms (e.g. RMS)

sub- cooling

mass balance

increasing/ decreasing

PRZ pressure/ level trends; alarm levels; trip points

new large screen display: primary

SG pressure / level trends

procedure flowchart for STA

Link to background material >

Key decision point >

Critical action steps > < Grouping of procedure steps

< Procedure /step transfers

< Procedure transfers (clickable)

< Short description of step links to full text

Step number >

Link to high-level summary >

CSF status, link to decision tree >

Foldout page, link to appendix >

Some flowcharts contain diagrams with live data >

*

03.10.2013 40

A State Based Alarm System for Halden Boiling Water Reactor (research reactor)

HWR-1065

Christer Nihlwing and Magnhild Kaarstad

General alarm system challenges

03.10.2013 41

Too many alarms Misleading alarms Secondary malfunctions difficult to detect Difficult to get an overview of the situation

NOT USEFUL DURING DISTURBANCES

Purpose and recommendations of the alarm system

The primary function of the alarm system (EEMUA 191, p. 4): • The purpose of an alarm

system is to direct the operator’s attention towards plant conditions requiring timely assessment or action.

03.10.2013 42

The alarm functions shall provide the following features (NUREG-0700/ IEC 62241): • Alert • Inform • Guide • Confirm

Number of alarms first 10 minutes of an upset

Consequence

More than 100 alarms Definitely excessive and very likely to lead to the operator abandoning the use of the system

20-100 Hard to cope with Less than 10 Should be manageable, but may be difficult if several alarms

require a complex operator response

SBA - A different approach to Alarm System Development • Several tests and efforts – difficult to find optimal solution • Basic alarm management - priorities and suppression - not

satisfactory • State Based Alarm (SBA) system developed in multilateral

cooperation • Based on a number of well-defined process states (19) • Continuously adjust itself to actual state • Shows only alarms relevant for the current state of the process • Add-on to an ordinary alarm system • Displayed in a separate list, separate alarm sound

03.10.2013 43

State based alarm presentation

03.10.2013 44

SBA performance - Alarm rate Turbine trip and condensate problem

03.10.2013 47

Ordinary alarm system: 257 State Based alarm system: 26

Turbine trip and condensate problem

FW problem and scram

FW regulation problem and loss of power

Alarm system SBA Ordinary SBA Ordinary SBA Ordinary Alarms 10 first minutes

23 259 33 297 21 424

Alarm rate first 10 minutes into disturbance

Operator performance with the SBA

03.10.2013 49

State based Ordinary Percentage detected 100% 62% Percentage corrected 100% 62%

Detection of secondary disturbance during ongoing disturbance

State based Ordinary Percentage detected 83% 67% Average time for detection

20 sec 33 sec

Average time to detect the return of the 400kV

Usability of SBA

System Usability Scale (SUS) (Brooke, 2001) • Ordinary alarm system: 75 • State based alarm system: 83 Operators´ subjective opinion regarding usability: • Faster control of the situation, easy to notice new relevant alarms • Manageable amount of alarms • Only alarms that need attention were presented • Liked the way alarms were presented in displays and in the list • The colours were clear, and informed about when to enter EOPs

03.10.2013 50

Conclusion • High usability score, and operators expressed that with

the SBA it was easy to take care of a disturbance. • Alarm rate of SBA can be within EEMUA

recommendations. • Operators seemed to detect and correct a secondary

disturbance faster and more often with the SBA. • Promising with regard to usefulness of an advanced

alarm system tool like the SBA. • Provide input to industry with regard to design and

development of advanced alarm system tools.

03.10.2013 51

Large Screen Displays – a Usability Study of Three Different

Designs

HWR-1025

Magnhild Kaarstad Stine Strand

Background • Part of an ongoing Nordic research cooperation since 2000 • Evaluate various large screen design (LSD) solutions

• Traditional LSD • Advanced LSD • Information Rich Design LSD

• Identify how they support operators in their control room tasks

• Operator feedback with regard to usability, use, usefulness and areas of improvements.

53

The evaluated Large screen displays

54

TRADITIONAL - Physical layout of process - Traditional diagrams - System lines indicate flow - Trends (3) - Traditional symbols - Bright colours - Frames around active alarms - Safety systems - Actuated safety system

3

The evaluated Large screen displays

ADVANCED - Physical location of process areas - No system lines - 12 mini-trends and combination trends - Traditional and advanced symbols, bar graphs - Bright colours - Alarms are small squares next to affected system - Safety systems aligned - Actuated safety systems - Frame around selected objects

3

The evaluated Large screen displays

INFORMATION RICH DESIGN (IRD) - Physical layout in areas - A few system lines - Pattern recognition by normalised trends - Traditional and advanced symbols - Dull colours - Alarms: small squares and alarm spot - Safety systems aligned - Actuated safety system

Operator feedback Traditional LSD

Positive • Worked well as a general

overview in normal situation • Simple, systematic display • Reactor and containment • Direction of flow • Symbols, mimics, lines • Background colour, contrast • Yellow alarm frames • Safety system actuation

5

Negative • Too few mini-trends • Unclear design of running

pumps • Design of safety system a bit

confusing • Too much in yellow alarm

colour in disturbance • Need more information to be

able to analyze a disturbance

Positive • Good overview in all situations • Appreciated mini-trends • Pump symbols • Intermediate position of valves • Background colour, contrast • Safety systems • Safety system actuation • Alarm information • Frames around active objects

6

Negative • Direction of flow not clear • Interpretation of some

diagrams • Information represented in bar

graphs • Control rod map • Alarm information • Larger disturbances

Operator feedback Advanced LSD

7

Positive • Good general overview • Normal operation, periodic

tests • Straight line: everything is

normal • Mimics and lines • Background colour • Reactor core • Alarm spot

Negative • Too few numbers • Information too small • Location of some valves • Lack of process lines • Safety system actuation • Mini-trends should have had

frames and scales • Contrast, information too dim • Difficult when too many alarms

Operator feedback IRD LSD

Usability evaluation System usability scale

• Traditional: 75 + Process understanding - Information availability

• Advanced: 60 + Deviation detection - Graphical interpretation • IRD: 59 + Deviation detection - Graphical interpretation

03.10.2013 8

Additional usability evaluations: • All LSDs particularly useful for:

• Colleague cooperation • Shared understanding

Summary of LSD evaluation • Three different LSD designs with

different philosophies and different strengths and weaknesses evaluated • Challenge: Alarm amount • Advantage: Common frame of reference

• The traditional was rated slightly higher than the two others

• None of the designs are completely adequate but all LSD concepts seem useful

9

03.10.2013 62

Information Rich Design (IRD) principles

Alf Ove Braseth

03.10.2013 63

IRD Large screen design philosophy • Provide a stable frame of reference; no pup-ups, new windows

• Shall be easy to spot deviations from normal operating conditions;

ideally before alarm state

• Prevent information overflow to the operators

• Shall complement the operator stations, no intervention or operation in the large screen.

03.10.2013 64

This color scheme leads to information overflow

Dullscreen Bright colors and contrast do not contribute to better design

03.10.2013 65

It is possible to create information rich design without going into “information overflow”

Edward Tufte

Pat. Pending

This is a Dullscreen example

03.10.2013 66

Dullscreen is not enough; enabling pattern recognition is also important

Jens Rasmussen (Risø); the SRK model: Skills :Riding a bike :Effortless, unconscious, high capacity Rules :Soccer :Fun, but can be demanding Knowledge :von Neumann problems :Demanding, conscious low capacity

Knowledge based

Skills based using pattern recognition

03.10.2013 67

What is normalization?

With normalization all six variable can be read as one unit Without normalization the six variables must be read and interpreted separately

03.10.2013 68

IRD component examples

Liquid levelPressureTemperature Flow Liquid levelFlowPressureTemp.

Alarm presentation

With trend information Without trend information

48

H2

130

H3

H4

IRD for the Halden BWR control room

03.10.2013 69

In operation since 2012

03.10.2013 70

03.10.2013 71

Further reading on IRD

JOINT POSTER ON:

HWR-936: Innovative HSI concepts - rationale and design solutions C. Nihlwing, J. Kvalem

HWR-996: Work practices and new technologies – iPad as information source

M. Kaarstad, S. Strand, C. Nihlwing, T. Karlsson

iPad How it Works and

How it can be Used in NPPs

Purpose iPad

73

Present continuously updated process information to staff located outside the main control room

Examples of information on the iPad • All process displays • Overview display • Alarms • Plant status circles • Plant automation • State based alarm list • Process flow charts • Logic flow charts • Procedures

• Updated on process situation before entering control room

• Directly prepared for discussions and decisions with crew members

• Can start immediate actions

How the iPad works

iPad client requests display to server and read it as jpg file from web server

Server selects, capture and put the display as jpg file on web server

iPad operator Process display area: A computer with two 30” displays

74 HWR-936: Innovative HSI concepts, rationale and design solutions

HSI working areas visualized on two 30” displays

75

Plant automation State based alarms

Plant status circles Overview Display

A server program handles the communication between iPad, HAMBO and web server

HWR-936: Innovative HSI concepts, rationale and design solutions

Process displays

iPad connection and dataflow

Client app on iPad in objective-C

1. Sending request for Statebased Alarmlist

2. Capture selected area, put it on the webserver cyclic

76 HWR-936: Innovative HSI concepts, rationale and design solutions

A server program handles the communication between iPad, HAMBO and web server

Area for capturing requested display

Web server

3. Reading from Web server

Controlling simulator from iPad

Start, stop, loading snapshot, set malfunction, operating objects in the HAMBO simulator can be done from a pop-up display with a picker wheel

77 HWR-936: Innovative HSI concepts, rationale and design solutions

Display navigation

78 HWR-936: Innovative HSI concepts, rationale and design solutions

Zooming overview display on iPad

79 HWR-936: Innovative HSI concepts, rationale and design solutions

Zoom

Work practices and new technologies- use of iPad for supervisor

• Background • Identify challenges and opportunities with introduction of new

technologies and computer-based control rooms • Possible changes:

• Communication • Operator tasks • Information needs • Transparency of team member activities

• Current focus on Shift supervisor (SS) • Needs information about the process and team member activities

80 HWR-996: Work practices and new technologies – iPad as information source for shift supervisor

Research goal Explore whether iPad can be used as an information source for the shift supervisor when he/she is located outside the

main control room

81 HWR-996: Work practices and new technologies – iPad as information source for shift supervisor

82

HAMMLAB Location of Crew

Experimenters gallery

”Office/ Meeting area” Location of Shift supervisor

The iPad Study

HWR-996: Work practices and new technologies – iPad as information source for shift supervisor

Data collection • Simulator: HAMMLAB PWR (RIPS) • Five crews from nuclear power plants participated in the

experiment • Three to six licensed operators in each crew

• 4 short test scenarios • In two scenarios, Shift supervisor was equipped

with iPad when located outside the control room • In two scenarios, Shift supervisor was NOT equipped

with iPad when located outside the control room

83 HWR-996: Work practices and new technologies – iPad as information source for shift supervisor

Scenarios & Data collection - overview

84

After EACH scenario Operator ratings Process overview Mental demand CR operator activities Teamwork Shared understanding

After ALL scenarios iPad usability ratings System Usability Scale iPad display usefulness Frustration Satisfaction Interview During ALL scenarios

Process Expert performance evaluation

HWR-996: Work practices and new technologies – iPad as information source for shift supervisor

Start 1 min

Supervisor located outside CR

4 min

Supervisor enters CR

10 min

Scenario end Disturbance

Preliminary findings

Examples of iPad benefits stated by operators • Overall, a very useful tool when out of CR (e.g. meetings) • Supervisor already updated on process situation when

entering the control room, directly prepared for discussions • Only a short turnover is needed

• Decreased communication needs • Control room operators less distracted from their tasks

• Misunderstandings less likely

• Can help avoid tunnel vision • Supervisor do not tie up CR personnel in their procedures

85 HWR-996: Work practices and new technologies – iPad as information source for shift supervisor

Preliminary findings Usefulness personnel groups • Shift supervisor/ Shift

manager • Operations manager • Plant manager • Upper level management • Chemistry personnel • Maintenance • Emergency personnel • Technical support centre • Field operators

86 HWR-996: Work practices and new technologies – iPad as information source for shift supervisor

iPad application areas • Training issues • Procedures • In meetings • Take pictures/videos of

components, send to CR staff for immediate evaluation

• Work orders • Day-to-day monitoring, avoiding

disturbances • Evacuation of control room • Easier for field operator to find

correct objects

Preliminary findings

87

3.8

5.0 4.8 4.8

3.8 4.4 4.4

4.8 4.4 4.6

0

1

2

3

4

5

• Very high usability ratings (High score = Good Usability) • All supervisors would like to use this iPad functionality in home plant

6.4 6.4 6.0 6.2 6.2 6.0

1

2

3

4

5

6

7

System Usability Scale (5-point) Overall Usability ratings (7-point)

88 HWR-996: Work practices and new technologies – iPad as information source for shift supervisor

Preliminary findings When iPad available in the first part of

the scenario, Shift Supervisor experienced Significantly less mental demand Significantly better process overview Significantly better overview of the CR

operator activities at the end of the scenario

When iPad available in the first part of the scenario, Shift Supervisor experienced that obtaining shared understanding of the process state was Significantly easier Significantly less mentally demanding Significantly less time consuming

Preliminary findings

89

When iPad available, Shift Supervisor needs significantly less time to be updated on the process situation when he/she enters the control room

HWR-996: Work practices and new technologies – iPad as information source for shift supervisor

Preliminary findings - Summary • iPad is a very useful tool for the shift supervisor when not located

in the control room • Operators very positive to the iPad, and imagine a broad range of

application areas and use for different personnel • Very high usability ratings • Significantly less mental demand, better process overview, better

overview of the CR operator activities, better shared understanding when iPad available

• Important considerations • Avoid that operators feel monitored by management • The iPad should NOT be used to control the process

• Technical solutions to avoid this are available

90 HWR-996: Work practices and new technologies – iPad as information source for shift supervisor

03.10.2013 91

Procedure and Process Integration Displays

Presentation of the Concept and the HAMMLAB User-test 2006

03.10.2013 92

Presentation of the concept

03.10.2013 93

Design criteria / Goal • Easy to use and in a way that is natural for process operators § Maintain the process overview when performing the procedures § Relevant information provided when performing procedure steps § Relevant feedback from process after step completion § Possible to mark steps with problems - easy to find later

• High transparency of operators work • Keep overview of performed procedures and procedure steps • Minimize navigation. The relevant information shall come to the operator • Minimize the risks for erroneous operations • Design tool: Possible for operators to implement/change procedures

without programming knowledge. • Easy to review • Possible to use for normal-, abnormal- and emergency situations

03.10.2013 94

Procedure and Process Integration Displays. HAMMLAB 2009

2009 the concept was adapted to one 30 inch screen.

03.10.2013 95

Procedure and Process Integration Displays. HAMMLAB 2013

Procedure Selection and Overview Display

Procedure Performance Display

Event-dependent Assistance display

Now the process displays has 30 inch format and by that two 30 inch screens needs for the concept.

03.10.2013 96

Procedure Selection and Overview Display

Reactor procedures Turbine procedures

Emergency Operating Procedures

03.10.2013 97

Push buttons to activate or observe procedures

Procedure Selection and Overview Display

03.10.2013 98

Overview of the selected procedure

Not active Actuated protection signal Active Interrupt without remains Interrupt with remains Completed Observation

Color codes for

procedure status

Displays with procedure steps

Procedure Selection and Overview Display

Reactor run-back

Containment isolation

03.10.2013 99

Close active procedure

Close procedure system

Alternative steps

Procedure Selection and Overview Display

03.10.2013 100

Procedure Performance Display Procedure text

Actual components are marked with a frame. The arrow shows that the valves shall be closed.

Navigation buttons

03.10.2013 101

When the procedure step is performed the operator press the ”Sign” button if the step is OK. If the operator wants to go back to the step later, he/she press the ”Rest” (remaining/postponed) button.

Procedure Performance Display

03.10.2013 102

The remaining/postponed list

”Rest” marked steps.

”Rest” marked steps in all procedures are visible in the Reactor operator or Turbine operator ”Rest” list.

It is possible to select a “Rest” marked step and jump directly to the procedure step.

03.10.2013 103

The 2005/2006 Usability Test

Ø Performed in the HAMBO simulator Ø 3 crews (RO, TO, SS) participated in 4 scenarios Ø The concept was implemented for all operators Ø The purpose of the test was to assess the usability of

the concept - not performance

03.10.2013 104

Results – the overall concept Ø Easy to learn Ø 4 hours of training sufficient Ø Relative comfortable in using the concept in simulator Ø Considered the concept as good and advantageous Ø Positive toward application of the concept for other tasks

• Complexity rated as low • Easy to use • Organization of functions good • Felt confident in using • Considered as safe to use

Usability scale

0

1

2

3

4

5

6

7

1 2 3 4 5 6 7

Response scale

Freq

uenc

y

Complexity (mean = 6.5)

Ease of use (mean = 5.9)

Function organisation (mean =6.4)

Confidence in use (mean = 6)

Safety in use (mean = 6)

03.10.2013 105

Results – display specific All 3 display types Ø Useful Ø Easy to understand Ø Low frustration Ø Did not miss information

in the displays • Clear results for

understanding and frustration for all display types

• Larger variation and lower mean values for information amount (mean value still above 5 on 7-point scale)

Understanding1: very difficult to underst. 7: very easy to underst.

0

2

4

6

8

1 2 3 4 5 6 7

Response scale

Fre

qu

ency

PO (mean = 6.5)

Proc.perf (mean = 6.5)

Eda (mean = 6.1)

Frustration1: very frystrated 7: not frustrated

0

2

4

6

8

10

1 2 3 4 5 6 7

Response scale

Fre

qu

ency

PO (mean = 6.7)

Proc.perf (mean = 6.7)

Eda (mean = 6.8)

03.10.2013 106

Results – display specific (cont.) Procedure Sel./Overview Display

• Good overview of procedures • Rest-list and

observation function appreciated • Intuitive, self-explanatory • Easy to select procedure,

easy to see procedure status • Easy to navigate

Procedure Performance Display • Good illustration of components • Easy, fast, safe • Few navigation problems • Disagreement in the need for

textboxes • Suggestions for improvement

e.g. color, text amount

Event-Dependent Assistance Display • Recognized benefits of a display relevant for the respective procedures • Provided quick overview (level, pressure, pumps, trends most useful) • Generally positive toward use of expert-help • Not all noticed and/or used the display (poor training + additional

information available, i.e. large overview display) • Suggestions for improvement, e.g. training in use of the display

03.10.2013 107

Process Expert (PE) rated performance

Ø PE qualitative ratings related to procedure handling and scenario handling judged to be good (no problems in handling procedures or scenarios)

Ø PE quantitative ratings relatively high

• Communication • Crew Efficiency • Goal achievement • Procedure handling

PE quantitative crew performance ratings (average)

5,17

5,75

6,255,83

0

1

2

3

4

5

6

7

Communication Crew efficiency Scenario goal attainment Procedure handling

03.10.2013 108

Comparison with paper-based proc. (“Mental” judgment) Ø Easier Ø Less time-consuming Ø Less mentally demanding

Ø Easier to recall deviations Ø Easier to recall previous

activities Ø General preference for the

computer based (60%) Ø Pitfall: process overview

Comparison between TBD and paper-based procedures

0

1

2

3

4

5

6

7

1 2 3 4 5 6 7

Response scale

Freq

uenc

y

diff icult/easy (mean = 5.6)

more/less timeconsuming(mean = 6.2)more/less mental demand(mean = 5.9)

more diff icult/easier to recalldev. (mean = 5.7)more diff icult/easier to recallact. (mean = 5.3)

03.10.2013 109

Summary and further plans Ø All measures provided results in the same direction

• The concept well received by the operators • The concept considered as valuable and useful

Ø Input to further refinements provided Ø The initiated work should be continued

• Currently incorporated in HAMMLAB Experiment set-up for Work Practices Experiment

Ø Plans (initiated):

• Continue with refinements of the current design • Further development of the Emergency Operating

Procedure (EOP led by the shift supervisor) • Expand the concept to other tasks

(e.g. start-up/shut-down procedure)

Summary • Halden developed and tested many different interfaces • Most have received positive feedback from crews • Performance benefits are evident in many cases

• But not all

• More difficult to show direct effects on PSA scenarios • Methodology, scenarios, sample size • Highly reliable EOS, low failure rates

• Many of these interfaces may have their strength in supporting knowledge-based emergency operation • Resilience

03.10.2013 110