Upload
brianna-cole
View
216
Download
0
Tags:
Embed Size (px)
Citation preview
Theorising Learning from Incidents: A Human-Machine
Systems PerspectiveLing Rothrock
The Harold and Inge Marcus
Department of Industrial and Manufacturing Engineering
The Pennsylvania State University
University Park, PA 16802
University of Aberdeen, 11-12 June 2014
Ling Rothrock
EMPLOYMENT HISTORY:
Associate Professor, The Pennsylvania State University, 2008-present
Assistant Professor, The Harold & Inge Marcus Department of Industrial & Manufacturing Engineering, The Pennsylvania State University, 2002-2008
Assistant Professor, Department of Biomedical, Industrial, and Human Factors Engineering, Wright State University, Dayton, OH, 2000-2002
Research Scientist, Army Research Laboratory, Human Research and Engineering Directorate, FT Huachuca Field Office, AZ, 1998-2000.
Officer, United States Army, FT Bliss, TX, 1996-1998.
FIELD INSTITUTION DEGREE DATEIndustrial Engineering Georgia Institute of Technology Ph.D. 1995Industrial Engineering Georgia Institute of Technology M.S. 1992Applied Mathematics Florida Institute of Technology B.S. 1990
PhDs
Jung Hyup Kim, 2013, Assistant Professor, University of Missouri.
Namhun Kim, 2010, Assistant Professor, Ulsan National University of Science and Technology
Jing Yin, 2009, Consultant, The Ironside Group, Inc.
Damodar Bhandarkar, 2008, Senior HF Engineer, Pritney Bowes
Hari Thiruvengada, 2007, UX Design Manager, Honeywell, Inc.
Sungsoon Park, 2007, Principal Consultant, Samsung SDS
Dongmin Shin, 2005, Associate Professor, Hanyang University
Purpose of Visit
The objectives of my sabbatical leave are to improve my professional skills through working with notable researchers in my field; applying my skills toward challenging human-machine problems in refinery process control; and create case studies to give undergraduate and graduate students an appreciation for current problems in process control.
Department of Industrial and Manufacturing Engineering at Penn State University
World’s first industrial engineering department founded in 1909
Students: ~450 undergraduates; ~70 MS; ~70 PhD
32 faculty members
Research Areas Human Factors – ergonomics, human centered design, human-
computer interaction Manufacturing – distributed systems and control, design Operations Research – applied probability and stochastic systems,
optimization, game theory, statistics and quality, simulation Production, supply chain, health systems engineering, service
engineering
Consider Advanced Process Control
ASM Examples drawn from the Abnormal Situation Management (ASM) Consortium
Categories
0 3 6 9 12 15 18 21 24
Frequency
Defective Installation
Failure to Follow Procedure/Instruction
Failure to Recognise Problem
Inadequate/Incorrect action
Inadequate Work Practices
Inadequate or No Procedure
PeopleandWorkContextFactors
EquipmentFactors
Defective Equipment
Equipment Design Flaw
Equipment/Mechanical Failure
ProcessFactors
Operation Beyond Original Design Limits
Process Design Flaw
Source: ASM Consortium
Sources of Plant Disturbances
Presented by N Kosaric at 2005 Defect Elimination Conference
Causes of Equipment Failure
Source: ASM Consortium
Causes of Process Upsets
RISK OF HUMAN ERROR
40%
Equipment Failure Other
20%
Human Error
40%
Causes of Plant Disturbances
HUMAN FACTORS
TOOLS
MACHINES
SYSTEMS
TASKS
JOBS
WORK ENVIRONMENT
“Failure to Adequately Inform and Engage the
Human-in-the-Loop in Automated Processes.”
Source: ASM ConsortiumThe Fundamental Problem
Key Learning from ASM Projects
Typical analyses that focus on just root causes are insufficient for identifying systemic improvement opportunities: Root causes explain ‘why’ something occurred, not ‘what’
occurred in terms of failures Root causes are general and not specific enough to drive
continuous improvement – details are buried in incident report No effective methods for aggregating root cause details across
incidents for systemic analysis of problems and improvements
IncidentEvent NEvent 2Event 1 Event N+1
RootCause
‘Why’ event occurred
Missing ‘What’ went wrong
RootCause
How aggregate details withinand across incidents?
Models
Research Process to Improve Learning from Incidents
ScenarioHuman-in-the-loop (HITL) Simulation
Platform
Database
Implement dynamic events to test hypotheses
Data logging includes operator interaction, system states, and required activities
Enable data access to measure performance and inform model building
Validate findings in context
Subject-MatterExperts
Extend findings to industryCall Centers
Command and Control
Hypothesis and Experimental Design
Formulate hypothesis based on incidents and causal factors
Construct platform to simulation domain
Construct computational models of human performance and judgment
Process Control
HITL Performance Measurement: Windows of Opportunity
A construct that specifies a functional relationship between a required situation and a time interval that specifies availability for action.
FalseAlarm
MissCorrect Rejection
Situation No SituationRequired Required
Environment
Action
No Action
Response
Correct
Incorrect
Early On-time Late
A B C
D E F
G
H I
Human-in-the-loop (HITL) Simulation
Platform
Windows using Temporal Logic
Given an action , b, and a time window, w,
we define 6 predicates, M, such that,
situationmeet not does if 0
win specified situationmeets if Iw b
bb
1)(1
wtoward relevant not is if 0
wtoward relevant is if Iw b
bb
1)(2
)(]1)([1||),(|| 1,
1sijwTTji OwIthatsuchiiffwM
i bb
)(]1)([1||),(|| 1,
2sijwTTji UwIthatsuchiiffwM
i bbA
B
)(]1)([1||),(|| 1,
3sijwTTji CwIthatsuchiiffwM
i bbC
]1)([]0)([1||),(|| 21,
4 jwjwTTji iiIIthatsuchiiffwM bbbD E F
)0)((,1||)(|| 2,
5 jwTTj iIiiffM bbG
)0)((,1||)(|| 2,
6 jwTTi iIjiffwM bH
Rothrock, L., & Narayanan, S. (Eds.). (2011). Human-in-the-loop Simulations: Methods and Practice. London: Springer-Verlag.
Extension to Team Research
Output files A Output files B Output files C
Phase 3: Analyze Team/Individual Performance Integrated
Output files
PerformanceAnalyzer
Tool
Team/ Individual member performance
Phase 2: Run Simulation
Team Role A
Each member communicates with the other using speech and internal messaging system
TeamRole B
TeamRole C
Script Maker
ScenarioPhase 1: Scenario Generation
Experimentation Strategy
Human-in-the-loop (HITL) Simulation
Platform
No Training (NT) Team Coordination Training
(TCT)Task Delegation Training (TDT)
1. No specific training is imparted.
2. Team members are provided with information on the definition of team coordination and task delegation.
3. No specific tasks are delegated to each operator role.
1. Team Coordination is emphasized during training.
2. Team members are instructed on how to achieve effective coordination via demonstration of good and bad practices.
3. No specific tasks are delegated to each operator role.
1. Task Delegation is emphasized during training.
2. The operator’s display is split into two distinct areas and is designated to each of the two roles. Operators monitor and perform actions within the designated area, while passing information pertaining to the other area onto their teammate.
3. Specific tasks are delegated to each operator role based on competencies and operator capabilities.
Research Question: Is one form of training superior?
Database
Rothrock, L., Cohen, A., Yin, J., Thiruvengada, H., & Nahum-Shani, I. (2009). Analyses of Team Performance in a Dynamic Task Environment. Applied Ergonomics, 40(4), 699-706.
Teamwork Dimension (Dependent Variables)Task Type Teamwork
DimensionResponsibilities for operator roles
Aircraft Information Coordinator (AIC) Sensor Operator (SO)
PrimaryInformation Exchange
Request Visual Identification (VID) report and pass it to other teammates.
Evaluate incoming sensor signals.
Correlate sensor signal to a particular aircraft.
Transmit the correlated sensor signal.
Backup Communication Operators did not use speech
channels for communication (not considered).
Operators did not use speech channels for communication (not considered).
PrimaryTeam Initiative/
Leadership
Vector Defensive Counter Air (DCA) within 256 Nautical Miles (NM) from ownship.
Vector DCA outside 20 NM from ownship.
Vector DCA outside danger zones. (Vectoring of DCA is done by changing its speed, course and altitude)
Issue level one warning to hostile aircrafts.
Issue level two warning to hostile aircrafts.
Issue level three warning to hostile aircrafts.
BackupSupporting Behavior
Assign identification to unknown aircrafts.
Assign missiles to hostile aircrafts.
Engage missiles upon hostile aircrafts.
Assign identification to unknown aircrafts.
Error Correction Change the identification of
incorrectly identified aircrafts. Change the identification of
incorrectly identified aircrafts.
Experimental Design
78 participants from a major university (in 39 teams) randomly received one of three training conditions (between-subjects design)
Each participant trained on 6 10-min scenarios and then tested on 2 10-min scenarios (high and low workload)
Three training conditions (no training, team coordination training, task delegation training) used varying in type of presentation (nothing, reading material, or video)
Team Performance Assessment
Environment
Operator Response
Team HITL
Simulation
Truth Maintenance
System
Relative Accuracy
Index (RAI)
Latency Index
(LI)
Information Exchange
Communication
Supporting Behaviour
Team Initiative/ Leadership
Teamwork dimensions
Ontime-Correct( )1Relative Accuracy Index (RAI)
mi
im
The Statistical Model
consider logistic regression as the model for analyzing data in which the dependent variable is a proportion:
that can also be expressed as:
Which is a particular case of the Generalized Linear model, in which linear regression models are extended to the exponential family of distributions that includes both the normal and the binomial distributions.
1
1
exp( )( )
1 exp( )
J
ij jji i i J
ij jj
XE Y p
X
1log
1
Jiij jj
i
pX
p
The Statistical Model (cont.)
Experiment designed to evaluate the effect of a certain type of training on RAI
Since the dependent variable (RAI) is a proportion, the suitable distribution for modeling it, is the binomial distribution
RAI was measured for each one of the two team members, at two stress levels (Low/High), where each team belonged to one of three training groups (NT, TCT, TDT).
For each of the 39 teams, divided randomly among the three types of training, there are four dependent measures of RAI since each team member (SO and AIC) has two outcome measures, corresponding to high and low levels of stress.
Findings
In IE condition TCT training significantly improved performance Negative correlation between AIC and SO under
stress
In SB condition Effects of stress more pronounced Activities involved require longer key sequences
and, under stress, fewer identifications were made
Findings (cont.)
In TI/L condition Absence of DCA activities suggesting limited
cognitive resources Participants in TCT condition outperformed
those in NT or TDT conditions