Improving Safety Performance Through Measurement
Improving Safety Performance Through Measurement
Mike ThomasManaging Director Prof. Andrew Hale
DirectorIOSH London Health& Safety Group
17 March 2008
OverviewOverview
Key questions
What is the role of performance measurement in a Safety Management System (SMS)? (“What gets measured gets done”, Drucker and HSE in HSG65)
How can measurement be used to improve performance and drive continual improvement?
How can we select appropriate key performance indicators (KPIs)?
How can we develop meaningful targets?
How can we promote performance measurement by line managers and link this to audits?
A Success Story: Corus NLA Success Story: Corus NL
Year -3 -2 -1 0 +1 +2
LTI Frequency/1000 man
17.3 16.1 14.3 7.9 0.9 1.9
Reported dangerous situations
186 237 642 1401 1643 1252
Maintenance department of an integrated steel works, employing about 1100 used an intervention driven by KPIs to improve performance.
Improvement driven by KPI’s for managers: • Compulsory + free choice, ‘proactive’ + ‘reactive’.• Targets/KPI + score (1-10 with 6 for hitting target) • KPIs given different weights & totalled for overall score/period• Reported on a ‘dashboard’ every 6 weeks
STOP-GO cards for workforce – dynamic task risk assessment
Key Performance IndicatorsKey Performance Indicators
Output: LTI frequency + Recordable frequency
Compulsory:
Reporting of dangerous situations (0.5pppy)
Dealing with dangerous situation reports (80%)
Toolbox meetings (including safety) (80%)
Choice (at least 1 of 4):
Behaviour observation rounds (80%)
Safety communication rounds (80%)
‘5 S’ housekeeping inspections (80%)
Risk assessment + plan of action (80%)
OHSAS 18001OHSAS 18001
Continual improvement
Checking and corrective action
OH&S Policy
Planning
Implementationand operation
Management Review
MeasurementMeasurement
Within 18001
“Checking and corrective action”
Includes
Measurement of proactive data
Measurement of reactive data
Internal audit (independent)
Reactive and proactive are key issues
Risk assessment & controlRisk assessment & control
Review & improvement
Risk management system Audit
EvaluationDirect risk controls:
Hard-, soft- & liveware
Primary processes
Inspection
Incident analysis
Risk assessment
Known risk
Comparison
Unknown riskReview RIE system
Types of indicatorTypes of indicator
Damage, injury, loss
Precursors – for each scenario, different precursors (incidents, leaks, breakdowns)
Re-design
Choice and design of (sub)system
Normal situation
with in-built hazards
Recovery
Deviations from normal situation
Loss of control(release of energy
+ exposure
Transmission
Damage process
Stabilisation
Elimination of hazard
Hazard control measures
Detection & recovery
Escape
Secondary safety
Rescue, damage limitation, treatment
Reporting
Learning
Choice & design of prevention &
control measures
Accident Deviation Model
For each hazard, different deviation scenarios, with different control measures, failing in different ways
LTI’s don’t predict disastersLTI’s don’t predict disasters
The lesson of Texas City:
BP became distracted from efforts to control process safety because they were lulled into a false sense of security by their very excellent control of lost time injuries.
The fallacy of Bird’s interpretation of Heinrich’s triangle
Heinrich’s thesisHeinrich’s thesis
Reasoning from top down
Each scenario or accident type apart
Average ratio over scenarios
1
29
300
Disabling Injury
Minor Injury
No injury
From Bird onwardsFrom Bird onwards
Overall statistics of a department, factory, or country
Different categories of severity
1
100
500
Property damage
(Bird 1966)
Disabling Injury
Minor Injury
Types of indicatorTypes of indicator
Damage, injury, loss
Precursors – for each scenario, different precursors (incidents, leaks, breakdowns)
Presence & readiness of risk control measures (hardware, software, people’s behaviour)
Management system processes to deliver the preventive measures
Resources and controls in the SMSResources and controls in the SMS
- Competence of staff- Availability of manpower- Commitment to safe operations & conflict resolution- Communication within and between teams- Procedures, goals & rules- Hardware & software
- Selection & training- Manpower planning- Incentives, supervision, appraisal, culture, management priority- Handover, briefing, communication channels- Task/policy analysis & design- Design, layout, maintenance
These processes are the ones to be audited
Problems with AuditingProblems with Auditing
Seen as a panacea
“Bolted on” – imposed from outside
May not match policy & own management system: independent of management
Passive process for manager
“Economical with the truth”
Wait for next audit
No ownership
Types of indicatorTypes of indicator
Damage, injury, loss
Precursors – for each scenario, different precursors (incidents, leaks, breakdowns)
Presence & condition of risk control measures (hardware, software, people’s behaviour)
Management system processes to deliver the preventive measures
Attitudes and values (culture) to use the SMS processes and the risk control measures
Safety climate/culture measurementSafety climate/culture measurement
Existing climate questionnaires: limited validation (TRIPOD is one exception) Interpretation is still more an art than a science – lack
of clarity over underlying models of culture Gap between measuring (profile) and improving
Safety culture maturity scales (Hearts & Minds): Similar criticisms Individual completion followed by group discussion
generates much useful debate
Improving PerformanceImproving Performance
Once you are clear on meaning & objectives
Look at measurement and improvement
Three key issues
How to define performance in measurable terms?
How can performance be measured?
How to extend to target setting and to drive improved performance?
How is Improvement Achieved?How is Improvement Achieved?
“What gets measured gets done”
Apply sound management techniques to safety
Performance standards - what people must do Who is responsible? – whose KPI? What are they responsible for? – tasks, processes When should the work be done? - plans What is the expected result? – (intermediate) outputs
Set targets and measure performance in terms meaningful to each individual
Criteria for performance measuresCriteria for performance measures
Validity
Reliability
Representativeness
Sensitivity
Openness to bias
Cost-effectiveness
NB. Exposure measures
Typical targets in the pastTypical targets in the past
“eliminate lost time (or all) accidents”
“reduce all accidents to n or by x% per annum”
“reduce number of days lost to n or by x%”
“reduce cost of claims or other losses by y%”
“eliminate notices as a result of enforcement action”
Obvious approach = what we want! Commendable, but typically, reactive
Accidents as performance measureAccidents as performance measure
Validity
Reliability
Representativeness
Sensitivity
Reporting bias
Cost-effectiveness
PM Exposure measure
Very high
High, but absence serious
Good if enough
Low for good organisation
Strong for minor
High cost of analysis
/man or /manhour
Problems with Reactive TargetsProblems with Reactive Targets
Problems - well recognised?
Not helpful to managers!
They accept the value and sense but … What must they do?
“Promotes” under-reporting and “manipulation” of outcome
What if zero accidents already?
Past performance may be a poor predictor of the future
Accident RatesAccident Rates
"A low accident or incident rate over a period of years is no guarantee that risks are being effectively controlled.This is particularly so in organisations involved in major hazard activities, where the probability of an accident may be low but where the consequences could be extremely serious.In this type of organisation, the historical incidence of reported accidents alone can be an unreliable indicator of safety or environmental performance and can lead to complacency."
Guidance on COMAH Regulations, UK
Reactive TargetsReactive Targets
Why still set? Easy option that sounds good! Mirrors UK Government’s targets Accident reduction IS the desired output of the SMS
Are Reactive “Targets” Still Valid? At a high level within the organisation as “aim” or
“vision” But NOT as targets for managers Need to move to better data: step 1
Minor injuries and health effects Near misses (surprises) or dangerous situations
Way Forward?Way Forward?
Aim
Reduce accidents but numbers of accidents are NOT used as exclusive targets for managers
Move to proactive measures
Set targets on proactive measures of performance
Compare with quality management
Contrast: Safety and QualityContrast: Safety and Quality
ISO approach to quality as:
An essential feature
Not an optional extra
Emphasise:
managing quality in, not inspecting defects out get the management processes right, but still measure defects
Apply this philosophy to safety
What is Proactive Measurement?What is Proactive Measurement?
Traditional measures (input measures): e.g. PPE being used Guards in place Documents reviewed and updated All employees trained Toolbox meetings held Maintenance conducted to plan
More dynamic measures (input + intermediate) Hazards identified and put right Competence tested & used Behaviour observed, discussed & improved
Planning and Target SettingPlanning and Target Setting
If you accept the philosophy, how can measures be translated into targets?
BS8800 approach
Annex C (BS 8800:1996)
Simplified version (BS 8800:2004, Annex D)
Planning and Implementing
Planning for SafetyPlanning for Safety
Measure Baseline Data
Measure Baseline Data
Devise Plan(Management Programme)
Devise Plan(Management Programme)
Topics withQuestions as Targets
Topics withQuestions as Targets
IMPLEMENT PLANIMPLEMENT PLAN
AimAim
ObjectiveObjective
Define Outcome Indicator
Define Outcome Indicator
Measuring SuccessMeasuring Success
IMPLEMENT PLANIMPLEMENT PLAN
Measure Outcome Data
Measure Outcome Data
Achieving Objective ?Achieving Objective ? NoNo
YesYes
Take Corrective Action as Necessary
Take Corrective Action as Necessary
Continue ProcessContinue Process
Measure Compliance with Programme (Targets)
Measure Compliance with Programme (Targets)
Meeting Targets ?Meeting Targets ?
YesYes
Combining DataCombining Data
Compliance Monitoring Data (Management Programme)
Outcome Data (Objective)
Improving Not Improving
Improving Not Improving
Why questions posed in plan?Why questions posed in plan?
Forces clear logical thought
Define the programme
Define the targets
Questions are a blue print for the managers
Questions can be used to measure success
“Correct” answer is “yes”
Link to audit (e.g. CHASE)
Proactive Monitoring and AuditProactive Monitoring and Audit
HASTAM’s CHASE system
Questions developed by planning process
Used by managers to measure performance
Used by specialists for audit
Audit by verification of
Managers’ answers
Recommended remedial action
Reinforces ownership by managers
ConclusionConclusion
Explained
Need for and effective means of setting realistic targets
Targets NOT based on “reactive” measures
Based on proactive measures of system
However, no conflict with reactive aims
Targets provide
Clear implementation mechanism for managers
Effective measurement of improved performance