Development and validation of a KPI - based method and a user - friendly software tool for
resilience – focused measurement of OSH management system performance
KPI – OSH – Tool
Anna Skład (CIOP - PIB, [email protected])
Daniel Podgórski (CIOP-PIB, [email protected])
Tuula Räsänen (FIOH, [email protected])
Riikka Ruotsala (FIOH, [email protected])
4th SAF€RA symposium "Emergence of a New Collaborative Work Programme on Industrial Safety", Athens 12-13 April 2016
KPI-OSH Tool
4th SAF€RA symposium "Emergence of a New Collaborative Work Programme on Industrial Safety", Athens 12-13 April 2016
Project title Development and validation of a KPI-based method and a user-friendly software tool for resilience-focused measurement of OSH management
system performance — KPI-OSH-Tool
Project consortium
Scientific advisory panel
Prof. Eric HOLLNAGEL
Prof. Gerard ZWETSLOOT
Dr. Markku AALTONEN
Mario CALDERÓN
Milestones
4th SAF€RA symposium "Emergence of a New Collaborative Work Programme on Industrial Safety", Athens 12-13 April 2016
1. To define a set of KPIs for measuring performance of OSH MS (at least one KPI per OSH MS component) – KPIs SELECTION AND DESCRIPTION
2. To Develop an e-tool for KPI-based measurement of OSH MS performance – TOOL DEVELOPMENT
3. To conduct pilot implementation and validation of the tool for OSH MS performance measurement - TOOL EVALUATION
KPIs selection
4th SAF€RA symposium "Emergence of a New Collaborative Work Programme on Industrial Safety", Athens 12-13 April 2016
373 PPIs derived from literature sources
120 PPIs
60 PPIs
27 KPIs
Assigning PPIs to 23 OSH MS components (based on the ISO CD 45001 model) and elimination of redundant items
Elimination of indicators which are non-realistic, too complex, over-fancied (Partners and enterprises)
Analytic Hierarchy Process
Podgórski D., Measuring operational performance of OSH management system –
A demonstration of AHP-based selection of leading
key performance indicators, Safety Science 73 (2015)
146–166
4th SAF€RA symposium "Emergence of a New Collaborative Work Programme on Industrial Safety", Athens 12-13 April 2016
List of selected KPIs 1/3
• Audits reports reviewed • Perception of leadership by workers • Workers positively evaluating leadership • Job descriptions complying with OSH • Hazards under control • Hazard investigations reviewed • Risk assessments completed or reviewed • Risk control measures implemented • OSH objectives achieved
4th SAF€RA symposium "Emergence of a New Collaborative Work Programme on Industrial Safety", Athens 12-13 April 2016
• Permits to work complying with OSH • Workers trained according to plan • OSH training plan accomplished • Effectiveness of OSH training • Near-misses reported by workers • Managers’ OSH-related meetings with workers • Workers involved in OSH improvements • Cost equipment failures • Preventive maintenance activities performed
List of selected KPIs 2/3
4th SAF€RA symposium "Emergence of a New Collaborative Work Programme on Industrial Safety", Athens 12-13 April 2016
• Purchase specifications with reference to OSH • OSH related meetings with contractors • Workers trained in emergency response • Monitoring activities completed • Non-conformities identified during audits • Internal audits completed • OSH management system issues reviewed • Corrective / preventive actions completed • Measurement activities with positive results
List of selected KPIs 3/3
4th SAF€RA symposium "Emergence of a New Collaborative Work Programme on Industrial Safety", Athens 12-13 April 2016
Examples of KPIs’ definitions and calculation formulas (1/3)
KPI Definition KPI Calculation formula
Parameters’ definitions
Workers’ perception of management leadership and commitment to OSH (rating resulting from the survey)
N LPj = (Σ Xi) / N i=1 M LP = (Σ wj • LPj) / M j=1
Xi - Evaluation of an individual j-th element of management leadership and commitment to OSH improvement made by a single worker - expressed in number, e.g. in a Likert scale; M - No. of elements of management leadership and commitment included in a survey and evaluated by workers; N - No. of workers evaluating individual elements of management leadership and commitment to OSH improvement; wj - The weight of j-th element of management leadership and commitment to OSH improvement according to the theoretical model being applied in a survey methodology.
LP1 Perception of leadership by workers
Examples of KPIs’ definitions and calculation formulas (2/3)
4th SAF€RA symposium "Emergence of a New Collaborative Work Programme on Industrial Safety", Athens 12-13 April 2016
KPI Definition KPI Calculation formula Parameters’ definitions
Percentage of hazard investigations reviewed according to schedule in a given reporting period (against the total number of identified hazards) /
HRi = Xi / Yi • 100% N Σ Xi i=1 HR = -------- * 100% N Σ Yi i=1
Xi - No. of revisions of results of hazard investigations performed in i-th unit according to the established plan of the revisions; Yi - Total no. of hazards identified and recorded (e.g. in hazard inventory) in i-th unit, with regard to which respective investigations had been carried out and documented, followed by the implementation of appropriate control measures (if necessary); N - No. of units conducting revisions of hazard investigation results.
HR Hazard investigations reviewed
4th SAF€RA symposium "Emergence of a New Collaborative Work Programme on Industrial Safety", Athens 12-13 April 2016
KPI Definition KPI Calculation formula
Parameters’ definitions
Percentage of workers with valid training in emergency preparedness, response and coordination per unit (against total number of workers in a given reporting period)
ERi = Xi / Yi • 100%
N Σ Xi i=1 ER = -------- * 100% N Σ Yi i=1
Xi - No. of workers in i-th unit holding within a given reporting period valid certificates of introductory and/or refresher training courses on emergency preparedness, response and coordination actions; Yi - Total no. of workers employed in i-th unit (i.e. no. of persons doing work or work-related activities, regularly or temporarily, under the organization’s direct and indirect control); N -No. of organisational units, in which workers are obliged to maintain and improve their competencies in emergency preparedness, response and coordination actions.
Examples of KPIs’ definitions and calculation formulas (3/3)
ER Workers trained in emergency response
Tool development
A standalone software based on MS Excel and Word
Functionalities:
Selecting KPIs (out of 27 predefined items)
Tailoring selected KPIs to the user’s needs
Calculation and graphical presentation of KPIs values (bar charts and speedometers)
Creating additional KPIs
4th SAF€RA symposium "Emergence of a New Collaborative Work Programme on Industrial Safety", Athens 12-13 April 2016
4th SAF€RA symposium "Emergence of a New Collaborative Work Programme on Industrial Safety", Athens 12-13 April 2016
Tool operation
User interface – example screens (1/5)
4th SAF€RA symposium "Emergence of a New Collaborative Work Programme on Industrial Safety", Athens 12-13 April 2016
Tool operation
User interface – example screens (2/5)
4th SAF€RA symposium "Emergence of a New Collaborative Work Programme on Industrial Safety", Athens 12-13 April 2016
Tool operation
User interface – example screens (3/5)
4th SAF€RA symposium "Emergence of a New Collaborative Work Programme on Industrial Safety", Athens 12-13 April 2016
Tool operation
User interface – example screens (4/5)
4th SAF€RA symposium "Emergence of a New Collaborative Work Programme on Industrial Safety", Athens 12-13 April 2016
Tool operation
User interface – example screens (5/5)
4th SAF€RA symposium "Emergence of a New Collaborative Work Programme on Industrial Safety", Athens 12-13 April 2016
HR: 83,3%
90% 100%
60%
0
50
100
150
Unit 1 Unit 2 Unit 3
73,3%
76,7%
83,3%
68707274767880828486
Quarter 4 2015 Quarter 1 2016 Quarter 2 2016
80% 80%
90%
70
80
90
100
Quarter 42015
Quarter 12016
Quarter 22016
100% 100% 100%
0
50
100
150
Quarter 42015
Quarter 12016
Quarter 22016
40% 50% 60%
0
50
100
Quarter 42015
Quarter 12016
Quarter 22016
Tool operation
Example diagrams
TREND UNIT 1
TREND UNIT 2
TREND UNIT 3
KPI CURRENT VALUE
KPI TREND
SUB-KPIs CURRENT VALUES
Tool operation
4th SAF€RA symposium "Emergence of a New Collaborative Work Programme on Industrial Safety", Athens 12-13 April 2016
Tool evaluation
4th SAF€RA symposium "Emergence of a New Collaborative Work Programme on Industrial Safety", Athens 12-13 April 2016
System Usability Scale - ten-item questionnaire based on the 5-point Likert scale enabling subjective evaluation of software usability
Methodology 1/2
SUS Items: I think that I would like to use this system frequently
I found the system unnecessarily complex
I thought the system was easy to use
I think that I would need the support of a technical person to be able to use this system
I found the various functions in this system were well integrated
I thought there was too much inconsistency in this system
I would imagine that most people would learn to use this system very quickly
I found the system very cumbersome to use
I felt very confident using the system
I needed to learn a lot of things before I could get going with this system
4th SAF€RA symposium "Emergence of a New Collaborative Work Programme on Industrial Safety", Athens 12-13 April 2016
KPI OSH Tool – User Interview
Tool evaluation
Interview A – Introductory Phase
Interview B – Implementation Phase
Interview C – Closing Phase
Respondent background information Company background and measuring OHS OSH-related KPI measuring tools
Installation of the tool KPI selection Defining additional KPIs Report generation Input configuration
Using the KPI OSH Tool Utilizing KPI data Taking action Utilizing KPI data
Methodology 2/2
4th SAF€RA symposium "Emergence of a New Collaborative Work Programme on Industrial Safety", Athens 12-13 April 2016
What have we learned? Feedback from enterprises - introductory phase
• Enterprises measure OSH KPIs :
OSH lagging performance indicators (accident rate, absenteeism rate, occupational diseases rates) OSH leading performance indicators (safety culture level, no. of near-misses , no. of identified hazards)
• Enterprises use on average 5-6 OSH KPIs • Enterprises use e-tools to calculate OSH KPIs
• E-tools used in enterprises include : MS Excel and Power Point, SharePoint and self-made special applications.
THERE ARE NO DEDICATED E-TOOLS AVAILABLE ON THE MARKET.
Tool evaluation
4th SAF€RA symposium "Emergence of a New Collaborative Work Programme on Industrial Safety", Athens 12-13 April 2016
Tool evaluation
What have we learned? Feedback from enterprises on usability
KPI-OSH Tool usability evaluation
Easy to use
Most people would learn to use this system very quickly
System functions poorly integrated Users were did not feel confident using the system
4th SAF€RA symposium "Emergence of a New Collaborative Work Programme on Industrial Safety", Athens 12-13 April 2016
What have we learned? Feedback from enterprises - implementation phase 1/2
Areas of potential improvement:
• Installation process should be simplified
• Additional KPIs tab should be more user – friendly
Tool evaluation
4th SAF€RA symposium "Emergence of a New Collaborative Work Programme on Industrial Safety", Athens 12-13 April 2016
Areas of further development of the tool:
• Dashboard functionality
• KPI value trend extrapolation
• Source data available in the tool (e.g. accident register)
• Flexibility to modify KPIs definitions
• Flexibility to aggregate indicators in time
What have we learned? Feedback from enterprises - implementation phase 2/2
Tool evaluation
Why KPIs?
KPIs:
• Can measure wide range of aspects
• Are objective measures (it is difficult to manipulate numbers)
• Allow precise evaluation
• Enable to follow progress
but…
4th SAF€RA symposium "Emergence of a New Collaborative Work Programme on Industrial Safety", Athens 12-13 April 2016
KPIs – the other side of the coin
Companies should not fall into the trap of <<if you can’t measure what you want, want what you can measure>>.
Kaplan R.S., Norton D.P.,1996, Translating Strategy into Action. The Balanced Scorecard, Harvard Business School Press, Boston, Massachusetts, p.100.
• Keep in mind that your overarching goal is to prevent accidents and occupational diseases
• Read the information behind KPIs values instead of putting your faith in numbers
• Interpret KPIs values to make decisions rather than simply command people to achieve KPI-based goals
4th SAF€RA symposium "Emergence of a New Collaborative Work Programme on Industrial Safety", Athens 12-13 April 2016
CONCLUSIONS
• Companies need to measure their OSH performance
• OSH performance measurement can be facilitated by the use of e-tools
• KPI-OSH Tool can be an effective tool to support OSH performance measurement
4th SAF€RA symposium "Emergence of a New Collaborative Work Programme on Industrial Safety", Athens 12-13 April 2016