Development and validation of a KPI - based method and a

Development and validation of a KPI - based method and a user - friendly software tool for

resilience – focused measurement of OSH management system performance

KPI – OSH – Tool

Anna Skład (CIOP - PIB, [email protected])

Daniel Podgórski (CIOP-PIB, [email protected])

Tuula Räsänen (FIOH, [email protected])

Riikka Ruotsala (FIOH, [email protected])

4th SAF€RA symposium "Emergence of a New Collaborative Work Programme on Industrial Safety", Athens 12-13 April 2016

mailto:[email protected]




KPI-OSH Tool


Project title Development and validation of a KPI-based method and a user-friendly software tool for resilience-focused measurement of OSH management

system performance — KPI-OSH-Tool

Project consortium

Scientific advisory panel

Prof. Eric HOLLNAGEL

Prof. Gerard ZWETSLOOT

Dr. Markku AALTONEN

Mario CALDERÓN

Milestones


1. To define a set of KPIs for measuring performance of OSH MS (at least one KPI per OSH MS component) – KPIs SELECTION AND DESCRIPTION

2. To Develop an e-tool for KPI-based measurement of OSH MS performance – TOOL DEVELOPMENT

3. To conduct pilot implementation and validation of the tool for OSH MS performance measurement - TOOL EVALUATION

KPIs selection


373 PPIs derived from literature sources

120 PPIs

60 PPIs

27 KPIs

Assigning PPIs to 23 OSH MS components (based on the ISO CD 45001 model) and elimination of redundant items

Elimination of indicators which are non-realistic, too complex, over-fancied (Partners and enterprises)

Analytic Hierarchy Process

Podgórski D., Measuring operational performance of OSH management system –

A demonstration of AHP-based selection of leading

key performance indicators, Safety Science 73 (2015)

146–166


List of selected KPIs 1/3

• Audits reports reviewed • Perception of leadership by workers • Workers positively evaluating leadership • Job descriptions complying with OSH • Hazards under control • Hazard investigations reviewed • Risk assessments completed or reviewed • Risk control measures implemented • OSH objectives achieved


• Permits to work complying with OSH • Workers trained according to plan • OSH training plan accomplished • Effectiveness of OSH training • Near-misses reported by workers • Managers’ OSH-related meetings with workers • Workers involved in OSH improvements • Cost equipment failures • Preventive maintenance activities performed



• Purchase specifications with reference to OSH • OSH related meetings with contractors • Workers trained in emergency response • Monitoring activities completed • Non-conformities identified during audits • Internal audits completed • OSH management system issues reviewed • Corrective / preventive actions completed • Measurement activities with positive results



Examples of KPIs’ definitions and calculation formulas (1/3)

KPI Definition KPI Calculation formula

Parameters’ definitions

Workers’ perception of management leadership and commitment to OSH (rating resulting from the survey)

N LPj = (Σ Xi) / N i=1 M LP = (Σ wj • LPj) / M j=1

Xi - Evaluation of an individual j-th element of management leadership and commitment to OSH improvement made by a single worker - expressed in number, e.g. in a Likert scale; M - No. of elements of management leadership and commitment included in a survey and evaluated by workers; N - No. of workers evaluating individual elements of management leadership and commitment to OSH improvement; wj - The weight of j-th element of management leadership and commitment to OSH improvement according to the theoretical model being applied in a survey methodology.

LP1 Perception of leadership by workers



KPI Definition KPI Calculation formula Parameters’ definitions

Percentage of hazard investigations reviewed according to schedule in a given reporting period (against the total number of identified hazards) /

HRi = Xi / Yi • 100% N Σ Xi i=1 HR = -------- * 100% N Σ Yi i=1

Xi - No. of revisions of results of hazard investigations performed in i-th unit according to the established plan of the revisions; Yi - Total no. of hazards identified and recorded (e.g. in hazard inventory) in i-th unit, with regard to which respective investigations had been carried out and documented, followed by the implementation of appropriate control measures (if necessary); N - No. of units conducting revisions of hazard investigation results.

HR Hazard investigations reviewed


KPI Definition KPI Calculation formula

Parameters’ definitions

Percentage of workers with valid training in emergency preparedness, response and coordination per unit (against total number of workers in a given reporting period)

ERi = Xi / Yi • 100%

N Σ Xi i=1 ER = -------- * 100% N Σ Yi i=1

Xi - No. of workers in i-th unit holding within a given reporting period valid certificates of introductory and/or refresher training courses on emergency preparedness, response and coordination actions; Yi - Total no. of workers employed in i-th unit (i.e. no. of persons doing work or work-related activities, regularly or temporarily, under the organization’s direct and indirect control); N -No. of organisational units, in which workers are obliged to maintain and improve their competencies in emergency preparedness, response and coordination actions.


ER Workers trained in emergency response

Tool development

A standalone software based on MS Excel and Word

Functionalities:

Selecting KPIs (out of 27 predefined items)

Tailoring selected KPIs to the user’s needs

Calculation and graphical presentation of KPIs values (bar charts and speedometers)

Creating additional KPIs



Tool operation

User interface – example screens (1/5)


Tool operation



Tool operation



Tool operation



Tool operation



HR: 83,3%

90% 100%

60%

0

50

100

150

Unit 1 Unit 2 Unit 3

73,3%

76,7%

83,3%

68707274767880828486

Quarter 4 2015 Quarter 1 2016 Quarter 2 2016

80% 80%

90%

70

80

90

100

Quarter 42015

Quarter 12016

Quarter 22016

100% 100% 100%

0

50

100

150

Quarter 42015

Quarter 12016

Quarter 22016

40% 50% 60%

0

50

100

Quarter 42015

Quarter 12016

Quarter 22016

Tool operation

Example diagrams

TREND UNIT 1

TREND UNIT 2

TREND UNIT 3

KPI CURRENT VALUE

KPI TREND

SUB-KPIs CURRENT VALUES

Tool operation


Tool evaluation


System Usability Scale - ten-item questionnaire based on the 5-point Likert scale enabling subjective evaluation of software usability

Methodology 1/2

SUS Items: I think that I would like to use this system frequently

I found the system unnecessarily complex

I thought the system was easy to use

I think that I would need the support of a technical person to be able to use this system

I found the various functions in this system were well integrated

I thought there was too much inconsistency in this system

I would imagine that most people would learn to use this system very quickly

I found the system very cumbersome to use

I felt very confident using the system

I needed to learn a lot of things before I could get going with this system


KPI OSH Tool – User Interview

Tool evaluation

Interview A – Introductory Phase

Interview B – Implementation Phase

Interview C – Closing Phase

Respondent background information Company background and measuring OHS OSH-related KPI measuring tools

Installation of the tool KPI selection Defining additional KPIs Report generation Input configuration

Using the KPI OSH Tool Utilizing KPI data Taking action Utilizing KPI data

Methodology 2/2


What have we learned? Feedback from enterprises - introductory phase

• Enterprises measure OSH KPIs :

OSH lagging performance indicators (accident rate, absenteeism rate, occupational diseases rates) OSH leading performance indicators (safety culture level, no. of near-misses , no. of identified hazards)

• Enterprises use on average 5-6 OSH KPIs • Enterprises use e-tools to calculate OSH KPIs

• E-tools used in enterprises include : MS Excel and Power Point, SharePoint and self-made special applications.

THERE ARE NO DEDICATED E-TOOLS AVAILABLE ON THE MARKET.

Tool evaluation


Tool evaluation

What have we learned? Feedback from enterprises on usability

KPI-OSH Tool usability evaluation

Easy to use

Most people would learn to use this system very quickly

System functions poorly integrated Users were did not feel confident using the system


What have we learned? Feedback from enterprises - implementation phase 1/2

Areas of potential improvement:

• Installation process should be simplified

• Additional KPIs tab should be more user – friendly

Tool evaluation


Areas of further development of the tool:

• Dashboard functionality

• KPI value trend extrapolation

• Source data available in the tool (e.g. accident register)

• Flexibility to modify KPIs definitions

• Flexibility to aggregate indicators in time

What have we learned? Feedback from enterprises - implementation phase 2/2

Tool evaluation

Why KPIs?

KPIs:

• Can measure wide range of aspects

• Are objective measures (it is difficult to manipulate numbers)

• Allow precise evaluation

• Enable to follow progress

but…


KPIs – the other side of the coin

Companies should not fall into the trap of <<if you can’t measure what you want, want what you can measure>>.

Kaplan R.S., Norton D.P.,1996, Translating Strategy into Action. The Balanced Scorecard, Harvard Business School Press, Boston, Massachusetts, p.100.

• Keep in mind that your overarching goal is to prevent accidents and occupational diseases

• Read the information behind KPIs values instead of putting your faith in numbers

• Interpret KPIs values to make decisions rather than simply command people to achieve KPI-based goals


CONCLUSIONS

• Companies need to measure their OSH performance

• OSH performance measurement can be facilitated by the use of e-tools

• KPI-OSH Tool can be an effective tool to support OSH performance measurement


Thank you for your attention


Documents

Development and validation of a KPI - based method and a