Using the SMART Method to Assess SMS in Aviation · Using the SMART Method to Assess SMS in Aviation Ronald J. Heslegrave, ... regard to all aspects of the evaluation. ... Q38 Q39

Using the SMART Method Using the SMART Method to Assess SMS in Aviationto Assess SMS in Aviation

Ronald J. Ronald J. HeslegraveHeslegrave, Ph.D., Ph.D.University Health Network & University Health Network &

University of TorontoUniversity of Toronto

CASS 2007 CASS 2007 -- COUNTING THE ACCIDENTS COUNTING THE ACCIDENTS YOU DON’T HAVE…YOU DON’T HAVE…

May 1, 2007 May 1, 2007 –– GatineauGatineau QCQC

Successful Implementation of a Safety Management System or Policies Need to:

Develop a Vision for the Organization Develop a Vision for the Organization –– without a without a culture it will failculture it will failDevelop Policies and Procedures designed to Develop Policies and Procedures designed to Support a Safety Culture and the VisionSupport a Safety Culture and the VisionImplement Policies and Procedures through Implement Policies and Procedures through Training, Organizational Strategies, Operational Training, Organizational Strategies, Operational Strategies, Enabling ToolsStrategies, Enabling ToolsDevelop Assessment and Evaluation MechanismsDevelop Assessment and Evaluation MechanismsEvaluate what you have put in placeEvaluate what you have put in place

The Question sometimes is…

How do you do that?How do you do that?

One Answer …AdherenceAdherence

to Recognized Standards or Adopted to Recognized Standards or Adopted Regulations where there is consensus Regulations where there is consensus

that such Standards or Regulations that such Standards or Regulations reflect an appropriate reflect an appropriate Vision and PolicyVision and Policy

for example, ISO standardsfor example, ISO standards

Another Answer …

EvaluationEvaluationof objective measures where most agree of objective measures where most agree that the measures reflect the Vision and that the measures reflect the Vision and

PolicyPolicy

for example, reduced accidents, incidents for example, reduced accidents, incidents or near missesor near misses

Near MissesNear Misses

Or WorseOr Worse

Another Answer …

Accept Accept expert testimony and evaluation that expert testimony and evaluation that

the Vision and Policy have been the Vision and Policy have been achieved in good measureachieved in good measure

for example, Expert External Auditfor example, Expert External Audit

Auditors and Regulators are sometimes Auditors and Regulators are sometimes viewed differently than intendedviewed differently than intended

It depends on your perception?It depends on your perception?

Auditor and Henchmen may be viewed NegativelyAuditor and Henchmen may be viewed Negatively

The Audit Process can be viewed The Audit Process can be viewed as Confrontationalas Confrontational

We should strive for a Reasonable Dialogue with We should strive for a Reasonable Dialogue with Enlightened Views between Auditors and OrganizationsEnlightened Views between Auditors and Organizations

Is there a simple, effective and multidimensional approach to

evaluating SMS in your organization that is value-

added?

Perhaps?

Today I will introduce the SMARTSMARTapproach to safety management

and evaluation?

While this approach has been investigated in the civil Aviation

sector, it should be transferable to other organizations and systems

Safety Management Audit & Review Tool- SMART method

Evaluates the Understanding ofEvaluates the Understanding ofFundamental System Elements of a Fundamental System Elements of a

Safety Management SystemSafety Management System

Assesses whether the Assesses whether the Safety Management System has Safety Management System has Penetrated the Organization at Penetrated the Organization at

all levelsall levels

Evaluates Supporting Systems:Evaluates Supporting Systems:Training and Measurement, Training and Measurement,

Assessment and Evaluation ToolsAssessment and Evaluation Tools

SMARTSMART method method evaluates key evaluates key elements of safety elements of safety systems or policiessystems or policies

Safety Systems and Policies are Multidimensional -Evaluations Need to be as well

Assess the reasonableness of what is proposedAssess the reasonableness of what is proposedAssess how well it is implemented Assess how well it is implemented Assess whether it is effective Assess whether it is effective Develop a system for continuous quality improvement Develop a system for continuous quality improvement to monitor and promote changeto monitor and promote changeFeedback the evaluation to the Organization to Feedback the evaluation to the Organization to provide continuous quality improvementprovide continuous quality improvement

BASIC EVALUATION APPROACH OF THE SMARTSMART METHOD

Fundamentally at least 3 Criteria or Dimensions:Fundamentally at least 3 Criteria or Dimensions:Understanding of the Policy and ProceduresUnderstanding of the Policy and ProceduresQuality of Implementation of the SystemQuality of Implementation of the SystemPenetrance of Safety Policy and Procedures Penetrance of Safety Policy and Procedures throughout the organizationthroughout the organization

SMART METHOD1)1) Identify the goals of the systemIdentify the goals of the system2)2) Develop a set of tools designed to target the primary Develop a set of tools designed to target the primary

requirements of that systemrequirements of that system3)3) Collect objective and subjective data from relevant Collect objective and subjective data from relevant

stakeholders across the hierarchy of the organizationstakeholders across the hierarchy of the organization4)4) Blindly evaluate and analyze these data in a Blindly evaluate and analyze these data in a

multidimensional contextmultidimensional context5)5) Provide feedback for ongoing quality improvement Provide feedback for ongoing quality improvement

for the organizationfor the organization

Key to the approach is to Quantitatively Assess and

Evaluate the Data collected on a Multidimensional Basis

Can this SMART method be implemented Practically, Easily and Successfully?

You be the Judge?

Recently the SMART method was validated in a major corporation in the Aviation Community with national operations to study the successful implementation of their Safety Management Program and Policies

An Example of the Application of An Example of the Application of the SMART Methodthe SMART Method

SMART APPROACHIdentified Key Policies and Objectives of the Identified Key Policies and Objectives of the Safety Management ProgramSafety Management ProgramDeveloped an Assessment TeamDeveloped an Assessment TeamDeveloped a Survey to Assess Safety Developed a Survey to Assess Safety ManagementManagementImplemented a multidimensional approach to the Implemented a multidimensional approach to the evaluation of written and interview responses to evaluation of written and interview responses to the Survey Toolthe Survey Tool

Expert Team AnalysisAn Audit Team was established with An Audit Team was established with varying expertise including Operational varying expertise including Operational Systems, Safety Management Operations, Systems, Safety Management Operations, Specific Content Experience, and Scientific Specific Content Experience, and Scientific ExpertiseExpertisePrior to assessing the data collected, a Prior to assessing the data collected, a decision was made to “blind” the evaluators decision was made to “blind” the evaluators to each others scoring of responses with to each others scoring of responses with regard to all aspects of the evaluationregard to all aspects of the evaluation

Develop a Survey ToolQuestions were grouped into Common Safety Questions were grouped into Common Safety Dimensions Dimensions Questions were developed along 5 Dimensions Questions were developed along 5 Dimensions representing the Continuum of Safety Management representing the Continuum of Safety Management Systems Systems –– Fundamental Program ElementsFundamental Program Elements–– Implementation of Fundamental Program ElementsImplementation of Fundamental Program Elements–– Training Training –– Assessment and Evaluation Strategies Assessment and Evaluation Strategies –– Measurement SystemsMeasurement Systems

Collect Data from OrganizationSubmit the Survey Tool to the Organization for a Submit the Survey Tool to the Organization for a Formal Written ResponseFormal Written ResponseBased on the Survey, Interview Key Stakeholders Based on the Survey, Interview Key Stakeholders across the Hierarchy of the Organizationacross the Hierarchy of the Organization

Executive Executive Senior ManagementSenior ManagementLine ManagersLine ManagersSafety ProfessionalsSafety ProfessionalsLine PersonnelLine Personnel

Quality of the ResponseQuality of the Response -- How well do the How well do the answers address the questions asked?answers address the questions asked?Policy UnderstandingPolicy Understanding -- How well do the How well do the answers demonstrate an understanding of answers demonstrate an understanding of the Safety Management Policies?the Safety Management Policies?Penetrance Penetrance -- How well does the answer How well does the answer demonstrate the implementation of the demonstrate the implementation of the Safety Management Policies throughout Safety Management Policies throughout the organization?the organization?

Develop a Multidimensional Evaluation StrategyDevelop a Multidimensional Evaluation Strategy

Evaluation ProcessEach Audit Team Member Independently Each Audit Team Member Independently and Blindly assessed each written and and Blindly assessed each written and interview response interview response

All ratings were made on a 5All ratings were made on a 5--point point LikertLikertScale:Scale:–– 5= Good and Full Response5= Good and Full Response–– 3= Reasonable Response3= Reasonable Response–– 1= Inadequate or Low Response1= Inadequate or Low Response

Organizational FeedbackResults are presented in the following fashion:Results are presented in the following fashion:–– Overall ratings of Quality, Policy, and Overall ratings of Quality, Policy, and PenetrancePenetrance

Dimensions Dimensions –– Ratings on a question by question basis to Ratings on a question by question basis to

identify areas of strength and opportunity for identify areas of strength and opportunity for improvement in the Organization improvement in the Organization

–– Summarized data by common dimensions based Summarized data by common dimensions based on the questionson the questions

Findings

Overall Ratings of Audit Team

1

2

3

4

5

WrittenQuestions

InterviewQuestions

Average

QUALITY POLICY PENETRANCE

High

Low

Reasonable

Overall Ratings of Audit Team for Written Responses

1

2

3

4

5


High

Low

Reasonable

Opportunities for Opportunities for ImprovementImprovement

Ratings of Audit Team for Ratings of Audit Team for Written Responses who were blinded in their Written Responses who were blinded in their

assessmentsassessments

1

2

3

4

5

Auditor 1 Auditor 2 Auditor 3


High

Low

Reasonable

It is often said that:

The Devil is in the DetailsThe Devil is in the Details

In this case…

The Added Value is in the DetailsThe Added Value is in the Details

Individual Question Results

0

1

2

3

4

5

Q1 Q2 Q3 Q5 Q6 Q7 Q8 Q9 Q10


High

Low

Reasonable

0

1

2

3

4

5

Q11 Q12 Q13 Q14 Q15 Q16 Q17 Q18 Q19


High

Low

Reasonable


0

1

2

3

4

5

Q20 Q21 Q22 Q23 Q24 Q25 Q26 Q27 Q28


High

Low

Reasonable


0

1

2

3

4

5

Q29 Q30 Q31 Q32 Q33 Q34 Q35 Q36 Q37


High

Low

Reasonable


0

1

2

3

4

5

Q38 Q39 Q40 Q41 Q42 Q43 Q44 Q45 Q46


High

Low

Reasonable


The SMARTSMART Process:Analysis of Individual Questions

The results of individual questions reveal areas The results of individual questions reveal areas of excellence as well as specific opportunities for of excellence as well as specific opportunities for improvement:improvement:

Questions 4 and 5 show companyQuestions 4 and 5 show company--wide wide excellenceexcellence

SMART SMART -- IndividualIndividual QuestionsQuestions

0

1

2

3

4

5

Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9


High

Low

Reasonable


Questions 8 and 9 show strong enabling Questions 8 and 9 show strong enabling tools and goodtools and good penetrancepenetrance despite less than despite less than adequate policy description adequate policy description -- perhaps due perhaps due to selfto self--evident dataevident data

SMARTSMART-- Individual QuestionsIndividual Questions

0

1

2

3

4

5

Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9


High

Low

Reasonable


Question 2 shows good policy development Question 2 shows good policy development but poor but poor penetrancepenetrance throughout the throughout the organizationorganizationQuestion 1 and 3 indicate opportunities for Question 1 and 3 indicate opportunities for improvementimprovement

SMART SMART -- IndividualIndividual QuestionsQuestions

0

1

2

3

4

5

Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9


High

Low

Reasonable

Rather than looking at individual responses to Rather than looking at individual responses to written and interview data, another approach was written and interview data, another approach was to aggregate the information into Safety Processes. to aggregate the information into Safety Processes. Data was aggregated following processes:Data was aggregated following processes:

Fundamental RequirementsFundamental RequirementsTraining ProcessesTraining ProcessesEvaluation ProcessesEvaluation ProcessesMeasurement ProcessesMeasurement ProcessesImplementation ProcessesImplementation Processes

Ratings of Responses by Processes

1 2 3 4 5

Implementation

Measurement

Evaluation

Training

Fundamentals

Penetrance Policy Quality

HighLow Reasonable

In general these data show a strong emphasis on training and

implementation


1 2 3 4 5

Implementation

Measurement

Evaluation

Training

Fundamentals


HighLow Reasonable

**

**

These data show that penetranceinto the organization falls behind

policy understanding and an ability to articulate policy perspectives


1 2 3 4 5

Implementation

Measurement

Evaluation

Training

Fundamentals


HighLow Reasonable

******

**

Evaluation strategies and measurement tools could use more

emphasis


1 2 3 4 5

Implementation

Measurement

Evaluation

Training

Fundamentals


HighLow Reasonable

Below acceptableBelow acceptable

Mostly Below acceptableMostly Below acceptable

Interview Findings

Generally Safety Personnel Understood Policy and Could

Articulate the Policy but Generally Agreed that the Penetrance in the Organization could be improved

Ratings of Interviews for Managers and Safety Professionals

2.5

3

3.5

4

Senior Managers SafetyProfessionals

Line Managers


High

Low

Reasonable

Comparison of Interviews and Written Responses

1

2

3

4

5

SeniorManagers

Safety LineManagers

Written


High

Low

Reasonable

Ratings of Interview Questions

0

1

2

3

4

5

Q25 Q26 Q27 Q28 Q29 Q31 Q32 Q33


High

Low

Reasonable

Ratings of Interviews by Processes

1 2 3 4 5

Economic Impact on Safety

Contractor Training

Measurement of Safety

SMS Daily Duties


HighLow Reasonable

**

Are there examples of questions you can ask in your organization?

You bet!!

Example 1 –Non-Punitive Reporting Systems

Does the company have a non-punitive reporting policy?

If so, can you describe it and tell me if it is followed in day-to-day operations?

Non-Punitive Reporting Systems: Answer

The company has a wellThe company has a well--documented and documented and comprehensive noncomprehensive non--punitive reporting policy that can punitive reporting policy that can be found in the flight operations manual and is posted be found in the flight operations manual and is posted on all safety bulletin boards. on all safety bulletin boards. The policy includes easy to use webThe policy includes easy to use web--based and paper based and paper reporting forms, and has an option for anonymity if the reporting forms, and has an option for anonymity if the reporter requests it. reporter requests it. Unfortunately, most people won’t fill out reports Unfortunately, most people won’t fill out reports because they don’t believe they will be protected from because they don’t believe they will be protected from prosecution by management if they report errors.prosecution by management if they report errors.

Non-Punitive Reporting Systems: ScoringQuality of Response = 3Quality of Response = 3•• The answer is of reasonable quality and demonstrates a The answer is of reasonable quality and demonstrates a

fundamental understanding of the company’s nonfundamental understanding of the company’s non--punitive punitive reporting policyreporting policy

Policy Assessment = 4 Policy Assessment = 4 •• The answer reflects a good understanding of the nonThe answer reflects a good understanding of the non--punitive punitive

reporting system, including details such as where it can be reporting system, including details such as where it can be found and how to use itfound and how to use it

Organizational Penetration = 1Organizational Penetration = 1•• Even though the answer is of reasonable quality and Even though the answer is of reasonable quality and

demonstrates a good understanding of the policy, it shows demonstrates a good understanding of the policy, it shows poor adherence and the policy has not penetrated into daypoor adherence and the policy has not penetrated into day--toto--day operationsday operations

Example 2 –Reactive and Proactive

Safety Reporting

What processes or steps do you follow when you have an incident and observe

a hazard?

How do you think it is working?

Reactive and Proactive Reporting Processes: Line Staff Answer

When there is a safety incident, there are specific When there is a safety incident, there are specific procedures that we follow in the operations manual and procedures that we follow in the operations manual and forms that we fill out. forms that we fill out. [Prompt, how do you think this is working][Prompt, how do you think this is working] It works It works pretty well I guess but I don’t get much feedback. I used pretty well I guess but I don’t get much feedback. I used to fill out the forms for everything but now the incident to fill out the forms for everything but now the incident needs to be serious before I do the paperwork.needs to be serious before I do the paperwork.When I observe potential hazards I usually let my When I observe potential hazards I usually let my supervisor know.supervisor know.

Reactive and Proactive Reporting Processes: Line Staff

Good response and understanding of Policy and ProceduresGood response and understanding of Policy and Procedures

Good adherence to procedures but becoming diligence seems Good adherence to procedures but becoming diligence seems to be reducing over time indicating poorer to be reducing over time indicating poorer PenetrancePenetrance

Much better Policy and Procedures for Reactive Reporting of Much better Policy and Procedures for Reactive Reporting of Incidents than for Proactive Assessment of HazardsIncidents than for Proactive Assessment of Hazards

Reactive and Proactive Reporting Processes: Executive Answer

When there is a safety incident, there are specific procedures tWhen there is a safety incident, there are specific procedures that hat need to be followed in the operations manual and forms that we need to be followed in the operations manual and forms that we fill out. All employees are trained on these procedures and formfill out. All employees are trained on these procedures and forms s are completed and analyzed so that corrective action can be are completed and analyzed so that corrective action can be taken. These reporting procedures are flexible enough to cover taken. These reporting procedures are flexible enough to cover both reactive and proactive reporting and are used in both casesboth reactive and proactive reporting and are used in both cases..[Prompt, how do you think this is working][Prompt, how do you think this is working] It works very well. It works very well. We have binders of analyses and act on these analyses to improveWe have binders of analyses and act on these analyses to improvesafety. safety. It must be working well because the reports pick up data on It must be working well because the reports pick up data on serious incidents. Safety is improving as demonstrated by fewer serious incidents. Safety is improving as demonstrated by fewer reports every year.reports every year.

Reactive and Proactive Reporting Processes: Executive

Good response and understanding of Policy and ProceduresGood response and understanding of Policy and Procedures

No demonstration of the need for feedback to ensure No demonstration of the need for feedback to ensure maintenance of implementationmaintenance of implementation

Declining Declining PenetrancePenetrance given that line staff are restricting their given that line staff are restricting their reporting and the Executives interpret this as improved safety.reporting and the Executives interpret this as improved safety.

SummaryThe SMART method produced objective The SMART method produced objective assessments of the Safety Management assessments of the Safety Management System System The approach appears to provide reliable The approach appears to provide reliable ratings and consistent resultsratings and consistent resultsThe findings appear to identify areas of The findings appear to identify areas of excellence and opportunities for excellence and opportunities for improvementimprovement

SummaryClearly there are differences in the Quality Clearly there are differences in the Quality of Responses, Understanding of Policy, and of Responses, Understanding of Policy, and Penetration of Safety into the Organization Penetration of Safety into the Organization that need to be addressedthat need to be addressedThe approach clearly identifies strengths and The approach clearly identifies strengths and weaknesses in the Safety Management weaknesses in the Safety Management System to help focus Continuous Quality System to help focus Continuous Quality Improvement with regard to Safety Improvement with regard to Safety Management SystemsManagement Systems

SummaryImportantly, the feedback from this Importantly, the feedback from this analysis provides the organization analysis provides the organization with a baseline against which to with a baseline against which to measure change (improvement) of measure change (improvement) of future Quality Improvement Strategies future Quality Improvement Strategies and changes in subsequent Policy and changes in subsequent Policy Development Development

SummaryEqually important is that this type of Equally important is that this type of feedback helps target resources to feedback helps target resources to strategic areas strategic areas Using this strategy, decisions can be Using this strategy, decisions can be made empirically on the most made empirically on the most important areas to invest resources important areas to invest resources while protecting resources for areas while protecting resources for areas where accomplishments have occurred where accomplishments have occurred

SummaryThis approach will allow strategic decisions This approach will allow strategic decisions This approach will allow the strategic allocation This approach will allow the strategic allocation of resources to where they might be of most of resources to where they might be of most benefitbenefitThis approach needs to be informed by future This approach needs to be informed by future applications leading to a applications leading to a manualizedmanualized approach approach that can be widely available with appropriate that can be widely available with appropriate consultationconsultation

Quality of Response

Pene

tran

ce

Low

High

High

Low

Minimum Success

High

Policy Understanding Low

Multidimensional Conceptual ModelMultidimensional Conceptual Modelof Successful SMS implementationof Successful SMS implementation

ConclusionConclusion

ScientificScientificDefensibleDefensibleScaleableScaleablePerformance BasedPerformance BasedContinuous Quality Improvement ToolContinuous Quality Improvement Tool

ConclusionConclusion

Allows self evaluationAllows self evaluationComprehensiveComprehensiveSimple to useSimple to useTheory driven Theory driven -- Practically OrientedPractically OrientedTransparencyTransparency

“…“… the overall process was the overall process was helpful in identifying the method helpful in identifying the method of conducting a safety audit of conducting a safety audit ……[our organization] can learn [our organization] can learn significantly from this process.significantly from this process.””

President & Chief Executive Officer President & Chief Executive Officer

on the SMART Methodon the SMART Method

Documents

Using the SMART Method to Assess SMS in Aviation · Using the SMART Method to Assess SMS in Aviation Ronald J. Heslegrave, ... regard to all aspects of the evaluation. ... Q38 Q39