55
SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM 10 YEARS OF USING THE CSM FOR RISK EVALUATION AND ASSESSMENT (CSMRA) BY DRAGAN JOVICIC, IVAN LUCIC AND PETE GRACEY Slide 1 28 TH APRIL 2021 The Safety & Reliability Society is a Licensed Member of the Engineering Council for CEng and IEng Professional Registration

SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

  • Upload
    others

  • View
    3

  • Download
    0

Embed Size (px)

Citation preview

Page 1: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

SAFETY & RELIABILITY SOCIETY

LESSONS LEARNED FROM 10 YEARS OF USING THE CSM FOR RISK EVALUATION

AND ASSESSMENT (CSMRA)

BYDRAGAN JOVICIC, IVAN LUCIC

AND PETE GRACEY

Slide 1

28TH APRIL 2021

The Safety & Reliability Society is a Licensed Member of the

Engineering Council for CEng and IEng Professional Registration

Page 2: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

Slide 2

• Introduction

• The presentations:

• Background – Dragan Jovicic

• Infrastructure Manager – Ivan Lucic

• National Safety Authority – Pete Gracey

• European Rail Agency – Dragan Jovicic

• Q&A session

• Accessing the webinar recording and slides

• Feedback

Note: the Webinar is being recorded

Programme

Page 3: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

VIEWING IN FULLSCREEN AND Q&A FACILITY

Fullscreen select buttons

Write message icon

Ask question icon

Click here to move the AV screen to the right

Click here to go to Full Screen/reduce or expand the chat area

To exit the event at the end click here

Page 4: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

PRESENTATION 1

DRAGAN JOVICIC

European Rail Agency (ERA)

Page 5: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

Slide n° 5Introduction to Reg. 402/2013 on the CSM RA – History, Initial

Objectives, Method – SaRS Webinar, 28 April 2021

Introduction to the EU Regulation 402/2013 on the

CSM for risk evaluation and assessment (CSM RA)

History & Context - Initial Objectives – Method

Safety and Reliability Society (SaRS) Webinar, 28 April 2021

Dragan JOVICIC, EU Agency for Railways

Page 6: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

Slide n° 6Introduction to Reg. 402/2013 on the CSM RA – History, Initial

Objectives, Method – SaRS Webinar, 28 April 2021

History

Historicallyfirst EVENT legislator’s REACTION

then

❑ new laws❑ new codes of practice❑ new rules❑ new standards

Every country (27) had national

❑ technical & safety rules❑ standards & operational conceptsConsequences

❑ Differences in principles, technical standards, safety barriers and approaches to safety cultures hindering establishment of international transport by railways

❑ International transport dependent mostly on voluntary multilateral agreements (COTIF, RIV, …) and additional, very often unnecessary, checks, safety demonstrations & authorisations for cross-border operations

❑ To open railway market harmonisation of practices across EU became essential

Page 7: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

Slide n° 7Introduction to Reg. 402/2013 on the CSM RA – History, Initial

Objectives, Method – SaRS Webinar, 28 April 2021

EU "New Approach" for creation of a single European railway marketUse of CABs for checking compliance with applicable requirements

Opening of railway market to competition requires the removal of

the historical national barriers

Technical harmonisation through common TSIs

Safety harmonisation through common approaches to safety management

Interoperability Directive 2008/57 → 2016/797

EC Verification of conformity to TSIs

(EU Rules)

Verification of conformity vs. National Rules

MS Certification+ Surveillance

Accred./Recogn.

Assessment of Reg. 402/2013

Accred./Recogn.

Bodies notified to EC & MS (NoBos)

Bodies designatedby MS (DeBos)

NSA/ERA(RUs/IMs)

ECM Certification Body (ECM CB)

AsBos vs. Reg. 402/2013

these are all Conformity Assessment Bodies (CABs)

Safety Directive 2004/49 → 2016/798

AsBos

SMS Certification+ Supervision

"New Approach" requires mutual recognition of independent conformity assessment by CABs

Page 8: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

Slide n° 8Introduction to Reg. 402/2013 on the CSM RA – History, Initial

Objectives, Method – SaRS Webinar, 28 April 2021

Safety Directive 2004/49 repealed by Directive 2016/798Introduces concepts of both risk and “proactive risk management”

Safe Operation & Maintenance

(i.e. proactive risk management)

Activities undertaken by RUs, IMs

& ECMs

SMS/MSRisks

Put in place a certified

SMSMS

DO

CHECKACT

PLAN

Major shift in managing safety the railway operations, traffic and maintenance activities

❑ Past: compliance with well-established national rules, standards and legislation (→ technical differences and different approaches to safety among countries)

SMSMS

DO

CHECKACT/

ADJUST

PLAN

❑ New framework (based on concepts of risk):

RUs/IMs/ECMs fully responsible for safe management of their business through a proactive SMS/MS [PDCA]

SMS/MS certified by an acknowledged authority to attest company capability to manage safely all its activities, including the safe management of:

⚫ Technical;⚫ Operational, and;⚫ Organisational

changes

Page 9: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

Slide n° 9Introduction to Reg. 402/2013 on the CSM RA – History, Initial

Objectives, Method – SaRS Webinar, 28 April 2021

Risk Manage_

ment

Processes

(Existing)Rules

Procedures

Human Factors

CSM for risk assessment & CSM for monitoring) Cornerstones/pillars of the SMS Risk Management

Risk Monitoring1078/2012

Risk Assessment

402/2013

DO

CHECKACT/

ADJUST

PLAN

Page 10: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

Slide n° 10Introduction to Reg. 402/2013 on the CSM RA – History, Initial

Objectives, Method – SaRS Webinar, 28 April 2021

+

CSM for monitoring

CSM for risk assessment

(RCM - Risk Control Measures)

Railway System

Order E(p) Command U(p)

Action Plan

Actual System Performance

Predictive Risk Assessment

Operational Monitoring and Preventive/Corrective measures :

REGULATOR

Information on what to monitor & how to monitor it

Measure X (p)

Gap ε (p)

Corrector C(p) System under regulation G(p)

Sensor K(p)

Output S(p)

ACTION:

REACTION

Change

(Objectives)

Controlled Change Management Process based on the combined use of predictive Risk Assessment and Operational Monitoring

Page 11: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

Slide n° 11Introduction to Reg. 402/2013 on the CSM RA – History, Initial

Objectives, Method – SaRS Webinar, 28 April 2021

Proactive safety management based on the “concept of risk”Think in terms of «risk» and «proactive safety management»

Compared to historical practices: “react and fix” approach

where existing national railway systems were usually based on compliance with national rules & Standards, and a retrospective review of «bad experiences» from the past.

Safety Directive requires RUs, IMs (ECMs) to set up a documented Safety Management System (a Maintenance System) for controlling and managing risks linked to their activities, looking at both forward and retrospective

New proactive approach based on risk: “Predict and Prevent”

1) What can go wrong? (predict/identify hazards/risks)2) Can it reasonably occur? (likelihood)3) What are the impacts if it happens? (consequences)4) What can I do either to prevent it, or to minimise consequences, and

to manage safely my business? (risk control/mitigation measures)

Key questions to ask in a proactive “risk based approach”:

Page 12: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

Slide n° 12Introduction to Reg. 402/2013 on the CSM RA – History, Initial

Objectives, Method – SaRS Webinar, 28 April 2021

Initial objectives of CSM for risk assessment Main considerations

❑ Work with safety experts from EU railways →WG composed of NSAs (15), CER (5), EIM (2), UNIFE (3), UITP (1 – International Association of Public Transport)

❑ A generic process for a harmonised and systematic approach to risk and safety assessments based on existing practices, methods and recognised EU standards:

Recognition of strong railway safety culture → freedom left to each company to continue using existing risk assessment methods/tools/techniques

CSM provides common principles:

⚫ risk assessment process as in CENELEC 50126⚫ allows use of Codes of Practice and Similar Reference Systems, whereas

CENELEC 50126:1999 focused just on explicit risk estimation principle⚫ transparency: documented evidence and use of Hazard Log/Record

→ As CSM requirements can be fulfilled in many ways (not a unique one), CSM does not fix mandatory tools (e.g. FTA, FMECA, etc.)

Advices on risk assessment “tools” given in guidelines developed alongside CSM

Page 13: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

Slide n° 13Introduction to Reg. 402/2013 on the CSM RA – History, Initial

Objectives, Method – SaRS Webinar, 28 April 2021

Initial objectives of CSM for risk assessment Main considerations

❑ As railway market opening split former railway companies into many different players → need to emphasise on:

everyone’s roles and responsibilities close cooperation for joint identification & management

of risks shared across interfaces between actors where necessary, exchange safety related information

❑ Equal treatment and transparent decision making for all railway actors →mutual recognition of results from risk assessments (independent check by 2nd pair of eyes)

Incumbent Railway

Company

NSA

RU & IM

ECM

❑ Keepers❑ Manufacturers❑ Suppliers❑ CABs❑ etc.

❑ Flexibility to decide: (concept of SIGNIFICANT change)

when a systematic approach, formalism and witnessingby an independent 2nd pair of eyes (AsBo) is needed

when a simpler formalism is sufficient, with no need for regulating and witnessing by a 2nd pair of eyes

Page 14: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

Slide n° 14Introduction to Reg. 402/2013 on the CSM RA – History, Initial

Objectives, Method – SaRS Webinar, 28 April 2021

Iterative Risk Management Process “triggered” by a Significant Change

Short overview of Regulation 402/2013CSM risk assessment process - Flowchart in Annex I

1) Common risk assessment process including:(essential requirements → legally binding)

(a) System definition

(b) Identification of hazards/risks & associated safety measures

(c) Risk analysis based on exiting risk acceptance principles (CoP, Ref. Syst, Explicit Risk Estimation - no priority)

(d) Risk evaluation for checking acceptance of risk(s)

(e) Definition of safety requirements from identified safety measures

2) Demonstration of system compliance with identified safety requirements

3) Requirements for mutual recognition:

(a) Hazard Management via a Hazard Log(b) Independent Assessment (Body)

1(a)

1(b)&1(c)

1(d)

1(e)

2

3(a)3(b)

1(c)(iii)1(c)(ii)1(c)(i)

Page 15: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

Slide n° 15Introduction to Reg. 402/2013 on the CSM RA – History, Initial

Objectives, Method – SaRS Webinar, 28 April 2021

²

Short overview of Regulation 402/2013 on the CSM for risk assessment

WHEN should it be applied? → always forTechnical, Operational and Organisational changes

❑ Always at least evaluation of “significance”❑ Risk assessment process in Annex I mandatory

only change is”significant”

NON-SIGNIFICANT safety related changes: risk control mandatory but risk assessment method not regulated/imposed, e.g. process of CENELEC 50126 or ISO 31000 acceptable

²

HOW should it be applied? (no constraints → freedom for details)

❑ Different ways possible - No prescriptions how to comply with “essential requirements”❑ Proposer free to use any tools, instruments or standards - Examples guidelines

WHO shall apply it?→ Proposer

❑ IMs, RUs, ECMs;❑ Actors requested to apply CSM by law (TSIs, directives,…) →Manufacturers

(e.g. an applicant for the Authorisation for placing a vehicle on the market)❑ Other actors when defined through contractual arrangements

WHO decides on significance?

❑ If no NNRs, Proposer decides ❑ NSA does not decide❑ Traceability of changes for

supervision by NSA/ECM CB

Page 16: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

Slide n° 16Introduction to Reg. 402/2013 on the CSM RA – History, Initial

Objectives, Method – SaRS Webinar, 28 April 2021

… in order to provide the necessary reasonable assuranceto an Authorising Entity who must take a decision based on the risk assessment results from the application of the CSM

Article 6 appoint an AsBo

❑ gives an expert judgement on the confidence that the change under assessment can fulfil safely the intended objectives,

❑ enables to gain trust in an effective risk management and allows the mutual recognition of the results from the risk assessment

correct application of risk management process in Annex I of CSM, and;

suitability of results from the risk management process;

AsBo is a reliable second pair of eyes who through independent assessment of:

Regulation 402/2013 – Obligations for anindependent safety assessment by an AsBo

Mandatory

Page 17: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

Questions? → Send e-mail on: [email protected]

Page 18: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

PRESENTATION 2

IVAN LUCIC

Network Rail

Page 19: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

19

Applying CSM-RAin real life…

Ivan LucicEastern Region Systems EngineerSRP ChairApril 2021

Page 20: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

20

DISCLAIMERThese are my own personal views, not necessarily

the official views of Network Rail.

Page 21: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

21

• Data from years 2006-2020. • Numbers are NR wide • Probably doesn’t include every single project (heavy

maintenance for example)• Few projects in dataset were either blank or of ‘pending’

status, these were discounted from below numbers

Total number of projects in dataset = 3152

Total number of significant projects = 561 = 17.8%

Total No of Non-significant = 2591 = 82.2%

Scale of things in NR ….

Page 22: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

22

Experience …. Background: selection of similar projects with and without CSMRA, NR vs LU:

• Simple depot upgrades

• Conventional signalling upgrade

• Station upgrade

Summary : whole experience was cheaper, less painful and more efficient in LU

Page 23: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

23

Why is it difficult? - LegislationIn principle nothing wrong with the concept…some improvements…

• Roles defined in legislation do not easily fit in the NR organisation and the life cycle change of accountabilities;

• What do we mean by Additionality in practice? In practice Who can manage Additionality in the organisation?

• Safety Justification vs Safety Case (50126);

• Competencies for actors and those who deliver not defined but defined for ASBO (with wrong focus on legislation as opposed on real life application of system safety NOT site safety);

• Focus on What without the guidance on How.

Page 24: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

24

Why is it difficult? – NR Implementation• Focus on What without guidance on the How

• ASBOs often doing tick box audits not assessments, overly bureaucratic?

• Most importantly, it is not mandated by legislation to have to have external ASBO, why we can’t do it internally? It works perfectly well in LU.

• System Safety and CDM/Site safetydo not mix well and should not be mixed

• What about projects that are not significant but shall be authorised?

Making sure that people aren’t

hurt while at work

Engineering Safety

(or “System Safety”)

Making sure that people aren’t

hurt as a result of the changes

that we make to the railway

Occupational Health

and Safety

Page 25: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

• Plain English

• Simple and Clear guidance on:

What, Where,

How, Who,

When, Why

25

What was wrong with the Yellow Book?

• Examples of what good looks like

• Training and certification available

• Scope includes maintenance as well as projects

Page 26: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

26

Conclusions and recommendationsERA:

• Simplify roles so its easer to fit in the generic railway IM organisation;

• Consult those who deliver what are points that need clarification and clarify (CSMRA vs CENELEC; additionality; direct to How …)

UK (NR, ASBOs and ORR):

• Simplify and take bureaucracy out of it please

• There is noting in the Yellow Book that contradicts CSMRA, bring it back as the industry good practice guidance

All involved:

• Please get the

• In doing the above please consult those who actually do it on the ground System Integration Hub

Page 27: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

PRESENTATION 3

PETE GRACEY

Office of Rail and Road (ORR)

Page 28: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

NSA reflections on 10 years of CSMRAPete GraceyHead of Interoperability & Rail Vehicle EngineeringOffice of Rail and Road

28/04/2021

Page 29: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

My CSMRA background (or the second half of my railway career to date)

Pre-CSMRA

• Designated Competent Person

Conformance Assessment Body

• AsBo/NoBo/DeBo Signatory

• Independent Competent Person (Pre-CSMRA ROGS 2006)

Project Delivery

• AsBo/NoBo/DeBo Client

• Proposer of change

National Safety Authority

• Authorisation process and policy

orr.gov.uk 29

Page 30: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

The UK transition to CSMRA

The Yellow Book

• Extensive guidance – not always followed

• Pete’s view – CSMRA, it’s a subset of YB and entirely compatible, but …

Safety Verification by Independent Competent Person (ROGS 2006)

• Over application?

• Bias of the individual (or system …)

• Retention issues (NR specific)

Transition to CSMRA

• Not starting from the right point …

• … but adhering to the spirit!

orr.gov.uk 30

Page 31: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

Further reflections

As AsBo

• Education, education, education

• Bias (or blinkers) of the client

As Client

• Education, education, education

• Bias (or background) of the assessor

As NSA

• Education

• Communication, communication, communication

orr.gov.uk 31

Page 32: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

View from the NSA

• Use the process – it works

• React & fix => predict & prevent

• Supports Authorisation (and specific case decisions)

• Understand the Codes of Practice or reference systems you apply

• Manage your assessment

• Document safety decisions (particularly in early stages of design)

• Utilise the competence of your own team

• Challenge the assessment, but resolve the issues

• Use the process to support (better) investment decisions

• Feeds into through life costing

orr.gov.uk 32

Page 33: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …
Page 34: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

PRESENTATION 4

DRAGAN JOVICIC

European Rail Agency (ERA)

Page 35: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

Slide n° 35REX of EU railway sector on Reg. 402/2013 on CSM RA – Experience – Lessons

learned – Next steps – SaRS Webinar, 28 April 2021

Return of EXperience (REX) of European railway sector

with EU Reg. 402/2013 on the CSM for

risk evaluation and assessment (CSM RA)

Experience – Lessons learned – Next steps

Safety and Reliability Society (SaRS) Webinar, 28 April 2021

Dragan JOVICIC, EU Agency for Railways

Page 36: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

Slide n° 36REX of EU railway sector on Reg. 402/2013 on CSM RA – Experience – Lessons

learned – Next steps – SaRS Webinar, 28 April 2021

Methodology for gathering EU sector experience withRegulation 402/2013 on the CSM for risk assessment

❑ ERA gather sector experience with method and, where necessary, make recommendations for improvements

❑ Survey carried out in 2018 across whole EU railway sector

applied by❑ RUs, IMs & ECMs through SMS

❑ Manufacturers placing on market

❑ NSAs❑ ECM Certification Bodies❑ AsBos

Reply rate38 over 115

(33%)

ERA web page: more details in report on the "Return of Experience (REX) with theuse of the CSM for risk assessment” [Location: Activities\Common Safety Methods]

Reminders

Page 37: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

Slide n° 37REX of EU railway sector on Reg. 402/2013 on CSM RA – Experience – Lessons

learned – Next steps – SaRS Webinar, 28 April 2021

❑ 10 years: 352/2009 → 402/2013 → 2015/1136 (risk assessment process unchanged)

❑ Actual experience with method still very poor -Not used very often because of fears

❑ Why is sector reluctant to use the CSM?

replaces blind application of rules and standards by a proactive analysis of what can go wrong and how to avoid it

requires a independent check by an independent assessment body

❑ CSM Achilles' heel: significant change filter

❑ Misperception of CSM: complicated, boring, requires a lot of unnecessary paper

❑ None likes it and none happy to deal with it

❑ → concept of significant change intentionally misused to avoid using CSM

Sector reluctant to use risk assessment and independent assessorsWrong perception of «risk» and «risk management» concepts

Page 38: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

Slide n° 38REX of EU railway sector on Reg. 402/2013 on CSM RA – Experience – Lessons

learned – Next steps – SaRS Webinar, 28 April 2021

Reminder: “use of rules” vs. “think in terms of risk”→ big change in responsibilities and in approach to safety management of railway traffic and railway operations

Sector reluctant to use risk assessment and independent assessorsToo much self-confidence while complexity of railway system increased

❑ Sector too much self-confident that railways are safe because “We are running trains for more than 150

years using well-known rules. We know our business

We do not need being taught what & how to do!”

❑ Underestimation: in 150 years of railway history, risks also changed a lot and new risks arose:

complexity of architecture and technology used in railway system drastically changed

operating speeds and traffic density increased

interactions with human operators (HOF) more complex (design, installation, operation, maintenance)

much higher staff turn over with shorter careers →less time to build a railway safety culture (mindset)

Page 39: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

Slide n° 39REX of EU railway sector on Reg. 402/2013 on CSM RA – Experience – Lessons

learned – Next steps – SaRS Webinar, 28 April 2021

❑ Only 2% to 3% (<5%) of changes significant (big companies 800-1000 changes/year)❑ Common/accurate picture per country or stakeholder category not possible❑ Levels of understanding, application and experience vary in function of [Country;

Company in country; project team in company]

(1) a small minority of stakeholders well understands and correctly applies CSM

Not more than “10 % to 15%” of RUs, IMs & ECMs have a positive experience

For manufacturers ratio is probably much higher, because more familiar with the CENELEC 50126 standard and risk management concepts

(2) a large number of stakeholders for whom understanding and application of CSM were challenging but they are “learning by doing” and they are “continually improving” its use

(3) a large number of stakeholders across whole EU still facing serious difficultieswith understanding and correct application of CSM

General trends emerging from analysis of inputs [RUs, IMs, ECMs, Manufacturers]

Many consider a few years’ period of CSM application too short for being able to understand, apply correctly, “learn by doing” and improve experienced deficiencies

Overall picture regarding sector experience withRegulation 402/2013 on the CSM for risk assessment

Page 40: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

Slide n° 40REX of EU railway sector on Reg. 402/2013 on CSM RA – Experience – Lessons

learned – Next steps – SaRS Webinar, 28 April 2021

BIG COMPANIES:

1. Size not always the only determining parameter

Have complex organisational structures, higher needs and high ambitions in terms of change management → face different challenges due to organisational and interface complexity

2. Sufficient resources are allocated to risk management (usually a dedicated Safety Department)

3. Higher knowledge of “risk concepts“ and more familiar with “process thinking, and systematic & top-down approach” to risk management → CSM used more proactively way

4. Nevertheless, some big companies are still at a very early stage of implementation of the CSM and need still to improve

Variability of stakeholders’ experience with Regulation 402/2013Big vs. small companies

SMALL COMPANIES:

1. Need more time to react and adapt to changes of legislation

2. Usually smaller needs for changes → less opportunities to “learn by doing” & gain experience to improve

3. More problems regarding resources, as it is carried out by operational staff in charge of many also activities→ usually often subcontracted

4. However, sometimes flexible and able to implement smarter solutions that better capture the essence of CSM

5. In case of deficiencies, tend to solve problems instead of debating on them

Page 41: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

Slide n° 41REX of EU railway sector on Reg. 402/2013 on CSM RA – Experience – Lessons

learned – Next steps – SaRS Webinar, 28 April 2021

Former incumbent railway companies:

1. More resistant to change and applica-tion of new methods, in particular if they feel it requires more resources

→ prefer to continue using former processes/solutions

2. More rigid ways of thinking, managing, solving issues, etc. usually based on CENELEC 5012x guidelines

→ consider CSM as unnecessary duplication of CENELEC

3. More resources and railway experience for being able to apply the CSM

Newcomer railway companies:

1. Often very flat organisations with less resources and rarely staff competent in risk assessment

→ often application of CSM is sub-contracted

2. Capable adapting faster to new terminology

3. Flexible and can change easier their way of thinking in a more constructive way.

→ tend to be proactive and apply CSM for staying on market

Variability of stakeholders’ experience with Regulation 402/2013Newcomer vs. incumbent railway companies

Page 42: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

Slide n° 42REX of EU railway sector on Reg. 402/2013 on CSM RA – Experience – Lessons

learned – Next steps – SaRS Webinar, 28 April 2021

1. All countries: great lack of experts on market competent in railways, risk asses-sment and risk management → necessity for long term action plans across EU

2. Usually evidence and documentation of risk assessment with poor quality

3. Better understood & more applied for Technical changes (→ better experience)

4. Operational & Organisational changes rather use of standards and CoPs without necessarily formally documenting evidence of proper compliance with CoPs

5. CSM found being complex, not understandable → hardly applicable

6. Perceived as a heavy & costly legal obligation → applied very late at end of project or design when all choices are fixed and too late to change design (just for paper)

7. Usually not used as a proactive tool for risk identification and prevention/controlof risks - Almost never used for cost optimisation and process improvement

8. CSM & AsBo seen useless burden, costly bureaucratic formalism without benefits

9. Obligation to appoint AsBo discourages a lot decisions to apply CSM

10.Roles of AsBo & NSA vs. independent assessment of Op. & Org. changes unclear

Stakeholders’ experience with Regulation 402/2013Major observations

Page 43: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

Slide n° 43REX of EU railway sector on Reg. 402/2013 on CSM RA – Experience – Lessons

learned – Next steps – SaRS Webinar, 28 April 2021

1. System Definition poorly described: boundaries, interfaces, environmental conditions, assumptions, limits of risk assessment, etc. → incomplete HAZID

2. Poor or inexistent description of project organisation: unclear roles, responsibilities and competence of experts in charge of risk management

3. Poor interface management:

a) all concerned actors not systematically involved in risk identification and joint risk management

b) many proposers tend to transfer control measures for risks that go beyond scope of responsibility and reasonable domain of control by those actors

4. Non-conformities in area of another actor not always reported to right actor who is capable to take necessary corrective/preventive actions

5. Due to poor system definition and poor cooperation for interface management

a) no assurance that all risks, in particular risks shared across (internal & external) interfaces, are identified and appropriately controlled

b) no assurance that safe integration of sub-systems and in railway system is properly done (knowing also that concept is not well understood either)

Stakeholders’ experience with Regulation 402/2013Major observations

Page 44: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

Slide n° 44REX of EU railway sector on Reg. 402/2013 on CSM RA – Experience – Lessons

learned – Next steps – SaRS Webinar, 28 April 2021

Experience with the assessment of the SIGNIFICANCE of changes

1. Many proposers find assessment of significance of changes complex and difficult to arrive at right decision with criteria in Article 4 of Reg. 402/2013. Reason: objective is to conclude that change is non-significant, but not easy without proper risk assessment

2. Decision dependent on experience of people/teams in charge of assessment

3. To avoid applying formal process of CSM and appointment of independent AsBo only 2 to 3% (<5%) of changes considered significant, independently on whether country/company familiar with “rule based” or “risk based thinking”

4. Most of those 2 to 3% (<5%) of significant changes are “TECHNICAL changes”

5. Justifications of non-significance of safety related changes is poor. When existing, no assurance of proper risk identification & risk management of non-significance

6. Many actors wrongly interpret “non-significant” as a “permission to do nothing”

7. NSAs report they do not know how proposers assess and control risks arising from non-significant safety related changes → how do NSAs supervise RU/IM SMS Change Management Process?

Page 45: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

Slide n° 45REX of EU railway sector on Reg. 402/2013 on CSM RA – Experience – Lessons

learned – Next steps – SaRS Webinar, 28 April 2021

1. Although preferably used, poor experience with Codes of Practice (CoPs)

2. Used just to limit risk management efforts - Side effect constraining operational measures whereas technical preventive measures would be better

3. Usually railway CoPs or known standards used, relevance not questioned (wrong)

4. Unnecessarily detailed Hazard Identification whereas a low level of detail is largely sufficient + Use of CoP not always registered in Hazard Record

5. Detailed Hazard Checklists misused with CoPs (incompatibles levels of detail)

6. If Similar Reference Systems are used, safety requirements derived without questioning further relevance of Ref. Syst. → point § 2.4.2(a) of Annex I of CSM (“reference system still qualifies for approval in MS”) seems not to be checked

7. When CoPs or Ref. Syst. used proposer complies with all their requirements

8. Deviations vs. CoPs (or vs. Ref. Syst.) assessed using other two principles, including explicit risk estimation

Experience with the use of Codes of Practice (CoP) and Reference Systems

Page 46: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

Slide n° 46REX of EU railway sector on Reg. 402/2013 on CSM RA – Experience – Lessons

learned – Next steps – SaRS Webinar, 28 April 2021

Based on positive feedback of 10-15% inputs/actors, it can be concluded

Effectiveness of CSMConclusions from gathering of REX

(b) Objective justifications were not yet identified which would require an urgent amendment or revision of the risk assessment process in CSM

(a) CSM for risk assessment is effective for achieving market access and mutual recognition (objectives in Art. 1 & 15(5) of CSM)

(b) For a big majority of actors, period of application of CSM not long enough to “learn by doing” and improve;

(c) Main problem: lack of understanding of risk based concepts and finding out what can go wrong with a change – Often not aware that some risks are already controlled by existing SMS provisions;

→ Agency did not recommend yet to revise or amend the CSM for risk assessment

Page 47: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

Slide n° 47REX of EU railway sector on Reg. 402/2013 on CSM RA – Experience – Lessons

learned – Next steps – SaRS Webinar, 28 April 2021

Next stepsAgency recommendations

(b) As effectiveness of “CSM for risk assessment” depends on correct implementation of “CSM for monitoring”, combine measurement of return of experience of both methods

The lack of effective and complete use of these two CSMs shall not lead to an unreliable picture of the methods and to mislead conclusions that could result in unjustified revisions of CSM RA and CSM MO with additional complexity

(c) Leave sector enough time to “learn and improve by doing” and gain experience with risk management

(d) With support of NSAs, ECM CBs and AsBos, later on measure again effectiveness and experience with CSM RA & CSM MO

(a) In collaboration with NSAs, put in place an appropriate dissemination programmeand training activities to increase sector awareness with “risk management concepts” (i.e. CSM RA and CSM MO) [already ongoing by some NSAs]

Page 48: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

Slide n° 48REX of EU railway sector on Reg. 402/2013 on CSM RA – Experience – Lessons

learned – Next steps – SaRS Webinar, 28 April 2021

(e) Build a solid return of experience with clear justifications for necessary improvements or clarifications of legislation/guidance material, based on objective observations of real weaknesses and problems with the CSMs, and better target of any necessary additional dissemination or training activity

Revision of legislation without clear justifications may create additional administrative burden, complexity and misunderstanding of successive

amendments of legislation whereas sector is trying to implement correctly the existing pieces of legislation, to learn and improve by doing and to set up a

proper “risk based” safety culture in a very complex environment

Next stepsAgency recommendations

Page 49: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

Slide n° 49REX of EU railway sector on Reg. 402/2013 on CSM RA – Experience – Lessons

learned – Next steps – SaRS Webinar, 28 April 2021

1) Take strategic decisions and initiatives with EC and Member States in terms of education and training in order to:

a) compensate the current lack of expertise on the market in railway risk assessment, and;

b) anticipate the growing loss of railway expertise with a continually increasing number of employees reaching the age of retirement;

2) In scope of implementation of 4th Railway Package, monitor how NSAs supervise the RU/IM SMS and verify that the Change Management Control process of the RU/IM SMS identifies/captures, assesses and properly controls the risks arising from both non-significant and significant changes;

3) ERA to coordinate with EA, national accreditation and recognition bodies and AsBo Cooperation Group in order to harmonise further the requirements, acknowledgement and supervision of AsBo competence across whole EU;

4) Set up verification mechanisms of Recognition Bodies

Next stepsAgency recommendations

Page 50: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

Slide n° 50REX of EU railway sector on Reg. 402/2013 on CSM RA – Experience – Lessons

learned – Next steps – SaRS Webinar, 28 April 2021

“It is not because things are difficult that we do not dare, it is

because we do not dare that they are difficult”, Lucius Annaeus SENECA

What to keep in mind concerning Regulation 402/2013 on the CSM for risk assessment?

❑ Risk assessment is a means to an end, not an end in itself - The aim is to keep people safe, not only to have good paperwork

❑ Reluctance to CSM: Do not seek complicated interpretation of requirements – Read what is actually written & use common sense

❑ Not required to forget rules but to know why rules are used

❑ No constraints on how to comply with every step of process. Leaves room/flexibility for an effective risk assessment process that can be targeted to size & complexity of change

❑ It is a method/tool that guides Risk Analyst to do right things and take right decisionsIt does not replace expert judgement of analyst, only one who can take right decisions

❑ Single application of risk assessment and risk management techniques does not guarantee that right hazards/risks are actually identified and thus properly controlled

❑ It is crucial that method is applied by competent staff with a good knowledge of system to assess, experience and knowledge of risk assessment standards (e.g. CENELEC 50126, ICE 61508, etc.)

Page 51: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

Questions? → Send e-mail on: [email protected]

Page 52: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

QUESTIONS?

Page 53: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

Slide 53

• All webinar recordings are archived under the Resources Tab on the SaRS website

• SaRS members can access this archive as a member benefit:

• You can join as a full member, or

• The simplified “Associate of the Society” grade which gives you access to all the SaRS resources including the webinars.

• See www.sars.org.uk

• This webinar recording will be available in a couple of days so keep an eye out if you want to see it again

Accessing the webinar recordings

Page 54: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

JOINING SARS

54

If you aren’t a member and have enjoyed this webinar please take a look at our

website, see the benefits of membership and join us.

Safety & Reliability Society is a Licensed Member of the Engineering

Council for CEng and IEng Professional Registration

We accept membership applications from

candidates from all relevant backgrounds –

membership is open to everyone from students

to experienced professionals

More information available at www.sars.org.uk

Page 55: SAFETY & RELIABILITY SOCIETY LESSONS LEARNED FROM …

FEEDBACK

55

• I am now going to initiate a feedback form

• Please can I ask you to fill it in before you exit the webinar

• The information is vital for us to improve our offering

• Please take two minutes to fill it in and click Submit

• Your CPD Certificate will be sent to you if you requested

one at registration

• Thank you very much and STAY SAFE