IAEA International Atomic Energy Agency TOOLS AND METHODS FOR ENHANCING IRRS EFFECTIVENESS AND EFFICIENCY Presentation to the Workshop on Lessons Learned

IAEAInternational Atomic Energy Agency

TOOLS AND METHODS FOR ENHANCING IRRS EFFECTIVENESS

AND EFFICIENCY

Presentation to the Workshop on Lessons Learned from IRRS Missions

Moscow, Russian Federation

9-11 December 2014

IAEA IRRS enhancements 2

Outline

• On the IRRS missions in general

• Continuous improvement of the IRRS programme

• Conclusions


IRRS BASICS


IRRS Process

• Prep-meeting about 6 months before

• Self-assessment by host• First mission: 12 days• Report finalization about

3 months• Follow-up missions after

2-3 years, about 7 days


IRRS Guidelines

• Published in May 2013

• A series of Targeted Consultancies was held for collecting experience and advice for a thorough revision

• Novel edition will be compiled based on the consultancies


IAEA Safety Standards used in IRRS


Basis for findings: GSR Part 1 Requirements

IRRS Modules

+ other IAEA Safety Standards


Missions in 2006-2014

Korea (f)

Niger UAE Zimbabwe

Mexico Germany Canada (f) France

Mauritius Ukraine Russia Switzerland Netherlands

Cameroon Sierra Leone UK (part 2) Australia (f) Belgium Cameroon

Kenya Namibia Vietnam Slovenia Czech Rep. Vietnam (f)

Uganda Madagascar Lebanon Ukraine (f) Germany (f) Finland Russia (f) Slovenia (f)

France Gabon Botswana Canada USA Korea Greece UK (f) Jordan

UK Australia Spain Peru China Spain (f) Slovakia Poland Pakistan

Romania Japan Cote d'Ivoire France (f) Iran Romania Sweden Bulgaria USA (f)

2006 2007 2008 2009 2010 2011 2012 2013 2014

3 9 8 7 4 9 4 6 10


Participation in the missions (2006-2014)*

• No. of experts (including IAEA): 373• No. of participations: 707• From 59 countries

010

2030

4050

ALB

ALG

ARG

AUS

AUT

BEL

BIH

BKF

BUL

BRA

CAN

CHI

CPR

CUB

CZR

DEN

EGY

ETH

FIN

FRA

GFR

GRE

HUN

ICE

IND

IRE

ISR

ITA

JOR

JPNLIT

MEX

MO

RN

ETNIR

NO

RN

ZEPA

KPO

RRO

KRO

MRU

SSA

FSL

OSL

RSP

ASU

DSY

RSW

ESW

ITA

NTU

NTU

RU

AEUK

UKR

USA

Expe

rts

in IR

RS m

issi

ons

Num

ber

of p

artic

ipati

ons

Num

ber

of e

xper

ts

* by October 2014


CONTINUOUS IMPROVEMENT OF THE IRRS PROGRAMME


IAEA Nuclear Safety Action Plan

IAEA is to “Strengthen IAEA peer reviews in order to maximize the benefits to Member States”.

In specific:

2.4.1 IAEA Secretariat to review the effectiveness of their peer reviews: IRRS, and therein

2.4.1.2 Define effectiveness criteria, apply the criteria to a representative number of selected IRRS missions and prepare a report


Ways to improve

• IRRS Training• IRRS Mission Standards• Feedbacks• Measuring effectiveness and

efficiency• Prompt Evaluation Report


First Basic IRRS Training Course

• First held in October 2013• 62 participants from 38 MSs + EC• In the framework of the EC-IAEA

cooperation in the field of Nuclear Safety

• Covered all aspects of the IRRS programme, 16 Training Lectures, 3 Information Lectures

• Test Questionnaires for all Lectures


Feedback on the course

Evaluation of the BIT Course by participantsQuestion Mark

4.64

How do you rate the training course presentations from the point of view of

the information therein and their completeness 4.45

their clarity 4.30

their format 4.29

How do you rate the training course lecturers from the pointof view of

their competence in the field 4.69

the way they held the presentations 4.26

What is your opinion on the course-related organization and arrangements by IAEA? 4.69

Average: 4.47

Number of answers: 56

How useful you consider the training in your possible future IRRS activity?


Feedback from the Test Questionnaires

0.500.550.600.650.700.750.800.850.900.951.00

1 2 3 4 5 6 7 8 9 10 11

Solution ratios in Test Questions No.1

0.500.550.600.650.700.750.800.850.900.951.00

1 2 3 4 5 6 7 8 9

Solution ratios in Test Questions No.6


2nd Edition

• Held in October 2014• 63 participants from 39 MSs + EC• Even more favourable feedback

Evaluation of the BIT2 Course by participantsQuestion Mark

4.41



their clarity 4.18

their format 4.13





Average: 4.36




On-site training

• By invitation from MSs• Minimum 10 competent

participants• First held at ONR (Bootle, UK)• 15 participants• Excellent acceptance

Evaluation of the BIT2-ONR Course by participantsQuestion Mark

4.21



their clarity 4.21

their format 3.86





Average: 4.20




Refresher Training

• On the 0th day of the missions• Covers details important for the given

missions:• Overview of the IRRS Process• Methodology of the review• Evaluation and documentation of Observations• Report writing• IRRS Modules to review• IRRS Mission Schedule

• Held in 2014 in the Netherlands and France


Feedback

Evaluation of the Refresher IRRS Course (NET) by participantsQuestion Mark

4,70


the information therein and their completeness 4,45

their clarity 4,40

their format 4,47


their competence in the field 4,90

the way they held the presentations 4,76

What is your opinion on the course-related organization and arrangements by IAEA? 4,60

Average: 4,61



Evaluation of the Refresher IRRS Course (FRA) by participantsQuestion Mark

4.39



their clarity 4.23

their format 4.18





Average: 4.30




Ways to improve




Mission Standards – ARM Report Template

Under development / discussion


Mission Standards – Team Assignment


Mission Standards – Report Template

• Cross-contribution table• Instructions for reviewers

• Issues to cover• Format of observations


Mission Standards – Schedule


Ways to improve




Feedbacks on the IRRS process

• Questionnaire on the quality of the ARM – reviewers at mission start

• Questionnaire on effectiveness of the mission – reviewers at end of mission

• Round-table conclusions and suggestions – reviewers at end of mission

• Questionnaire on effectiveness of the mission – host after mission

• Questionnaire on effectiveness of the initial mission – host before follow-up mission

• Workshop for team leaders (January 2013)• Workshop for experienced reviewers (April

2015)


Ways to improve




Effectiveness and efficiency of IRRS

• NSAP requires establishing criteria for efficiency of IRRS missions

• 16 Performance Indicators introduced to characterize the E&E of a mission, such as:• Team size; team experience;• Time available to prepare for the mission;• Feedback from team members and from

hosts;• Coverage of the Action Plan by findings;• Balance of findings;• Conciseness of mission report; etc.

http://gnssn.iaea.org/regnet/irrs/Pages/IRRS_pub_docs.aspx





Examples of Performance Indicators (1)

3

3.5

4

4.5

5

Miss1 Miss2 Miss3 Miss4 Miss5 Miss6

mar

k

mission

Host Feedback

0.000.100.200.300.400.500.600.70

Fu-1 Fu-2 Fu-3 Fu-4 Fu-5 Fu-6 Fu-7

Ratio of open issues in follow-upsRecommendations Suggestions All

0.000.100.200.300.400.500.600.700.800.901.00

Fu-1 Fu-2 Fu-3 Fu-4 Ini-1 Ini-2 Ini-3 Ini-4 Ini-5 Ini-6 Ini-7 Ini-8 Ini-9

ratio

of m

embe

rs w

ith e

xper

ienc

e

missions

Team Experience

0306090

120150180210240270300330360390

FRA

UK

JPN

MEX

SPA

UKR GFR

FRA

-fu

CAN

UK-

fuR

US

CPR US

RO

MSP

A-f

uR

OK

GFR

-fu

SLO

SWI

CAN

-fu

SWE

SLR

FIN

BUL

UK-

fu2

RU

S-fu

CZR

BEL

com

pleti

on ti

me

in d

ays

Mission

Report Completion Time

Optimum Acceptable Need attention

0 0.2 0.4


Examples of Performance Indicators (2)

Balance of findings: R*GP, R/S const.

Expectations: a) R and GP are negatively correlatedb) R and S are positively correlated

1) The expected correlations are not present in all missions

2) The reasons in case of R and GP are:• reviewers feel obliged to compensate by GP • some topics (with possible issues) are more

frequently considered than others (with GP)

3) The reasons in case of R and S are:• R -> S: compromise between the team & host• S -> R: the team tends to be stricter

or the specificities of the given mission

0

5

10

15

20

25

30

35

O E S R J G T F H U P I N Q L V A M C D K B

Num

ber o

f obs

erva

tions

Mission

Correlation of Recommendations and Good PracticesR GP

0

5

10

15

20

25

30

35

O E S R J G T F H U P I N Q L V A M C D K B

Num

ber o

f find

ings

Mission

Correlation of Recommendations and SuggestionsR S


E & E of past IRRS missions

Note: The Performance Indicator values presented here characterize the effectiveness and efficiency of the IRRS process and represent no indication whatsoever on the safety performance of the host countries of the missions

0.000

0.020

0.040

0.060

0.080

0.100

0.120

0.140

0.160

0.180

0.200

0.220

ROK GFR-fu SLO SWI CAN-fu SWE SLR FIN BUL UK-fu2 RUS-fu CZR BEL US-fu SLO-fu

Dist

ance

from

opti

mum

Mission

Efficiency and effectiveness of IRRS missions

to analyse

acceptable

effective

optimum

The distance from optimum tends to decrease, i.e. the E&E tends to increase


Ways to improve




Prompt evaluation report

CONTENTS• Basic mission data• Feedback from the team• Feedback from the hosts• Overall evaluation of effectiveness

and efficiency• Discussion, conclusions• Summary


Evaluation of a mission via PI’s


Reports on nuclear missions






IN SUMMARY


Conclusions

• IRRS is an effective means of reviewing compliance of RB status and activities with IAEA safety standards

• Regular invitation of IRRS missions by all IAEA MS is highly recommended

• The IRRS process is mature and effective – yet it is under continuous development by using novel standardized tools and feedback from participants

• Lessons learned from past missions as well as mission effectiveness and efficiency analysed

• Prompt evaluations of IRRS missions to nuclear countries were performed in a pilot phase


THANK YOU FOR YOUR ATTENTION

Photo: V. Friedrich

This activity is conducted by the IAEA, with funding by the European Union. The views expressed in this presentation do not necessarily reflect the views of the European Commission

Documents

IAEA International Atomic Energy Agency TOOLS AND METHODS FOR ENHANCING IRRS EFFECTIVENESS AND EFFICIENCY Presentation to the Workshop on Lessons Learned