Lessons learned from the irrs missions to countries … Documents/OPEN … · Web viewLessons learned from the irrs missions to countries with OPERATING NPPS, 2006-2013 summary SUMMARY

Lessons learned from the irrs missions to countries with

OPERATING NPPS, 2006-2013 summary

SUMMARY OF THE TECHNICAL REPORT prepared for the IAEA Workshop on “Lessons Learned from the Integrated Regulatory Review Service

(IRRS)”, 29-31 October 2014, Moscow, Russian Federation

2006-2013

V2

LESSONS LEARNED FROM IRRS MISSIONS (2006-2013) TO COUNTRIES WITH NPPS P a g e | 1

ContentsI. EXECUTIVE SUMMARY............................................................................................................................2

II. INTRODUCTION......................................................................................................................................3

III. BACKGROUND.....................................................................................................................................4

IV. THE IRRS PROCESS AND THE IAEA SAFETY STANDARDS......................................................................5

IV.1. References to IAEA safety standards..............................................................................................5

IV.2. References to GSR Part 1 Requirements.........................................................................................6

V. LESSONS LEARNED ON THE REGULATORY FRAMEWORK........................................................................7

V.1. Numerical characteristics of observations......................................................................................8

V.2. Number distribution of observations..............................................................................................9

V.3. Technical contents of observations................................................................................................9

V.4. Issues appearing in both findings and Good Practices..................................................................10

V.5. Characteristic examples of regulatory lessons learned.................................................................11

V.6 Follow-up missions.......................................................................................................................11

VI. LESSONS LEARNED ON THE IRRS PROCESS........................................................................................12

VI.1. Correlations of observations.........................................................................................................12

VI.2 Measuring effectiveness and efficiency of the IRRS process.........................................................15

VI.3 Performance Indicators on the effectiveness and efficiency of the IRRS missions........................15

VI.4 Efficiency and effectiveness of recent missions............................................................................18

REFERENCES.................................................................................................................................................18


I. EXECUTIVE SUMMARY

IRRS missions conducted in 2006-2013 to Member States with operating nuclear power plants were analysed and evaluated in order to summarize and evaluate the lessons learned from the Integrated Regulatory Review Service (IRRS) missions and to draw conclusions of general validity on the nuclear regulatory framework of the MSs on one hand and on the effectiveness, efficiency and possible further enhancement of the IRRS process on the other hand. The primary results of the analysis and evaluation are reported in two background working materials [7, 8] also available from the IRRS website http://gnssn.iaea.org/regnet/irrs/Pages/IRRS_pub_docs.aspx. The present technical report summarizes the main results of the previous works. The analysis and evaluation covers the nuclear safety aspects in the scope of the core IRRS Modules.

The IRRS review process compares the national regulatory infrastructure of the host country against requirements by IAEA safety standards. Thus frequency of appearance of the safety standard requirements in mission findings is an indication of the importance of the related regulatory activities in the international regulatory framework. This frequency analysis of references to GSR Part 1 requirements shows that the ones fitting the most with the issues encountered in the missions are:

Demonstration of safety for the authorization of facilities and activities (Requirement 24) Staffing and competence of the regulatory body (Requirement 18) Liaison with advisory bodies and support organizations (Requirement 20) Establishing regulations and guides (Requirement 32)

In a broader sense Recommendations and Suggestions (in brief findings) predominantly refer to GSR Part 1 Requirements relating to core regulatory functions (authorization, review and assessment, inspection, enforcement, establishing regulations and guides).

The main results of an IRRS mission are summarized in Recommendations, Suggestions and Good Practices (in brief observations). The analysis of the number of observations revealed that although the number of observations in various missions may reflect specific differences in regulatory frameworks of the host countries, the overall picture of the distribution of the observations is fairly uniform. Furthermore, the number of observations arising from missions confirms that the practice followed in the actual IRRS process ensures a reasonable distribution for these numbers.

The number of various observations is expected to show certain correlations, thus e.g. the number of Recommendations and/or Suggestions is expected to be negatively correlated to the number of Good Practices or the number of Recommendations and that of Suggestions are expected to show positive correlation. Such correlations definitely exist, yet in certain missions they are less pronounced. Sometimes specificities of the host country regulatory practice may explain the deviation from the general tendency, in several cases, however, the reasons relate to the activity of the reviewers. Thus e.g. reviewers formulating a number of findings unintentionally feel obliged to compensate by also offering Good Practices. In other cases, as a result of compromises between the team and the host Recommendations are converted to Suggestions, or that the team tends to offer Recommendations also when a Suggestion might be more adequate.

The technical contents of the observations were analysed by first collecting the observations of similar character into Subject Groups. The Subject Groups with the highest number of findings in initial missions are:

Providing/using legal framework for regulatory activities Developing the inspection programme Reviewing regulations and guides Procedures and guides for inspection Staffing of regulatory body Developing an (integrated) Management System in general Complying with emergency preparedness and response technical requirements

The Subject Groups with the largest number of Good Practices are

http://gnssn.iaea.org/regnet/irrs/Pages/IRRS_pub_docs.aspx


External involvement in the regulatory process; Training of the RB; Follow-up of inspections; Transparency, public involvement;

Certain topics were the subjects of both findings and Good Practices. In order to point to the issues dominating the two extremities (findings vs. Good Practices) a balance value was calculated by taking the difference of the relative number of the two types of observations in the Subject Groups. According to the balance values the issues most frequently appearing in findings and seldom mentioned as Good Practices are related to the provision and use of the legal framework for regulatory activities. The subject most frequently appraised by Good Practices is the follow-up of inspections. Activities related to inspection represent the most controversial part of the regulatory regime involving some fields with frequent non-compliance and others often commended by the missions.

The findings remained open in follow-up missions were also examined. Based on the limited data available it appears that complying with findings addressed to the government needs more time, or needs other prerequisites than those for the regulatory body. The Subject Groups containing the most findings remained open are

Details of the Management System (MS), developing MS manual Resources of the RB;

An important conclusion of the analysis of mission results is that there is no essential difference between the missions conducted to EU Member States and missions to non-EU Member States.

The lessons learned from missions on the regulatory framework are followed by lessons learned on the IRRS process itself. 16 Performance Indicators (PIs) are defined to characterize the effectiveness and efficiency of a mission. The PIs were evaluated for the past missions and general conclusions were drawn on them. The PI-s introduced are: size of the IRRS team; length of the IRRS mission report; time available for reviewing the Advance Reference Material; number of advance reviewers comments; feedbacks from the team and from the host (three types); experience of the team; extent and coverage by findings of the host’s Action Plan (three types); Balance of findings (Recommendation – Good Practice and Recommendation – Suggestion correlations); completeness of the mission report; completion time of the mission report and number of open issues in a follow-up mission. The relative distance of the PIs from their optimum ranges are defined and calculated for past missions. The average of the indicator-wise relative distances defines the measure of effectiveness and efficiency of a mission. Ranges for optimum, effective, acceptable and worth-of-analysing values of the measure of effectiveness and efficiency were defined on a heuristic basis. The measure was determined for the missions in 2011 through 2013 and it was found that the effectiveness and efficiency of these missions all fall into the effective range while the values, although fluctuating, show an improving trend.

II. INTRODUCTION

This report summarizes and evaluates the lessons learned from the Integrated Regulatory Review Service (IRRS) missions conducted in the period of 2006 – 2013 to Member States operating nuclear power plants.

The evaluation of the IRRS missions and of the process will cover only the nuclear safety aspects, and thus shall address the IRRS missions to countries with nuclear power plants conducted between 2006 and 2013 and the follow-up mission to USA in 2014; Modules 1 through 10 (the ‘core review areas’) of the IRRS missions as shown in Figure 1 shall be considered, Modules 5 to 9 (i.e. the ‘core regulatory processes’) include the following facilities and activities: nuclear power plants, research reactors, fuel cycle facilities, and decommissioning of nuclear facilities. The present report is a summary of a more extensive and comprehensive report [12].


This report may be of interest to governmental and regulatory bodies in the field of nuclear safety, international organisations including the European Union and the OECD NEA, Technical Support Organizations of regulatory bodies in IAEA Member States and the IAEA Secretariat.

III. BACKGROUND

The IAEA Integrated Regulatory Review Service (IRRS) was established to strengthen and enhance the effectiveness of national regulatory infrastruc-tures for safety. The IRRS process reviews both regulatory technical and regulatory policy issues against IAEA safety standards.

Figure 1: IRRS Modules

The IRRS has a modular structure. Figure 1 illustrates the structure and the facilities and activities within the scope of IRRS and how core regulatory functions interface with them.

An IRRS review team comprises international experts recruited by the IAEA from the regulatory bodies and support organizations of its Member States. In the 50 missions conducted between 2006 and 2013 351 experts participated from 57 Members States and from the IAEA altogether 645 times. The IAEA provides a team coordinator and deputy team coordinator plus additional experts and administrative support as needed.

The observations arising from an IRRS mission are gathered in the mission report as ‘Recommendations’ (which reflect non-compliance with specific IAEA Safety Requirements), ‘Suggestions’ (which identify opportunities for further improvement) – these two together ‘findings’ – and ‘Good Practices’ (which recognise outstanding regulatory practices superior to those generally observed elsewhere).

In order to review the progress in the implementation of the Recommendations and Suggestions of an IRRS mission, a follow-up mission is usually invited two to four years after the initial mission. A follow-up mission results in a report whereby the Recommendations and Suggestions of the initial mission are rated either ‘closed’ or ‘open’.

Detailed information about the IRRS may be found in Service Series 23; ‘ Integrated Regulatory Review Service (IRRS) Guidelines for the Preparation and Conduct of IRRS Missions’ [1].

The ‘IAEA Fundamental Safety Principles’ (SF-1) [2] provides the basis for the IAEA safety standards and for safety related programmes. In support of effective regulation, the IAEA has established safety standards in the area of the ‘Governmental, Legal and Regulatory Framework for Safety’ (GSR Part 1), ‘The Management System for Facilities and Activities’ (GS-R-3), ‘Radiation Protection and Safety of Radiation Sources: International Basic Safety Standards’ (GSR Part 3 interim edition) and ‘Preparedness and Response for a Nuclear or Radiological Emergency’ (GS-R-2) [3-6].

GSR Part 1 [3] comprises 36 overarching requirements on the governmental, legal and regulatory framework for safety. These requirements are addressed in all IRRS Modules and as such represent the backbone of the IRRS.

This report covers lessons learned from 22 initial and nine follow-up IRRS missions conducted in countries with operating NPPs (+Iran’s NPP started up after the mission).


The analysis was conducted in the framework of an Agreement between IAEA and the European Commissions and it also covers a comparison between missions to EU member states and non-EU states. Figure 2 lists all IRRS missions held, the IRRS-recipient countries included in this analysis are marked in colour. Follow-up missions are denoted by (f).

Niger UAEMexico Germany Canada (f)

Mauritius Ukraine Russia SwitzerlandCameroon Sierra Leone UK (part 2) Australia (f) Belgium

Kenya Namibia Vietnam Slovenia Czech Rep.Uganda Madagascar Lebanon Ukraine (f) Germany (f) Finland Russia (f)

France Gabon Botswana Canada USA Korea Greece UK (f)UK Australia Spain Peru China Spain (f) Slovakia Poland

Romania Japan Cote d'Ivoire France (f) Iran Romania Sweden Bulgaria USA (f)2006 2007 2008 2009 2010 2011 2012 2013 2014

Figure 2: IRRS missions selected for this analysis and evaluation (in colour)

The main results of the IRRS peer review are summarized in ‘observations’ (recommendations, suggestions and good practices) that are based on IAEA safety standards. The term ‘reference’, as used in this report, means the act of referencing those requirements in the IAEA safety standards that serve as bases for IRRS mission observations.

This report is a summary of the analysis, evaluation and development work performed in recent years related to the IRRS missions to nuclear countries in 2006-2013. Details of this work are given in two background Working Documents of the IAEA [7, 8], both available from the IRRS homepage on the IAEA website1. Full version of the report [12] summarized here is also available from the same site.

In the Chapters below only the most important Conclusions of the full report are quoted, therefore the numbering of the Conclusions is not continuous.

IV. THE IRRS PROCESS AND THE IAEA SAFETY STANDARDS

This chapter summarizes the statistical characteristics of the numbers of references to the various IAEA safety standards in general and to GSR Part 1 (the basic standard in IRRS) in specific.

IV.1. References to IAEA safety standards

The 36 overarching requirements on the governmental, legal and regulatory framework for safety in GSR Part 1 are addressed in all IRRS Modules and as shown Figure 3, comprise the most references in the observations. The frequency of references to various safety standards may provide information on both the IRRS process and the regulatory issues. Figure 3 shows the ratios of references from findings in the various IRRS Modules to the basic standard GSR Part 1 [3] and to other IAEA safety standards.

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1 2 3 4 5 6 7 8 9 10

Num

ber o

f ref

eren

ces

Module number

R&S references to GSR Part 1 and elsewhere from Modules to GSR Part 1 to other standards

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1 2 3 4 5 6 7 8 9 10

Num

ber o

f ref

eren

ces

Module number

GP References to GSR Part 1 and elsewhere from Modules to GSR Part 1 to other standards

Figure 3: Module-wise reference ratios to GSR Part 1 and to other standards from findings (left) and from Good Practices (right)

It is seen from Figure 3 that the IRRS Modules have the following referencing character:

1 http://gnssn.iaea.org/regnet/irrs/pages/default.aspx

http://gnssn.iaea.org/regnet/irrs/pages/default.aspx


Module 4 (Management system of the regulatory body) mainly refers to GS-R-3; Module 10 (Emergency Preparedness and Response) mainly refers to GS-R-2; Module 6 (Review and assessment) refers both to GSR Part 1 and to other standards; The majority of the references from other IRRS core Modules relate to GSR Part 1.

(Note that the high number of references from Module 9 to standards other than GSR Part 1 is the result of a single mission and should not be considered typical.)

Table 1 below characterizes the roles of the IAEA safety standards in the IRRS process:

IAEA safety standard Number of references

References/missionEU non-EU

GSR Part 1 654 26.8 33.9GS-R-3 192 9.2 8.1GS-R-2 83 4.5 2.8

GS-G-1.3 55 2.5 2.4All other 222 7.8 8.9

Table 1: References from IAEA safety standards in EU and non-EU missions

Conclusion 1: The data presented above confirm that the General Safety Requirements, and primarily GSR Part 1, provide basis for the IRRS process, further basic references are GS-R- 3 and GS-R-2.

Table 1 also reflects

Conclusion 2: There is no substantial difference between the EU and non-EU missions in referencing the IAEA safety standards.

IV.2. References to GSR Part 1 Requirements

GSR Part 1 has 36 Requirements and all Requirements are unanimously linked to one of the IRRS Modules [1]. In Ref. [12} referencing of the GSR Part 1 Requirements is analysed in full details, the main results are summarized here.

The highest reference frequencies in decreasing order are shown in Figure 4. The integers on the top are the IRRS Module numbers where the Requirements belong. The values above the bars give the average number of references to the given Requirements from findings. The overall average number of references from findings in a mission is 29.7.

Figure 4: Average number of references from findings per mission, to GSR Part 1 Requirements

Conclusion 3: The GSR Part 1 Requirements most frequently appearing in issues encountered in IRRS missions are

R24: Demonstration of safety for the authorization of facilities and activities in Module 5; R18: Staffing and competence of the regulatory body in Module 3; R20: Liaison with advisory bodies and support organizations in Module 3; R32: Regulations and guides (most frequent in EU missions) in Module 9; R2: Establishment of a framework for safety in Module 1; R27: Inspection of facilities and activities in Module 7; R25: Review and assessment of information relevant to safety in Module 6; R29: Graded approach to inspection of facilities and activities.

Figure 4 also shows that among the most frequently referred GSR Part 1 Requirements

five (R24, 32, 27, 25, 29) are related to core regulatory functions (Modules 5 through 9) and make up almost one third of all references to sum up to 9.18 references/initial mission;

2.41 2.362.09 2.09

1.68 1.64 1.59 1.45

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

4.00

4.50

24 18 20 32 2 27 25 29

Num

ber o

f ref

eren

ces p

er m

issio

ns

GSR Part 1 Requirement number

Most frequent references to GSR Part 1EU missions non-EU missions all missions

3 635 9 1 7 7


R20, although not belonging to the core regulatory modules 5 through 9, is directly related to the core regulatory functions.

Conclusion 4: Recommendations and Suggestions, referencing GSR Part 1 Requirements relating to core regulatory functions from initial missions, comprise a dominant part of all references.

The reason for this dominance may be that: the IRRS process focusses on these areas; there are issues that need to be addressed in the international regulatory framework in these areas; the GSR Part 1 requirements related to these areas have broader validity and may be applied to

other areas as well.

These possible reasons and their consequences need further investigations.

Figure 5 shows the cumulated numbers of references from findings in initial missions to the GSR Part 1 Requirements summed up from the largest to the smallest (blue curve) and vice versa (red curve).

Figure 5: Cumulative references to GSR Part 1 Requirements

The figure reflects that 10 Requirements (28% of all) receive about 60% of the references from the recommendations and

suggestions of all missions (c.f. green arrow in the figure); 17 Requirements (47%) are the subject of about 80% of the references (c.f. red arrow); the nine least referenced Requirements (25% of all) are the subject of about 5% of the references

(blue arrow).

V. LESSONS LEARNED ON THE REGULATORY FRAMEWORK

In this chapter the numbers, distributions, correlations and technical contents of the various observations of the IRRS missions are investigated. It is to be stressed that the number of recommendations suggestion and good practices is in no way a measure of the status of the regulatory body and comparison of such numbers among IRRS missions to various countries provides no meaningful information.

Figure 6: Total number of observations from all missions

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1.00

24 18 20 32 2 27 25 29 4 31 33 15 22 26 16 19 36 23 11 1 8 3 30 5 14 28 7 17 35 10 21 34 6 13 9 12

Cum

ulati

ve ra

te


Cumulative reference rate of findings to GSR Part 1from largest from smallest

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

4.00

18 24 32 20 2 27 25 29 4 31 33 22 15 26 16 19 36 23 11 1 8 3 30 5 10 28 7 14 17 35 21 34 6 13 9 12

Aver

age

num

ber o

f ref

eren

ces p

er m

issio

ns


References per missions to GSR Part 1EU missions non-EU missions all missions

0

20

40

60

80

100

120

140

160

1 2 3 4 5 6 7 8 9 10

Num

ber o

f obs

erva

tions

Modules

Number of observations in all missionsRecommendations Suggestions Good Practices


V.1. Numerical characteristics of observations

For the 22 initial missions (13 EU and nine non-EU) Figure 6 and Table 2 show the distribution of the various observations by Module included in the present analysis (c.f. Figure 4).

(Note again that these numbers refer to the nuclear safety related parts of the missions only, observations related to other safety areas have not been included into the investigations.)

Observations EU non-EU Follow-up Summed Average/missionRecommendations 161 127 24 312 13.1Suggestions 223 172 41 436 18.0Good Practices 110 94 25 229 9.3

Summed 494 393 90 977 40.4

Table 2: Number of observations from all IRRS missions analysed

Figures 7 present the numbers of observations from the 22 initial missions analysed in two forms. It is seen that these numbers vary substantially from mission to mission, yet (apart from outlying values) they are quite uniformly scattered around their average values.

0

5

10

15

20

25

30

35

40

A B C D E F G H I J K L M N O P Q R S T U V

Num

ber o

f ons

erva

tions

Missions

Observations in initial missionsRecommendations Suggestions Good Practices

0

10

20

30

40

50

60

70

A B C D E F G H I J K L M N O P Q R S T U V

Num

ber o

f ob

serv

ation

s

Missions

Observations in initial MissionsR S GP R+S Ʃ

Figures 7a and 7b: Numbers of observations in the initial missions analysed

Further investigations [7, 12] also reflect that – apart from a few outlying values – the distributions of the number of observations resemble normal distribution. In Figure 8 the averages and ranges of the observations are given for the EU and the non-EU missions.

Thus although the number of observations in various missions may reflect specific differences in the nuclear installations, regulatory frameworks and scope of missions, the overall distribution of the observations is fairly uniform

Conclusion 6: The number of observations arising from missions confirms that the practice followed ensures a reasonable distribution for these numbers.

Table 2 and Figure 8 suggest the following

Conclusion 7: There seems to be no essential difference between the EU and non-EU missions in the statistical characters of observations.

It is to be stressed that the number of recommendations suggestion and good practices is in no way a measure of the status of the regulatory body and comparison of such numbers among IRRS missions to various countries provides no meaningful information.

0

5

10

15

20

25

30

Minimum Average Maximum

Ranges of observationsRecommendations-EU Recommendations-nonEU

Suggestions-EU Suggestions-nonEU

Good Practices-EU Good Practices-nonEU


Figure 8: Ranges of all observations from initial mission

V.2. Number distribution of observations

In Figure 6 above the distribution of the various observations on nuclear safety subjects is shown over the Modules addressing core regulatory functions. The figure suggests that

Conclusion 8: Recommendations and Suggestions, relating to most of the core regulatory functions from initial missions demonstrate equal emphasis by the peer review, while Modules 2 and 8 have a considerable lower rate of findings. The definition of the IRRS Modules may therefore need revision to be better balanced out.

Notwithstanding what was stated in Conclusion 6, there may be logical natural rules valid to the observations in missions the knowledge of which might help increasing the effectiveness and objectivity of the missions.

The following relationships seem intuitively logical:

1. In missions where the number of Recommendations is considerably higher than the average, the number of Good Practices is expected to be lower than the average.

2. In missions where the number of Recommendations is considerably higher than the average, the number of Suggestions is also expected to be higher than the average.

0

5

10

15

20

25

30

35

O E S R J G T F H U P I N Q L V A M C D K B

Num

ber o

f obs

erva

tions

Mission

Correlation of Recommendations and Good PracticesR GP

0

5

10

15

20

25

30

35


Num

ber o

f find

ings

Mission

Correlation of Recommendations and SuggestionsR S

Figures 9 and 10: Correlation of Recommendations with Good Practices and with Suggestions

Figure 9 and 10 visualize these correlations for the past missions. These figures as well as numerical analysis in Ref. [7] suggest that although the expected correlations as tendencies exist, in some missions the numbers of observations do not reflect it.

More conclusions are drawn on this subject in Section V.1 below.

V.3. Technical contents of observations

In order to analyse the technical contents of the findings they are classified into Subject Groups (SGs). The purpose of this grouping is to collect the observations having similar characters into statistically meaningful groups. The Subject Groups defined by the past mission observations are introduced in Ref. [7] and further discussed in details in Ref. [12].

Figures 11 and 12: Subject Groups with highest population of observations

0

5

10

15

20

25

30

35

40

45

50

1.d 7.a 9.b 7.c 3.a 4.a 10.e 3.f 4.b 9.a 3.c 3.g 1.a

Num

ber o

f find

ings

per

miss

ions

Subject Group

Subject Groups with the most findings

0

2

4

6

8

10

12

14

16

18

3.f 3.g 7.d 1.e 10.b 9.a 6.a 7.a 2.a 3.a 10.d

Num

ber o

f Go

od P

racti

ces

Subject Group

Subject Groups with the most GPs


Figures 11 and 12 summarize the number of findings and the number of Good Practices in the most populated Subject Groups for all missions. The first ten of them are as follows:

SG For findings SG For Good Practices

1.d Providing/using legal framework for regulatory activities 3.f External involvement in the regulatory

process7.a Developing the inspection programme 3.g Training of the RB9.b Reviewing regulations and guides 7.d Follow-up of inspections7.c Procedures and guides for inspection 1.e Transparency, public involvement

3.a Staffing of regulatory body 10.b Emergency Preparedness and Response communication and cooperation

4.a Developing an (integrated) Management System, general 9.a Developing regulations and guides in

general10.e

Complying with Emergency Preparedness and Response technical requirements 6.a Developing the review and assessment

process3.f External involvement in the regulatory process 7.a Developing the inspection programme4.b Details of the MS, developing MS manual 2.a Operational experience exchange with

international community9.a Developing regulations and guides in general 3.a Staffing of regulatory body

Table 3: Subject Groups most populated by findings and by Good Practices, respectively

The number in the SG identifier refers to the IRRS Module where it belongs to, c.f. [7] and [12].

V.4. Issues appearing in both findings and Good Practices

According to Figure 13 there are four Subject Groups that appear in the top 10 both for findings and for Good Practices (3.f, 9.a, 7.a and 3.a). It is also seen that no Subject Group exists that is among the seven most populated ones for both types. This fact may suggest that there are issues of more or less general nature and are either subjects of findings or are more to be commended. These specific issues are attempted to be made visible by a technique called Balance-values.

The Balance-value of a Subject Group – as introduced in [12] – is the difference of the relative number of Good Practices and relative number of findings belonging to the given Subject Group.

Figure 13: Top ten SGs for findings and for GPs

The Balance-value approach as discussed in details in Ref [12] leads to the conclusions as below (c.f. also Table 3 and Figure 13).

Conclusion 10: The issues most frequently raised by findings and seldom mentioned as Good Practice are related to the provision and use of legal framework for regulatory activities.

Conclusion 11: The subject most frequently appraised by Good Practices is the follow-up of inspections.

The SGs dominated by one or another of the observation types can be used to draw conclusions on problematic as well as on commendable issues generally appearing in IRRS missions. As an example for this let us note that Module 7 (Inspection) often appears in this frequency analysis. SG 7.d (Follow-up of inspections) is on the top of the GP list, while SG 7.c (Procedures and guides for inspection) is second in the findings list based on the Balance-values. Furthermore SG 7.a (Developing the inspection programme) is the second in the list of most populated SGs by findings and is the 8 th in the most populated GP list (c.f. Figure 12 and Table 3). Accordingly:

Conclusion 12: Activities related to inspection are frequently highlighted as non-compliances as well as good practices.

Conclusions 10 through 12 point to possible weak and strong points as well as to issues of interest in the regulatory framework of the Member States. In-depth analysis of the observations in these areas may highlight the underlying causes for these weak and strong points.

R+S GP 1) 1.d 3.f 2) 7.a 3.g 3) 9.b 7.d 4) 7.c 1.e 5) 3.a 10.b 6) 4.a 9.a 7) 10.e 6.a 8) 3.f 7.a 9) 4.b 2.a10) 9.a 3.a


V.5. Characteristic examples of regulatory lessons learned

1. The most populated Subject Group, Providing/using legal framework for regulatory authorities, includes the following typical recommendations:

The government should establish the legal framework for an effectively independent, unambiguously authorized regulatory

body with clear division of responsibilities; provide the regulatory body with the authority to issue or the involvement in issuance of regulatory

requirements; provide full provisions for appeals.

Accordingly the addressee of these findings is the government and the issue raised is lack of specific legal provisions necessary for discharging regulatory responsibilities.

2. SG Developing inspection programme is the second most frequent. Typical findings therein are:

The regulatory body should (or should consider to) expand the inspection programme (in scope, in types, in frequency); develop further its inspection system (initiation, methodology, monitoring, evaluation); improve inspection planning.

This is a fairly homogeneous group of findings, the addressee of which is the regulatory body and the main issue is shortcomings in the inspection methodology applied.

3. Subject Group Reviewing regulations and guides is the third among the most frequent ones with typical findings as below:

The regulatory body should (or should consider to); systematically and periodically review and revise as necessary the regulations, regulatory requirements

and guidance; introduce formalized gap analysis between IAEA requirements and national regulations.

This group also addresses the regulatory body and points to weaknesses in review and revision of regulations .

4. The most frequent Subject Group of Good Practices is External involvement in the regulatory process. Typical observations are

taking advice and assistance from advisory committees, Technical Support Organizations; regular contact with the licensees and manufacturers.

The most frequently commended activity is accepting technical assistance. (Note that this activity is fairly common among the regulatory bodies and as such it would not qualify for Good Practice).

V.6 Follow-up missions

Follow-up missions are primarily meant to review the progress in addressing the Recommendations and Suggestions of the initial missions. Additionally, however, new observations may be formulated, either because the review process raises new issues or because the mission is an ‘extended follow-up mission’.

The progress reached by host countries after initial missions can be measured by the ratio of the number of closed issues to the number of original findings. Figure 14 shows the average values as well as the ranges of closed Recommendations and Suggestions and the sum of them.

Figure 14 Ranges of open issues 0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

Minimum Average Maximum

open

issu

es/i

ntial

issu

es

Ratio of open issues in follow-upsRecommendations Suggestions All


Although the number of follow-up missions is too low to be statistically significant and the findings are unevenly distributed among the various follow-up missions, based on the figure above one concludes:

Conclusion 13: The progress made by host countries between initial and follow-up missions was higher in Modules 5 through 9 (i.e. in the core regulatory functions) than in Modules 1, 3, 4.

Conclusion 14: The relatively high number of issues that remained open suggests that either the time between the initial and follow-up missions was not sufficient to reach compliance with the findings of the initial missions, or the host country did not or could not place equal emphasis on all improvements.

It is often argued that Recommendations or Suggestions addressed to the government need substantially more time to solve than those addressed to the regulatory body. In the three follow-up missions where such findings were investigated the ratio of open issues addressed to the Government indeed exceeded that to the regulator [12]. Therefore one concludes:

Conclusion 15: Based on the limited data available it appears that complying with findings addressed to the government needs more time, or other prerequisites than with those for the regulatory body.

VI. LESSONS LEARNED ON THE IRRS PROCESS

In this chapter the lessons learned from previous missions are drawn based on the correlation properties of the quantities; on the quantitative values of a set of Performance Indicators introduced to evaluate effectiveness and efficiency of the IRRS missions.

VI.1. Correlations of observations

Correlations of the various observations are investigated along four types of data-sets: numbers of references from the observations to various GSR Part 1 Requirements; numbers of observations in various missions (as also seen in Sec. IV.2); IRRS Modules and Subject Groups

Numerical values of the correlation coefficients related to the various data-sets are reported in Reference [7], further plots and graphs visualizing the correlations are given in [12].

A. Correlation of Recommendations and Suggestions

It is shown in References [7] and [12] that the numbers of references from the various observations to the Requirements in GSR Part 1 move together for all types of observations, i.e. they are strongly correlated.

It was demonstrated in Section IV.2 that the number of Recommendations and of Suggestion in various missions – although are expected to be positively correlated – in many cases do not exhibit this correlation. Also large similarity was shown in the variation of numbers of Recommendations and Suggestions along IRRS Modules, [7], [12].

For the sake of illustration Figure 15 shows the distribution of findings among the Subject Groups. (Here and in the following figures the letters on the horizontal axis identify the SGs within the modules.)

05

1015202530354045

a b c d e f g x y a b x y a b c d e f g h x y a b c d e f x y a b c d e f x y a b c d e f g h i x y a b c d e f g h x y a b c d x y a b c d e x y a b c d e x y

Findings/Subject Groups in all missions to NPP countries in 2006-2013 Recommendations Suggestions

Mod.3 Mod.4Mod.1 Mod.7 Mod.8 Mod.9 Mod.102 Mod.5 Mod.6

Figure 15: Number of various findings belonging to the various Subject Groups from all missions

The figure illustrates in a suggestive manner the correlated nature of the two types of findings. The facts presented above suggest that Recommendations and Suggestions show the following relational characters:

their numbers are strongly (positively) correlated over the references to GSR Part 1 Requirements, over IRRS Modules and over Subject Groups;


they fail to show definite correlation over IRRS missions.

Conclusion 16: The frequencies of Recommendations and Suggestion have very similar characters in all cuts of the IRRS process (i.e. by references, missions, Modules and Subject Groups).

Conclusion 17: The lack of distinct positive mission-wise correlations between the numbers of Suggestions and of Recommendations may be attributed to the observations that in some missions Recommendations are converted to Suggestions as results of compromises between the team and the hosts; whereas in other cases (especially when there are a number of serious findings) the team tends to offer Recommendations although a Suggestion might be more adequate.

B. Correlation of Recommendations and Good Practices

These correlations are investigated in details in References [7] and [12] and in Section V.2. According to these results Recommendations and Good Practices exhibit the following relational characters:

their numbers show moderately positive correlation over references to GSR Part 1 Requirements, over IRRS Modules and over Subject Groups;

this correlation may be attributed to the similarities of the topics addressed by all kinds of observations and to the fact - also expressed in Sec. V.5 - that in most cases for the very same subject findings are raised in some mission and exemplary practices are found in others;

the lower than expected negative correlation over missions may be attributed to a few particular missions, detailed analysis of which would give valuable insight.

C. Correlation of Suggestions and Good Practices

As shown above, in certain aspects Suggestions and Recommendations have different statistical characters and especially they show much weaker correlation in their mission-wise distribution than expected. Therefore examination of the correlation of Suggestions and Good Practices may reveal further characteristics of the IRRS process.

Figure 16: Correlation of Suggestions and Good Practices over missions

Referring again to [7] and [12] we see that these two observations have similar characters in referencing to GSR Part 1 Requirements.

One would expect that similarly to the case of Recommendation-Good Practice the numbers of Suggestions in various missions would negatively correlate with the number of Good Practices (by arguing that whenever there is much room for development less outstanding practices are expected).

This, however, is not the case; (c.f. Figure 16) due to strong positive correlation in some of the missions.

0

5

10

15

20

25

30


GP vs. S/Subject Groups in all missions to NPP countries in 2006-2013 Suggestions Good Practices


Figure 17: Number of Suggestions and Good Practices belonging to the various Subject Groups

0

5

10

15

20

25

30

35


Num

ber o

f find

ings

Mission

Correlation of Suggestions and Good PracticesS GP


In contrast to that, as seen in [12] and also shown in Ref. [7] the numbers of Suggestions and Good Practices by IRRS Modules show very strong positive correlation. In other words in those modules where the number of Suggestions is high/low the number of Good Practices tends to be high/low too.

A similar correlating tendency is shown by Figure 17 for the number of these observations belonging to the various Subject Groups.

Thus Suggestions and Good Practices exhibit the following relational characters:

their numbers show strong positive correlation over references to GSR Part 1 very likely for the same reasons as discussed on Recommendations and Good practices above;

the lack of negative correlation of their numbers over missions may partly be attributed to the specificities of certain missions (to be further analysed), partly to other reasons discussed in the subsequent Conclusion 18;

the very strong correlation of these numbers over the various Subject Groups also suggest a generic reason for parallel establishing of Suggestions and Good Practices.

Conclusion 18: The numbers of Suggestions and Good Practices are positively correlated over IRRS Modules as well as

over Subject Groups since there possibly are topics that are particularly important in the regulatory practice (or are consistently pointed out by the missions);

another reason (more psychological than technical) may be that reviewers formulating a number of findings unintentionally feel obliged to compensate by also offering Good Practices;

these issues need to be part of the reviewer training.

The results of this Section lead to the following general

Conclusion 19: All observations tend to refer to GSR Part 1 in a statistically consistent way;

D. Correlation of EU and non-EU missions

In Conclusion 2 it was stated that there is no essential difference between the references from EU and non-EU missions. Conclusion 7 summarized that statistical characters of the observations in EU and non-EU missions are similar. Figure 18 shows the numbers of the various observations in Modules for the two sets of missions.

Figure 18: Number of observations from EU and non-EU missions in various Modules

It is seen from the figure that in most cases these numbers run fairly parallel to each other for the EU and non-EU missions, although there are exceptions and also the values are sometimes very different.

05

1015202530354045


Findings/SGs in EU and non-EU missions to NPP countries in 2006-2013 EU-missions non-EU missions


Figure 19: Number of findings belonging to the various Subject Groups from EU and non-EU missions

In Figure 19 the numbers of findings in the various Subject Groups are shown for the EU and non-EU missions, respectively. The figure demonstrates fairly parallel run of the two sets.

Calculations [7] show strong positive correlations of EU and non-EU missions in all types of data sets.

0.000.501.001.502.002.503.003.504.00

1 2 3 4 5 6 7 8 9 10

Num

ber o

f obs

erva

tions

Modules

Observations/missions in ModulesR-EU R-nEU S-EU S-nEU GP-EU GP-nEU


Conclusion 20: the IRRS missions to EU and non-EU Member States seem to exhibit very similar distributions in the observations.

VI.2 Measuring effectiveness and efficiency of the IRRS process

The ultimate goal of the present work as well as of the initiatives meant to develop further the IRRS process is the maximization of the benefits offered to the Member States by the IRRS missions, which correlates to the maximization of the effectiveness of the IRRS process. In order to measure effectiveness and efficiency of missions a methodology and 16 Performance Indicators (PIs) were introduced in Reference [8] and were applied to recent missions. The PIs assume quantitative values that may fall into ‘optimum’, ‘acceptable’ and ‘needing attention’ ranges. The ranges are defined by experience of past missions and by common sense. The most important results of the effectiveness and efficiency analysis are quoted here; details are given in the full Technical Report [12].

VI.3 Performance Indicators on the effectiveness and efficiency of the IRRS missions

Values relative to their optimum of 14 PIs are quoted and respective conclusions are drawn below.

a) Size of the IRRS team

The optimum team size was assumed to be a linear function of the mission size. Figure 20 shows the past mission team sizes relative to their optimum.

-0.40

-0.30

-0.20

-0.10

0.00

0.10

0.20

0.30

UK FRA JPN MEX SPA UKR GFR CAN RUS CPR US ROM ROK SLO SWI SWE SLR FIN BUL CZR BEL

Team size relative to optimum - initial

-0.40

-0.30

-0.20

-0.10

0.00

0.10

0.20

0.30

FRA UK UKR SPA GFR CAN UK2 RUS US

Relative team size - follow-up

Figure 20: Team-sizes compared to their optimum values Accordingly team-sizes tend to stabilize in the neighbourhood of their optimum values; as the optimum team-size values were derived from the actual values of the missions, teams of any

future missions can only be optimized as relative to past experience. Absolute optimization needs further considerations.

Conclusion 21: It appears that there is a linear dependence of the optimum team size on the mission size.

b) Time available for ARM review

The IRRS Guidelines [1] requires the ARM be provided to the reviewers at least two months prior to the mission, this, however, often was not met in the latest missions.

Therefore a milder condition of minimum 45 days has been stipulated for the optimum range.

Figure 21: Time available for ARM review relative to optimum

Figure 21 illustrates the actual situation in past missions. Values above the green line are optimal, values between the red and green lines are acceptable. As seen in most of the missions the time available for ARM review is above 45 days, but it seldom exceeds 60 days (above the blue line).

-30.00

-20.00

-10.00

0.00

10.00

20.00

30.00

40.00

M Ef N Gf O P Hf Q R S T Af If U V Lf

revi

ew ti

me

in d

ays-

45

Mission

Time for ARM Review


Conclusion 22: The ARM should be provided with sufficient time preceding the IRRS mission.

c) Advance comments from reviewers

The reviewers were requested to provide their first impressions and potential questions prior to the mission based on the ARM received. It is assumed that in an ideal case the number of advanced written comments should be proportional to the size of the mission. In Figure 22 the number of advance comments relative to the optimum limit is shown.

Figure 22: Number of advance comments on ARM relative to the optimum

In most of the missions the number of advance comments on ARM is acceptable; in few of them it is optimum. There was no mission where the number of comments was below acceptable.

d) Feedback from the team and from the host

Four types of questionnaires were used in recent missions to obtain feedback from the reviewers and from the host. Details of the questionnaires are given in Ref. [8]. The four questionnaires are:

1. Feedback from the team on the quality of the ARM (at the beginning of the mission)2. Feedback from the team on the effectiveness of the mission (at the end of the mission)3. Feedback from the host on the effectiveness of the mission (after the mission)4. Feedback from the host on the usefulness of the initial mission findings (before the follow-up)

Each question in the questionnaires is answered by a mark between 1 and 5 (5 represents the most favourable opinion) and averages of the marks are calculated for each questionnaire. Results from recent missions are given in Table 4.

SWI SWE GRE SLR FIN BUL UK-fu2 RUS-fu CZR BEL US-fu AverageARM 4.06 3.36 4.05 3.60 3.54 3.58 3.70 3.65 3.70 3.69Team 4.24 4.34 4.66 4.28 4.26 4.10 4.43 4.18 4.31 4.01 4.40 4.29Host 4.29 3.83 4.29 4.14 3.57 4.40 4.71 5.00 4.14 3.91 4.71 4.27

H.b.f-u 4.28 4.20 4.24

Table 4: Feedback on recent missions

Conclusion 24: The quality of the ARM needs substantial improvement, for that purpose the hosts may need further guidance.

As seen from the table in all missions so far evaluated the effectiveness was judged by the team at or above 4 in a scale of five thus reflecting a fairly high satisfaction. In contrast to the uniformity of the team feedback, the opinion of the hosts substantially varies from mission to mission.

Conclusion 25: The contradicting results of the two feedbacks suggest the need for further analysis.

e) IRRS experience of the team members

We consider an initial mission team expertise optimum if at least half and no more than 2/3 of the reviewers have previous IRRS experience. In follow-up missions this range is from 2/3 to 1.

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

SPA GFR UKR CAN RUS CPR US ROM ROK SLO SWI SWE SLR FIN BUL CZR BEL

ratio

of m

embe

rs w

ith e

xper

ienc

e

Mission

Team Experience (initial missions)

0.00

0.20

0.40

0.60

0.80

1.00

1.20

FRA-fu UK-fu UKR-fu SPA-fu GFR-fu CAN-fu UK-fu2 RUS-fu US-fu

ratio

of m

embe

rs w

ith e

xper

ienc

e

Mission

Team Experience (follow-up missions)

0.00

0.20

0.40

0.60

0.80

1.00

1.20

SWI SWE SLR FIN BUL UK-fu2RUS-fu CZR BEL US-fu

N/0

.8 si

gma

Mission

Relative number of advance comments


Figure 23: IRRS experience of team members in past missionsFigure 23 shows the team experience values in the past missions and reflects that

in many of the initial missions the team experience is considerably below optimum; In follow-up missions in most cases the team experience is in the optimum range.

Conclusion 26: In the recruitment of the IRRS team more attention should be paid to the inclusion of sufficient number of experienced reviewers.

f) Extent and coverage of the Action Plan

As part of the Advance Reference Material the host country prepares an Action Plan that lists those actions which are deemed necessary. Three PI’s have been defined to cover this topic:

1. The number of items included into the Action Plan (Extent of Action Plan)2. Ratio of Action Plan items covered by mission findings (Coverage of Action Plan)3. Amount of findings that do not appear in the Action Plan (Beyond Action Plan Coverage)

Details on the analysis of these PIs are given in [8] and [12], the main characters are quoted here: the numbers of Action Plan items in various missions are very different; the coverage of Action Plan items largely varies by missions;

in the majority of missions the number of findings not covered by Action Plan items is in the optimum range.

Conclusion 27: More guidance might be expedient on preparation of Action Plans.

Conclusion 28: Reviewers need to be made aware of the importance of reviewing the Action Plan prepared by the host.

g) Balance of findings

Correlations among the various observations of a mission were discussed in Sections V.2 and VI.1. Based on the expected and partly experienced correlations discussed there, two PIs have been defined that describe the balance of Good Practices and Recommendations and the balance of Recommendations and Suggestions, respectively [8], [12]. From results of past missions one concludes:

most of the PI values on balance of Recommendations and Good Practices fall outside the optimum range (although all but one are within the acceptable range);

most of the PI values on balance of Recommendations and Suggestions fall into the optimum range, some of them are in the acceptable range and only one is outside these ranges

Conclusion 29: It would be beneficial to invite experienced IRRS reviewers to discuss the relevance and possible application in future missions of the balance of various observations.

h) Conciseness of Mission Report

A PI is defined to measure the extent how the actual mission reports cover all necessary topics. Figure 24 shows the ratio of topics not covered in recent missions. Accordingly only a few mission reports cover the report topics as foreseen by the Report Template in an optimum way, most reports are in the acceptable range, the contents of a few reports would need attention.

Figure 24: Conciseness of recent mission reports

Conclusion 30: Reviewers should be trained to use the IRRS Mission Report Template in a comprehensive manner.

0.00

0.05

0.10

0.15

0.20

0.25

0.30

ROM ROK SLO SWI SWE SLR FIN BUL CZR BEL

rela

tive

to a

n id

eal r

epor

t

Mission

Conciseness of the reports


i) Completion time of Mission Report

The target period of report completion is 90 days. On the other hand only a few mission reports have been completed within the optimum range and some reports were in the red range (above 120 days).

Conclusion 31: IAEA should investigate the possible ways of speeding up report completion.

VI.4 Efficiency and effectiveness of recent missions

The Performance Indicators shown in the previous section have been applied to the recent missions [8], [12]. The Average Measure of Deviation from Optimum was calculated for each mission. Figure 25 summarizes the overall efficiency and effectiveness of recent missions in terms of Average Measure of Deviation from Optimum.

Figure 25: Variation in time of the Average Measure of Deviation from

Optimum of missions

Conclusion 32: The efficiency and effectiveness of all missions, as demonstrated by the PIs proposed in this study fall into the effective range; the values, although fluctuating show an improving trend.

REFERENCES

[1] INTERNATIONAL ATOMIC ENERGY AGENCY, Integrated Regulatory Review Service (IRRS) Guidelines for the Preparation and Conduct of IRRS Missions, Services Series 23, IAEA, Vienna, May 2013.

[2] INTERNATIONAL ATOMIC ENERGY AGENCY, Fundamental Safety Principles, Safety Fundamentals No. SF-1, IAEA, Vienna (2006).

[3] INTERNATIONAL ATOMIC ENERGY AGENCY, Governmental, Legal and Regulatory Framework for Safety, General Safety Requirements Part 1, No. GSR Part 1, IAEA, Vienna (2010).

[4] INTERNATIONAL ATOMIC ENERGY AGENCY, Management System for Facilities and Activities, Safety Requirement Series No. GS-R-3, IAEA, Vienna (2006).

[5] INTERNATIONAL ATOMIC ENERGY AGENCY, Radiation Protection and Safety of Radiation Sources: International Basic Safety Standards, General Safety Requirements Part 3, No. GSR Part 3 (Interim Edition), IAEA, Vienna (2011).

[6] INTERNATIONAL ATOMIC ENERGY AGENCY, Preparedness and Response for Nuclear and Radiological Emergencies, Safety Requirement Series No. GS-R-2, IAEA, Vienna (2002).

[7] INTERNATIONAL ATOMIC ENERGY AGENCY, Analysis of IRRS Missions in 2006-2013 to Countries with Operating NPPs, Working Document, IAEA, Vienna, (2014).

[8] INTERNATIONAL ATOMIC ENERGY AGENCY, Efficiency and Effectiveness of the IRRS Missions, Working Document, IAEA, Vienna, (2014).

[9] INTERNATIONAL ATOMIC ENERGY AGENCY, Safety of Research Reactor, Safety Requirements Series No. NS-R-4. IAEA, Vienna (2005).

[10] INTERNATIONAL ATOMIC ENERGY AGENCY, Safety of Nuclear Fuel Cycle Facilities, Safety Requirements Series No. NS-R-5, IAEA, Vienna (2008).

[11] INTERNATIONAL ATOMIC ENERGY AGENCY, Regulatory Inspection of Nuclear Facilities and Enforcement by the Regulatory Body, Safety Guide Series No. GS-G-1.3, IAEA, Vienna (2002).

[12] INTERNATIONAL ATOMIC ENERGY AGENCY, Lessons Learned from the IRRS Missions to Countries with Operating NNPs, 2006-2013, Technical Report prepared for the IAEA Workshop on “Lessons Learned from the Integrated Regulatory Review Service (IRRS)”, IAEA, Vienna (2014)

0.000

0.020

0.040

0.060

0.080

0.100

0.120

0.140

0.160

0.180

0.200

0.220

ROK GFR-fu SLO SWI CAN-fu SWE SLR FIN BUL UK-fu2 RUS-fu CZR BEL US-fu

Dist

ance

fro

m o

ptim

um

Mission

Efficiency and effectiveness of IRRS missions

to analyse

acceptable

effective

optimum
