Prompt Evaluation of the IRRS Mission to Bulgaria Documents/OPEN Shared... · Web viewOn the Efficiency and Effectiveness of IRRS Missions, Draft v6, Working Material, IAEA, Vienna,

Prompt Evaluation of the IRRS mission to Bulgaria P a g e | 0

LUX, Ivan

IAEA

4/30/2013

2013Prompt Evaluation of the IRRS Mission to Bulgaria

Prompt Evaluation of the IRRS mission to Bulgaria Page | 1

Table of Contents1. Introduction...............................................................................................................................2

2. Basic Mission Data.....................................................................................................................2

3. Feedback from the Team Members on the Advance Reference Material of the IRRS Mission..2

Characteristic additional comments..............................................................................................3

Further specific remarks................................................................................................................3

General conclusions.......................................................................................................................3

4. Feedback from the Team Members on the Effectiveness of the Mission..................................3

Characteristic additional comments..............................................................................................4

Further specific remarks................................................................................................................4

General conclusion........................................................................................................................4

5. Feedback from the Host Country on the Effectiveness of the Mission......................................5

6. Overall Evaluation of the Effectiveness and Efficiency of the Mission.......................................5

Discussion and conclusions............................................................................................................7

7. Summary....................................................................................................................................7

Appendix 1: Module-wise Coverage in the Mission Report...............................................................8

Appendix 2: Module-wise Coverage of the Action Plan...................................................................10


1. Introduction

The Contribution Agreement ENER/11/NUCL/SI2.588650 between the European Atomic Energy Community (represented by the European Commission) and the IAEA among others foresees as an expected result a performance monitoring based on the evaluation of some key performance indicators of the IRRS missions.

The Nuclear Safety Action Plan of the IAEA has as one of its main objectives the requirement to “Strengthen IAEA peer reviews in order to maximize the benefits to Member States” and in specific, it calls the IAEA Secretariat to review the effectiveness of the IRRS peer reviews.

In reply to these requirements a system of performance indicators have been elaborated in order to measure the efficiency and effectiveness of the IRRS missions. Some of the performance indicators are based on direct feedbacks from the IRRS team members and from the representatives of the country hosting the IRRS mission.

Results of and conclusions from the feedbacks and efficiency and effectiveness evaluations are given in the present Prompt Evaluation Report.

2. Basic Mission Data

Host country: BulgariaHost organization: BNRAMission date: 8-19 April 2013Team Leader: Marta ŽIAKOVÁ (SLR)Deputy Team Leader: HIOKI Kazumasa (JPN) Team Coordinator: Jean-René JUBIN (IAEA NSNI)Deputy Team Coordinator: Teodros HAILU (IAEA NSRW)Number of external experts: 17Number of IAEA staff: 6Number of observers: 3Mission type: full scopeScope of the mission:

Core modules (No. 1 through 10) Facilities and activities: NPP, fuel cycle facilities, radioactive waste facilities, radioactive

sources facilities, decommissioning activities, transport activities Additional areas (Module 11): control of medical exposures; occupational radiation

protection; control of discharges and materials for clearance, environmental monitoring for public radiation protection

Fukushima module Policy issues: (1) Long term operation of nuclear power plants; (2) NORM legislation and

practices

3. Feedback from the Team Members on the Advance Reference Material of the IRRS Mission

The Advance Reference Material (ARM) normally includes the results of the self-assessment of the host country (usually performed with the aid of the Self-Assessment Tool – SARIS); the Action Plan for improvement in issues found in the self-assessment, a module-wise summary of the status and activity of the regulatory body reviewed and a number of other documents needed for an objective and well informed peer review.

The team members are requested to offer their opinions on the quality of the ARM by rating from 1 to 5 (5 reflecting the highest satisfaction) three questions and also expanding in free text their


comments on the questions. The questions and the average values of the team members’ ratings are given below:

No. Question Average markQr1 How complete do you consider the ARM? 3.8

Qr2 How realistic picture could you obtain on the area you will be reviewing from the ARM?

3.4

Qr3 What is your overall evaluation on the quality of the ARM? 3.7

Total average 3.6

Characteristic additional comments

Qr1: The ARM was considered reasonably complete although some documents were provided

very late, or missing (on organizations other than BNRA and facilities other than NPP); All important documents should be provided in the ARM even if not available in English; The self-assessment answers were clear, however module-wise summaries and a general

discussion would have been helpful .Qr2:

The ARM contained quotations from the legislation; little was said about practical implementation. Specific examples would also be needed;

Interviews changed the picture obtained from ARM; Self-assessment seemed to be “defensive”, trying to justify practice.

Qr3: Good documents, the enclosed reports were useful, yet some condensation, summaries and

country profile would have been needed.

Further specific remarks

This questionnaire should be filled in at the end of the first week when experience is available on the truthfulness of the ARM.

General conclusions

1) The ARM was generally well received, some documents were provided too late;2) Module-wise summaries and general information should be part of the ARM;3) The self-assessment should also reflect the practical implementation of the legal

requirements.

4. Feedback from the Team Members on the Effectiveness of the Mission

The team members are requested to offer their opinions on the effectiveness of the mission by rating from 1 to 5 (5 reflecting the highest satisfaction) five questions and also expanding in free text their comments on the questions. The questions and the average values of the team members’ ratings are given below:

No. Question Average markQt1 How effective do you consider the activity of the expert team during the

mission?4.0

Qt2 How effective do you consider the activity of the IAEA staff in the team during the mission

4.8

Qt3 How effective do you consider the activity of the Team Leader? 3.9

Qt4 How effective do you consider the activity of the Deputy Team Leader? 4.0

Qt5 How satisfied are you with the preparations of the mission (enough time for 3.7


preparing yourself, information provided by the IAEA, etc.)?Total average 4.1

Characteristic additional comments

Qt1:

The need for explaining the process to novice experts was underestimated; Team meetings were not always effective; More coordination among experts dealing with similar areas is needed; Good team spirit, well focussed work; More time is needed for discussions with the counterparts.

Qt2:

The staff was experienced, effective and always helpful; Excellent, reliable support, impressive contributions; IAEA should consider reducing the staff by one; It was not clear who is in charge, TC acted as if TL.

Qt3:

The TL played her role in an efficient way; Summarizing the daily observations would have helped the preparation of the report; More input in the report development would have been appreciated; Managed the team activities effectively, was particularly effective in reaching consensus; Should have listened to others a little more.

Qt4:

The DTL was not very visible for the team members; He effectively coordinated with the TL; Was highly effective when he had occasions to lead the team; Was heavily involved in review and drafting, more involvement in the team management

would have been appreciated.

Qt5:

There was not sufficient time to read the ARM; A means to interact with the hosts to clarify questions on ARM prior to the missions should

be provided; The report template was helpful; IAEA should ensure that the information package provided by the host is up to date; IAEA staff was very effective in providing information.

Further specific remarks

Methodological training prior to the mission is needed for the first time experts; More instructions for the preparation phase would be beneficial; Several parallel duties of the team members (reviewing, writing, coordinating, cross-reading,

discussing) result in lack of time and quality For submission of the first findings (for Wednesday, first week) a formalised means should

be developed; Cross-reading is a good practice, it should be still organized and formalized.

General conclusion

1) The team was effective, although more training and instruction would have been needed;


2) Timing of the mission and of the various activities by the experts should be better explained to the team;

3) The IAEA staff was well prepared, effective, supportive;4) Roles of the TL and the DTL should be better defined;5) The report template and other forms assisting the reviewers should be further developed;6) In the preparatory phase more instructions and guidance is to be given to the team on their

expected roles and inputs;7) IRRS training is needed for inexperienced reviewers.

5. Feedback from the Host Country on the Effectiveness of the Mission

The Liaison Officer of the host country is requested to offer the opinion of the host on the effectiveness of the mission by rating from 1 to 5 (5 reflecting the highest satisfaction) seven questions and also expanding in free text their comments on the questions. The questions, the host country’s ratings and the associated comments are given below:

No. Question MarkQh1 How effective do you consider the mission in assisting the continuous

improvement of nuclear safety in your country?5

Qh2 How objective was the peer review? 4

Qh3 How has the mission helped the exchange of information, experience and good practice with other countries?

4

Qh4 How consistent was the use of the IAEA safety requirements and guides in the mission?

5

Qh5 How justified are the findings of the peer review? 4

Qh6 How relevant are the findings of the peer review for the future development of your regulatory body?

4

Qh7 How competent were the reviewers in their review and findings? 5

Average 4.4

Further comments:

The experts should consider giving priority to IAEA Safety Standards rather than insisting on national practice;

Prior to recruitment, IAEA might consider verifying the expert’s professional competence, communication and language skill;

Optimization of the number of experts in the team should be considered.

6. Overall Evaluation of the Effectiveness and Efficiency of the MissionThe Performance Indicators developed for the measurement of the effectiveness and efficiency of an IRRS mission were evaluated in the extent as it was made possible by the data available at the time of the present evaluation.

In the next figure the values of the performance indicators as they follow from the evaluation of the mission data1, as well as the overall effectiveness of the mission are presented. The rightmost columns (EFF. INDICATION) present the ranges where the particular PIs fall (green – optimum, yellow – acceptable, red – needing attention), whereas the frame in the right lower part summarizes the overall effectiveness of the mission (green – optimum, white – effective, yellow – acceptable, red – to analyse).

1 On the Efficiency and Effectiveness of IRRS Missions, Draft v6, Working Material, IAEA, Vienna, 2013


The bar diagram in the figure shows the number ratio of issues covered by the mission report to those foreseen by the Standard Mission Report Template. Details on this coverage are given in Appendix 1.

The coverage of the Action Plan items (defined by the host country) by Recommendations and Suggestions by the reviewers of the various Modules is shown in the figure in Appendix 2.

Prompt Evaluation of the IRRS mission to the Slovak Republic P a g e | 7

Efficiency and Effectiveness of an IRRS Mission initial

INPUT QUANTITIES valueA priory mission data No(id) Unit Value min.(>) max.(≤) min.(>) max.(≤) col PI

Type of mission (1 - initial or 2 - follow-up) 1 optimum 0.0 C1(T) - 22 20 24 18 27 0.00 Team sizeHost country Bulgaria effective 0.1 C2(P) - 123 118 144 104 157 0.00 Report lengthStating time of the mission (yyyy-mm-dd) 2013-04-08 acceptable 0.2 C3(tARM) day 76 45 - 30 45 0.00 ARM review timeEnding time of the mission (yyyy-mm-dd) 2013-04-19 needs attention C4(NAWC) - 4 12 - 9 12 0.30 Advance commentsNumber of modules (nT) 13 C5(fbARM) - 3.64 4.00 5.00 3.00 4.00 0.09 Feedback on ARMNumber of supplementary modules (in follow-up) (nM+) T 0-first -0.96 C6(Exp) - 0.24 0.50 0.66 0.33 1.00 0.30 Team experienceNumber of facility and activity types (nx) 6 λ T-first 1.48 C7(fbHost) - 4.43 4.00 5.00 3.00 4.00 0.00 Host feedbackNumber of additional technical areas (nT) 3 T 0-fu 3.09 C8(fbTeam) - 4.09 4.00 5.00 3.00 4.00 0.00 Team feedbackNumber of policy discussion issues (nP) 2 λ T-fu 0.71 C9(β0) - 1.20 1.00 - 0.80 1.00 0.00 Action Plan ExtentNumber of NPP units in the host country (ν) 4 C10(β1) - 0.71 1.00 - 0.80 1.00 0.29 Action Plan coverage

Pre-mission data P 0-first 17.60 C11(β2) - 1.46 1.00 - 0.80 1.00 0.00 Beyond AP coverageNumber of Team Members (including IAEA) (T) 22 λ M-first 5.83 C12(ρ1) - 0.93 0.75 1.25 0.50 1.50 0.00 Balance of findingsNumber of Experts in the Team (without IAEA) (Te) 17 λ F-first 0.63 C13(ρ2) - 0.70 0.75 1.25 0.50 1.50 0.06 Balance of R & SNumber of Experts with IRRS experience (Tx) 4 P 0-fu 54.20 C14(RCont) - 0.81 0.90 1.00 0.80 0.90 0.10 Report concisenessARM distribution date (yyy-mm-dd) 2013-01-22 λ M-fu 1.55 C15(RTime) day 47 0 90 90 120 0.00 Report completionNumber of issues in the Action Plan (NAP) 28 λ F-fu 0.64 C16(OpenI) - - 0.00 0.20 0.20 0.40 - - Open issues leftNumber of advanced ARM comments from TM's (NAWC) 4 μ 0.90Number of findings (R+S) in the first mission (for follow-up)

Mission data (R+S) av 39Number of Recommendations (R) 15 R av 15.1Number of Suggestions (S) 34 G av 12.3Number of Good Practices (G) 9 (R*G) av 157.4Number of findings also in the Action Plan (N=) 10 (R/S) av 0.89Number of issues remained open (in follow-up)Number of Report Pages (P) 123 Size of nucl.pr. 0.8Conciseness of the Mission Report (ρ) 0.81 Mission size 15.6

Questionnaire data Opt.team size 22.1

Average mark on ARM quality 3.64 Opt.rep.length 131 EFF.& EFF. OF THE MISSION TOAverage mark by the host country 4.43 Mission type:Average mark by the Team Members 4.09 Missions duration:

Post-mission data Average Measure of Deviation (Δ):Date of isuance of mission report (yyyy-mm-dd) 2013-06-06 Overall Effectiveness:

April 2013

fields to input! effective

EFF. INDICATIONRel.Dist from Opt.

Optimum rangePERFORMANCE INDICATORS Acceptable range

2013-04-08

For report length

For findings

0.076

initial2013-04-19

CALCULATED DATA

IAEA, Version 4

Bulgaria

Bulgaria

LIMITS & PARAMETERSFor average deviation

For team size

0.000.100.200.300.400.500.600.700.800.901.00

I II III IV V VI VII VIII IX X XI

Module-wise report conciseness

Prompt Evaluation of the IRRS mission to the Slovak Republic P a g e | 8

Discussion and conclusions

The values of the effectiveness and efficiency Performance Indicators (upper right table in the figure headed by EFF. INDICATION) suggest the following conclusions:

1) The size of the team and the length of the report were near the optimum values;2) Although the main body of the ARM was provided two and a half months prior to the

mission, several experts complained of lack of time or mission materials;3) The team was very weak in providing advance comments (and also the few advance

comments arrived in the last days);4) Less than 25% of the experts in the team have prior IRRS experience as reviewers (the

respective PI is in the red), this was also reflected in their comments on too little instructions and certain difficulties in communication with the host counterparts. In spite of that, most of the novice experts could well cope with their tasks;

5) The feedback on the ARM was in the yellow range, the actual comments explain the reasons;

6) The team in general was satisfied with the effectiveness of the mission, yet provided suggestion on further possible developments;

7) The host appreciated rather high the mission, also with suggestions;8) The Action Plan provided by the host was not sufficiently taken into account by the team,

thus its coverage by mission findings is rather poor;9) The balance of recommendations and good practices is nearly optimum, that of the

recommendations and suggestions shows a slight shift towards suggestions;10) The mission report generally covers in sufficient details the subjects to review, yet as seen in

the bar diagram in the figure above, several aspects of the authorization area do not appear in the report, while slight shortages in the report on inspection and regulation and guides modules have been identified;

11) The overall effectiveness of the mission is in the white (effective range) and its value (0.076) is in harmony with other recent missions.

7. Summary

The data of and feedback from the full scope first IRRS mission to Bulgaria have been analysed. The following conclusions are drawn:

The ARM was generally well received, some documents were provided too late; Module-wise summaries and general information should be part of the ARM; The self-assessment should also reflect the practical implementation of the legal

requirements; The team was effective, although more training and instruction would have been needed; Timing of the mission and of the various activities by the experts should be better explained

to the team; The IAEA staff was well prepared, effective, supportive; The team included experienced reviewers in smaller proportion than desirable; Roles of the TL and the DTL should be better defined; The team has shown strength in effectively performing their tasks and providing results in

time; The team has shown weakness in providing advance opinion and in assessing the Action Plan

of the host; Certain parts of the mission report are less comprehensive than they should be; The host appreciated the results of the missions and provided valuable suggestions as for

increasing further the effectiveness and efficiency of the IRRS missions; The mission was rated effective by the evaluated Performance Indicators.


Appendix 1: Module-wise Coverage in the Mission Report



Appendix 2: Module-wise Coverage of the Action Plan

0.000.100.200.300.400.500.600.700.800.901.00

II III IV V VI VII IX X

Modules

AP coverage by Modules

The figure shows the ratio of the number of Recommendations and Suggestions offered by the mission team to the number of Action Plan items defined by the hosts for the various IRRS Modules.

Note that no Action Plan items were determined for Modules I, VIII and XI, therefore these modules do not appear in the figure.