Upload
others
View
4
Download
0
Embed Size (px)
Citation preview
1
1
CONTROL CHARTS TO EVALUATELONG-TERM PERFORMANCE
IN PROFICIENCY TESTS
Angela Sorbo
European Union Reference Laboratory for Chemical Elements in Food of Animal Origin (EU-RL CEFAO)
EURACHEM Workshop October 10-11, 2012
2
CONTENTS
ROLE OF THE EUROPEAN UNION REFERENCE LABORATORIES
EURL CEFAO AS PT PROVIDER AND RATIONALE FOR ITS SCHEME
CONCLUSIONS
PERFORMANCE EVALUATION: Z-SCORES USING “FITNESS FOR PURPOSE APPROACH”
CONTROL CHARTS OF Z-SCORES
EURACHEM Workshop October 10-11, 2012
This work has been produced with the financial support of the SANCO 2005/Food SAFETY/003-Residues Program of the European Commission. The contents of this presentation are the sole responsibility of the author and can in no way be taken to
reflect the views of the European Commission
2
3
COMMUNITY REFERENCE LABORATORIES FOR RESIDUESOF VETERINARY MEDICINES AND CONTAMINANTS INFOOD OF ANIMAL ORIGIN WAS DESIGNATED IN COUNCILDIRECTIVE 96/23/EC AND REFIXED IN COMMISSIONREGULATION (EC) No 776/2006 of 23 May 2006 AMENDINGANNEX VII TO REGULATION (EC) No 882/2004
Art. 12 CR (EC) 776/2006 ASSIGNED TO THE ISTITUTOSUPERIORE DI SANITÀ (ROME) THE COMMUNITYREFERENCE LABORATORY FOR THE RESIDUES LISTED INANNEX I, GROUP B 3(C) TO DIR. 96/23/EC
EURL (FORMER CRL) IS RESPONSIBLE FOR CHEMICALELEMENTS IN FOOD OF ANIMAL ORIGIN (EU-RL CEFAO)
ROLE OF THE EUROPEAN REFERENCE LABORATORIES
EURACHEM Workshop October 10-11, 2012
4
Tasks of the CRLs (Art. 32-Reg. 882/2004)
a) providing national reference laboratories with details ofanalytical methods, including reference methods
b) coordinating the national reference laboratories in theapplication of the methods referred to in (a), in particular byorganising comparative testing and by ensuring anappropriate follow-up of such comparative testing inaccordance with internationally accepted protocols, whenavailable
c) coordinating, within their area of competence, practicalarrangements needed to apply new analytical methods andinforming national reference laboratories of advances in thisfield
d) conducting initial and further training courses for the benefit ofstaff from national reference laboratories and of experts fromdeveloping countries
ROLE OF THE EUROPEAN REFERENCE LABORATORIES
EURACHEM Workshop October 10-11, 2012
3
5
MATRICES, ANALYTES AND LEVEL OF CONCENTRATIONFOR PTs ARE SELECTED TAKING INTO ACCOUNT BOTHTHE RELEVANT EUROPEAN REGULATION (EXISTINGMAXIMUM LEVELS) AND NATIONAL RESIDUES CONTROLPLANS
THE CONCENTRATION OF THE SAMPLES IS OFTENADJUSTED TO THIS AIM
EU-RL CEFAO IS ACCREDITED AS PT PROVIDERACCORDING TO ISO/IEC 17043:2010 (BEFORE 2010: ISO GUIDE
43-1)
THE LABORATORY IS ALSO ACCREDITED IN CONFORMITYWITH ISO/IEC 17025:2005, ENLARGED WITH FLEXIBLESCOPE
EURL CEFAO AS PT PROVIDER AND RATIONALE FOR ITS SCHEME
EURACHEM Workshop October 10-11, 2012
6
2006 2007 2008 2009 2010 2011Meat
freeze-dried
2 samples
Meat freeze-dried
1 sample
Liverfreeze-dried
1 sample
Fishfreeze-dried
1 sample
MeatFrozen1 sample
LiverFrozen
1 sample
Milk freeze-dried
2 samples
MilkLiquid
2 samples
MilkLiquid
2 samples
MilkLiquid
2 samples
Milk Liquid1 sample
Meat freeze-dried
1 sample
Fish freeze-dried
1 sample
Meat freeze-dried
1 sample
FishFrozen1 sample
Analytes: Cadmium (Cd), Lead (Pb), total Arsenic (As) and Mercury (Hg)
PROFICIENCY TESTS ORGANIZED IN THE FIVE-YEAR PERIOD 2006-2011
EURACHEM Workshop October 10-11, 2012
EURL CEFAO AS PT PROVIDER AND RATIONALE FOR ITS SCHEME
4
7
EUROPEAN FOOD CONTROL NETWORK
OFFICIAL LABORATORIES
NATIONAL REFERENCE LABORATORIES
EUROPEAN REFERENCE LABORATORIES
PROFICIENCY TESTS ORGANIZED BY EU-RL CEFAO AREADDRESSED TO THE NATIONAL REFERENCELABORATORIES (NRLs) OF THE EU MEMBER STATES
NETWORK IS NOW CONSTITUTED BY 27 PARTICIPANTS
EURACHEM Workshop October 10-11, 2012
EURL CEFAO AS PT PROVIDER AND RATIONALE FOR ITS SCHEME
8
TO IMPROVE THE PERFORMANCE OF THE NRLsANALYTICAL METHODS AROUND THE MLs
TO HARMONIZE THE INTERPRETATION OFRESULTS IN TERMS OF THE SAMPLEACCEPTANCE (CR (EC) 333/2007)
TO ASSIST NRLs IN FINDING THE CAUSE OFANALYTICAL PROBLEMS LEADING TO ANUNSATISFACTORY PERFORMANCE
GENERAL OBJECTIVES OF PTs according to EU-RL TASKS
EURACHEM Workshop October 10-11, 2012
EURL CEFAO AS PT PROVIDER AND RATIONALE FOR ITS SCHEME
5
9
“FITNESS FOR PURPOSE” APPROACH PROPER FOR THE ROLE OF NRLs
PERFORMANCE REQUIRED FROM NRLS SHOULD BEBETTER THAN THAT OF ROUTINE CONTROLLABORATORIES
STANDARD DEVIATION FOR PROFICIENCYASSESSMENT (pEU-RL CEFAO) LOWER THAN THE ONEDERIVED FROM HORWITZ EQUATION
EU-RL CEFAO HAS DEVELOPED PROPER EQUATIONSGIVING A p CORRESPONDING TO THEPERFORMANCE LEVEL THAT NRLs ARE EXPECTEDTO ACHIEVE FOR THE DIFFERENT COMBINATIONMATRIX/ANALYTE/CONCENTRATION LEVEL
PERFORMANCE EVALUATION
EURACHEM Workshop October 10-11, 2012
10
MEAT AND FISH
Cd
0
10
20
30
40
10 70 130 190 250 310 370 430 490 550 610
%
120
g/kg
p =10% → C > 601
Pb
0
10
20
30
40
20 120 220 320 420 520 620 720 820
%
g/k
p =10% → C > 801
ii
p
EURACHEM Workshop October 10-11, 2012
PERFORMANCE EVALUATION
Thompson
Horwitz
EU-RL CEFAO
6
11
As
0
10
20
30
40
120 180 240 300 360 420 480 540
%
g/kg
C < 120 →p =22% p =14.5% → C > 501
MEAT AND FISH
EURACHEM Workshop October 10-11, 2012
PERFORMANCE EVALUATION
0
10
20
30
40
30 160 290 420 550 680 810
%
Hg
g/kg
C < 30 →p =18%
p =10% → C > 801
p
Thompson
Horwitz
EU-RL CEFAO
12
Cd
0
10
20
30
40
50
1 2 3 4 5 6 7 8 9 10 11
%
g/kg
p =15% → C > 11
Pb
0
10
20
30
40
10 15 20 25 30 35 40 45 50 55
%
g/kg
C < 10 →p =30%
p =18% → C > 51
As
0
10
20
30
40
10 15 20 25 30 35 40 45 50 55
%
g/kg
C < 10 →p =40%
p =15% → C > 51
MILK
EURACHEM Workshop October 10-11, 2012
PERFORMANCE EVALUATION
p
Thompson
Horwitz
EU-RL CEFAO
7
13
Shewhart Control Chart for z-scores(ISO 13528:2005*, Par. 9.2.1)
To prepare this charts the z-scores for a laboratory areplotted as individual points, with action and warninglimit set at ±2.0 and ±3.0 (…). When severalcharacteristics are measured in each round, the z-scores for different characteristics may be plotted onthe same graph (…)
The Shewart control chart is an effective method ofidentifying problems that cause large erratic values ofz-scores
* ISO 13528:2005: Statistical methods for use in proficiency testingby interlaboratory comparisons
CONTROL CHARTS OF Z-SCORES
EURACHEM Workshop October 10-11, 2012
14
Rules(ISO 13528:2005*, Par. 9.2.1)
An out of control signal is given when:
b) two out of three successive points fall outside the same warning limit (±2,0)
a) a single point falls outside the action limits (±3,0)
* ISO 13528:2005: Statistical methods for use in proficiency testingby interlaboratory comparisons
EURACHEM Workshop October 10-11, 2012
CONTROL CHARTS OF Z-SCORES
8
15
EURACHEM Workshop October 10-11, 2012
CONTROL CHARTS OF Z-SCORES
16
EURACHEM Workshop October 10-11, 2012
CONTROL CHARTS OF Z-SCORES
9
17
EURACHEM Workshop October 10-11, 2012
CONTROL CHARTS OF Z-SCORES
18
-4
-3
-2
-1
0
1
2
3
4
2006Meat A
2006Meat B
2007Meat
2007Fish
2008Liver
2008Meat
2009Fish
2010Fr. Meat
2010Fr. Fish
2011Fr. Liver
2011Meat
z-s
core
PT
Cd
Pb
z=±2
z=±3
-6.2
CC Meat, Fish and Offal
2009: Visit of the EU-RL CEFAO representatives
2010: Change of analytical technique
EURACHEM Workshop October 10-11, 2012
CONTROL CHARTS OF Z-SCORES
10
-4
-3
-2
-1
0
1
2
3
4
2006Meat A
2006Meat B
2007Meat
2007Fish
2008Liver
2008Meat
2009Fish
2010Fr. Meat
2010Fr. Fish
2011Fr. Liver
2011Meat
z-sc
ore
PT
As
Cd
Pb
Hg
z=±2
z=±3
CC Meat, Fish and Offal
19
2010: Change of staff and technique (from more to less sensitive!)
EURACHEM Workshop October 10-11, 2012
CONTROL CHARTS OF Z-SCORES
-4
-3
-2
-1
0
1
2
3
4
10th-2A 10th-2B 11th-1A 11th-1B 12th-1 A 12th-1 B 13th-1 A 13th-1 B 14th-2 16th
z-s
co
re
PT
Cd
Pb
z=±2
z=±3
CC Milk
20
2010: Change of technique (from less to more sensitive!)
EURACHEM Workshop October 10-11, 2012
CONTROL CHARTS OF Z-SCORES
11
21
EU-NATIONAL REFERENCE LABORATORIES CONTROL CHART OF THE z-scores
(2006-2012) MILK
Edited by: A. Colabucci Verified by: L. Ciaralli Issued: 19.07.2012
EURACHEM Workshop October 10-11, 2012
CONTROL CHARTS OF Z-SCORES
22
CC Milk
-4
-3
-2
-1
0
1
2
3
4
2006-A 2006-B 2007-A 2007-B 2008-A 2008-B 2009-A 2009-B 2010 2012
z-s
core
PT
As
Cd
Pb
z=±2
z=±3
-4
-3
-2
-1
0
1
2
3
4
2006-A 2006-B 2007-A 2007-B 2008-A 2008-B 2009-A 2009-B 2010 2012
z-sc
ore
PT
As
Cd
Pb
z=±2
z=±3
Change of the staff in a laboratory with high standard of QS: excellent and steady performance
Change of staff: improved performance
EURACHEM Workshop October 10-11, 2012
CONTROL CHARTS OF Z-SCORES
12
-4
-3
-2
-1
0
1
2
3
4
2006-A 2006-B 2007-A 2007-B 2008-A 2008-B 2009-A 2009-B 2010 2012
z-sc
ore
PT
As
Cd
Pb
z=±2
z=±3
CC Milk
Steady performance in spite of no participation in PTs for 3 years and the change of the sample physical state
23
Analytical method under control
EURACHEM Workshop October 10-11, 2012
CONTROL CHARTS OF Z-SCORES
-4
-3
-2
-1
0
1
2
3
4
2006 Meat-A
2006 Meat-B
2007 Meat
2007 Fish
2008 Meat
2008 Liver
2009 Fish
2010 Fr. Meat
2010 Fr. Fish
2011 Fr. Liver
2011 Meat
z-sc
ore
PT
As
Cd
Pb
Hg
z=±2
z=±3
-4
-3
-2
-1
0
1
2
3
4
2006 Meat-A
2006 Meat-B
2007 Meat
2007 Fish
2008 Meat
2008 Liver
2009 Fish
2010 Fr. Meat
2010 Fr. Fish
2011 Fr. Liver
2011 Meat
z-sc
ore
PT
As
Cd
Pb
Hg
z=±2
z=±3
24
EURACHEM Workshop October 10-11, 2012
CONTROL CHARTS OF Z-SCORES
CC Meat, Fish and Offal
pEU-RL CEFAO
pHORWITZ/THOMPSON
13
25
The NRL was advised to accredit the method based on the most sensitive technique
-4
-3
-2
-1
0
1
2
3
4
2006-A 2006-B 2007-A 2007-B 2008-A 2008-B 2009-A 2009-B 2010 2012
z-sc
ore
PT
-4
-3
-2
-1
0
1
2
3
4
2006-A 2006-B 2007-A 2007-B 2008-A 2008-B 2009-A 2009-B 2010 2012
z-sc
ore
PT
Dry ashing + FAAS
MAD + ICP-MS
Pb
Pb
Cd
AsCd
As
EURACHEM Workshop October 10-11, 2012
CONTROL CHARTS OF Z-SCORES
26
CUSUM Control Chart for z-scores(ISO 13528:2005*, Par. 9.3.1)
To prepare this chart, the cumulative sum (cusum) of
the z-scores achieved by a laboratory over severalrounds of a PT is calculated and plotted.
The CUSUM control chart is an effective method ofidentifying problems that cause a bias in thedetermination of a characteristic that persists overseveral rounds.
* ISO 13528:2005: Statistical methods for use in proficiency testingby interlaboratory comparisons
EURACHEM Workshop October 10-11, 2012
CONTROL CHARTS OF Z-SCORES
14
27
EURACHEM Workshop October 10-11, 2012
-10
-8
-6
-4
-2
0
2
4
6
8
10
2006-A 2006-B 2007-A 2007-B 2008-A 2008-B 2009-A 2009-B 2010 2012
z-sc
ore
PT
As
Cd
Pb
THE ‘PERFECT’ CUSUM CHART
CONTROL CHARTS OF Z-SCORES
-10
-8
-6
-4
-2
0
2
4
6
8
10
2006MEAT-A
2006MEAT-B
2007MEAT
2007FISH
2008MEAT
2008LIVER
2009FISH
2010MEAT
2010FISH
2011LIVER
2011MEAT
cu
su
m o
f z-
sc
ore
s
PT
As
Cd
Pb Cusum CC
28
-4
-3
-2
-1
0
1
2
3
4
2006Meat A
2006Meat B
2007Meat
2007Fish
2008Liver
2008Meat
2009Fish
2010Fr. Meat
2010Fr. Fish
2011Fr. Liver
2011Meat
z-sc
ore
PT
As
Cd
Pb
z=±2
z=±3Shewart CC
Meat, Fish and Offal
EURACHEM Workshop October 10-11, 2012
CONTROL CHARTS OF Z-SCORES
Pb
-10
-8
-6
-4
-2
0
2
4
6
8
10
2006 Meat-A
2006 Meat-B
2007 Meat
2007 Fish
2008 Meat
2008 Liver
2009 Fish
2010 Fr. Meat
2010 Fr. Fish
2011 Fr. Liver
2011 Meat
PT
Cu
sum
of
z-sc
ore
s
15
29
EURACHEM Workshop October 10-11, 2012
-10
-8
-6
-4
-2
0
2
4
6
8
10
12
14
16
18
20
2006Meat-A
2006Meat-B
2007Meat
2007Fish
2008Meat
2008Liver
2009Fish
2010Fr. Meat
2010Fr. Fish
2011Fr. Liver
2011Meat
Cu
sum
of
z-sc
ore
s
PT
As
Cd
Pb
-4
-3
-2
-1
0
1
2
3
4
2006Meat A
2006Meat B
2007Meat
2007Fish
2008Liver
2008Meat
2009Fish
2010Fr. Meat
2010Fr. Fish
2011Fr. Liver
2011Meat
z-sc
ore
PT
As
Cd
Pb
z=±2
z=±3
Cusum CC
Shewart CC
CONTROL CHARTS OF Z-SCORES
30
CONCLUSIONS
The organization of recurrent PTs on the same matrix/analytecombination and at a similar level of concentration allowsparticipants to monitor the performance of their analyticalmethods and the PT provider to constantly check the qualitylevel of the network
The use of proper Control Charts is an effective means ofmonitoring the trends in the results produced by thelaboratory
The evaluation of CUSUM Charts together with ShewartControl Charts can be a useful way of highlighting bias in theanalytical methods
The analytical background of the Provider is an importantbasis to join all the information from the PT to assistparticipants in understanding their performance
EURACHEM Workshop October 10-11, 2012
16
31
EURACHEM Workshop October 10-11, 2012
Many thanks to dr. Rosa Giordano for the invaluable support