Guidance for water quality intercalibration

PP2 – National Administration Romanian WatersChem. Carmen Hamchevici Dr. biol. Gabriel Chiriac

Outline

• Terms and definitions needed• Objective• Methods

– General (widely used)– Specific (adjusted to needs)

• Results• Interpretation of results• Conclusions

Terms and definitions• Intercalibration

– Strictly metrological: (http://dictionary.reference.com/browse/intercalibration):

to determine, check, or rectify the graduation of any instrument giving quantitative measurements

– Extended• A state achieved by a group of laboratories engaged in a monitoring

program in which they produce and maintain compatible data outputs (McGraw-Hill Dictionary of Scientific & Technical Terms, 2003)

– Current use at the EU level given by the WFD Guidance Documents:• the intercalibration process is aimed at consistency and comparability

of the classification results of the monitoring systems operated by each Member State for the biological quality elements (CIS Guidance Document no. 14 – Guidance on the intercalibration Process 2004-2006)

active process between laboratories that includes all steps from sampling to analyses

achieving the same accurate results regardless of the method or lab

compatible data outputs are achieved and this situation is maintainedthe laboratories can be said to be intercalibrated

intercalibrationThe process, procedures and activities used to ensure that the several laboratories

engaged in a monitoring program can produce compatible data

Objective

Taylor, 1987

Achieving the objective (already done?!)

Quality Assurance

Quality System in place(accreditation acc to ISO/IEC 17025:2005)

Technical competence

Objective (partially) achieved

Methods (1)

• General– Proficiency testing (PT): the evaluation of participant performance against

pre-established criteria by means of interlaboratory comparisons (ILC)

– Interlaboratory comparisons (ILC): organization, performance and evaluation of measurements or tests on the same or similar items by two or more laboratories […] in accordance with predetermined conditions

– Other purposes of ILC than PT• to evaluate the performance characteristics of a method

• to characterise a reference material

• to compare results of two or more laboratories on their own initiative

• to support statements of the equivalence of measurement of National Metrology Institutes

Source: International Laboratory Accreditation Cooperation (ILAC) Policy for Participation in PT Activities (ILAC-P9:06/2014)

Proficiency Testing schemes (1)

• evaluation of the performance of laboratories and monitor

laboratories’ continuing performance

• identification of problems in laboratories and initiation of

corrective actions

• establishment of the effectiveness and comparability of

test or measurement methods

• provision of additional confidence to laboratory customers

• identification of inter-laboratory differences

Proficiency Testing schemes – who is involved?

• organization which takes responsibility for all tasks in the development and operation of a proficiency testing scheme

Provider

• laboratory, organization or individual that receives proficiency test items and submits results for review by the proficiency testing provider

Participant

Proficiency Testing schemes – use of analytical methods

• the laboratory should use its routine method

• the choice might be limited by e.g. legal regulations

• specified in the reporting sheet

• provider has a policy and follows a procedure regarding

comparison of results obtained by different methods

Proficiency Testing schemes – assigned value

• one of the most critical features of a PT

• inappropriate value will drastically reduce the value of the scheme

• 4.4.5.1 of the ISO/IEC 17043:2010: the PT provider shall document the procedure for determining the assigned values for the measurands or characteristics in a particular PT scheme

Assigned value (1)

• Reference Certified Material: ideal test material for a PT• high costs• limited availability

• Consensus of “Expert Laboratories“: mean of analysis by expert laboratories

• difficult to ensure the accuracy of the reference measurements• „nobody is perfect“: there might be doubts among the participants if

the result of the expert laboratories deviates from the mean of the participants

Assigned value (2)

• Formulated or “Synthetic” test materials• materials spiked with the analyte to a known extent• if material does not contain significant amounts of the analyte,

assigned value directly from added amount• difficult to achieve sufficient homogeneity, especially with

solid materials• analyte might be bound in a different chemical form

• Consensus of participants• easiest and cheapest, widely used• if method for analysis is easy - good estimate of „true“

value

• consensus value might be seriously biased• no consensus at all• these circumstances are frequent in trace analysis

Methods for calculating consensus value

• Arithmetic mean: requires an outlier test that assumes normal distribution (normally not true in trace analysis)

• Median value: – not affected by outlying data – makes not full use of the information content of the data

• Robust mean:– „trimmed“ data; a certain part of the data on both tails of the data set is

excluded prior to the calculation of the mean– e.g. mean of interquartile range (mean of data between the first and the

third quartile of the data set)– Huber statistics

Source: Koch, M.: Interlaboratory tests in Wenclawiak, Koch, Hadjicostas (eds.), 2003 Quality Assurance in Analytical Chemistry – Training and Teaching

Assessment of performance

• Basics: assigned value and standard deviation for proficiency assessment

• Means: z-score– Z – score =

• X – result reported by participant• – assigned value • – standard deviation for proficiency assessment

Source: Koch, M.: Interlaboratory tests in Wenclawiak, Koch, Hadjicostas (eds.), 2003 Quality Assurance in Analytical Chemistry – Training and Teaching

Performance criteria

│z│ ≤ 2,0 the score indicates “satisfactory” performance and generates no signal

2.0 < │z│ < 3.0 the score indicates “questionable” performance and generates a warning signal

│z│ ≥ 3.0 the score indicates “unsatisfactory” performance and generates an action signal

Example of z-score diagram

-15

-10

-5

0

5

10

75 39 40 36 34 20 71 56 3 85 2 59 43 25 54 90 1 80 44 76 64 29 79 73 15 7 57 32 37 72 52 10 17 95 63 49 4

laboratory code

Z-s

core

Source: Koch, M.: Interlaboratory tests in Wenclawiak, Koch, Hadjicostas (eds.), 2003 Quality Assurance in Analytical Chemistry – Training and Teaching

References used in PTs

• ISO/CEI 17043:2010 Conformity assessment – General requirements for proficiency testing

• ILAC-P9:06/2014: International Laboratory Accreditation Cooperation Policy for Participation in PT Activities

• ISO/IEC 17025:2005 General requirements for the competence of testing and calibration laboratories

• EURACHEM: Selection, use and interpretation of Proficiency Testing (PT) schemes, Second Edition 2011

• ISO 13528: 2005 Statistical methods for use in proficiency testing by inter-laboratory comparisons

• Specific – Transboundary QA/QC schemes: QUALCO-DANUBE the danubian

laboratories reporting to the Trans-National Monitoring Network of the

ICPDR

– Bilateral Agreements between neighboring countries

– Common or parallel actions during different types of activities: research

projects, investigative monitoring (i.e. Joint Danube Surveys)

Methods (2)

PT scheme Qualco-Danube (1)• established and implemented as the primary

inter-laboratory quality control program in the Danube basin

• started in 1993, extended in 1995 to 11 National Reference Labs (NRLs) and from 1996 to 19 NRLs

• Provider: (until 2012) the Institute for Water Pollution Control of VITUKI, Budapest, Hungary

• 2013 - present WESSLING (HU)• Quarterly distribution every year (until 2012)• 2013 – present: flexible (nutrients / every

year, heavy metals / every 2 years, organics / every 3 tears)

Sample preparation and evaluation scheme for AQC in the Danube river basin (Five-years Report on Water Quality in Danube River Basin Based on TransNational Montoring Network, 2003)

Selection of Determinands and Matrix

Real-world Sample(water/sediment)

In-house RM(concentrate/sediment)

Synthetic Sample(concentrate)

Preparation of AQC Samples

Homogenisation(testing homogeneity)

Distribution of SamplesSending Samples According to Schedule

Requesting Results by Strict Deadline

Evaluation of the Results(e.g. Youden-pairs, Z-score)

Initiate Actions ifPerformance is Poor

Evaluate MethodPerformance

Assigned Values forReference Material

Evaluate LaboratoryPerformance

In-house ReferenceMaterial Available

for AQC

Method Modification orReplacement if Needed

Follow-up(actions, utilization)

PT scheme Qualco-Danube - Assessment of performance

based on the tolerance intervals (until 2008)

2005/1 2006/1 2007/1 2008/1

-40

-30

-20

-10

0

10

20

30

40

14.2

-12

-1.7

0

9.3

-7

2.8

-0.90000000000000

1

Sample A Sample B Acceptable LimitAcceptable Limit Warning Limit Warning Limit

%

Z-score (2008-present)

NH4 NO3 NO2 PO4 PT

-4

-3

-2

-1

0

1

2

3

4

-0.44

-1.48

0

-0.55 -0.32-0.04

-0.34

0.340

-0.3

sample A sample B Rejection limitWarning limit Warning limit Rejection limit

z-sc

ore

ILC through Bilateral Agreement

• General framework: set within Protocol• Common sampling campaigns • Frequency and parameters: based on the transboundary

programmes requirements• Analytical methods: routine• Analysis of compliance of minimum technical requirements

from QA/QC Directive (2009/90/EC)• Bilateral meetings for discussions on results and agreement• Analysis of results from other PT schemes of interest • Investigation of unsatisfactory or questionable PT results• Monitoring of PT performance over time

ILC through common or parallel actions

• Sampling during different types of activities: research projects, investigative monitoring (i.e. Joint Danube Surveys)

• Objective: harmonisation of sampling techniques, analysis methods and assessment method

• Technical Reports available: supporting documents for further analysis

Conclusions

Laboratory intercalibration is not an easy task, but...• PT schemes are

• Required by the accreditation process • Available and fit for purpose• Widely used• Statement of laboratory’s performance

• Transboundary PT schemes – operational and good enough

• Bilateral Agreement in place – on regular basis and targeted to specific needs

Contacts

Thank you for your attention!

www.cbcromaniabulgaria.eu

Investing in your future!Romania-Bulgaria Cross Border Cooperation Programme 2007-2013 is co-financed by the European

Union through the European Regional Development Fund