Driving laboratory quality through proficiency testing · 2014-12-02 · The primary aim of...

1

Driving laboratory quality through proficiency testing

Science for a safer world

Measurements

Measurement

Health

Security

Trade

Compliance Decisions

Reliable measurements

Reliable decisions

Overview

PT within the quality assurance system

Purpose, types and benefits of PT

Selecting appropriate PT schemes

Interpretation of PT results

Method Validation

Calibration Training

IQC

Quality Assurance System

? Measurement

System Input Output

RMs

PT

Fit-for-purpose analysis requires:

Careful development and validation of methods

Accurate calibration of instruments and appropriate equipment

Qualified and competent staff

Comparability with measurements made in other laboratories

Well defined QC and QA procedures

Independent evidence of performance

CRMs

Traceable Standards

RMs & PT

RMs

RMs & QC

PT

6

Proficiency Testing – purpose and types

What is PT?

The primary aim of proficiency testing is:

“To provide the infrastructure for a laboratory to monitor and improve the quality of its routine analytical measurements”

•  PT provides an objective evidence of the competence of the participant which can be used to

–  Improve the performance of the participant

–  Give confidence in the participant’s ability to perform a specific measurement

•  PT provides information on other aspects of the management system

–  Reception/treatment of the sample, treatment of the data, result reporting etc

Types of PT

•  Various types of PT available – based on one or more elements of four different categories:

Qualitative

Quantitative

Interpretive

1

Descriptive – nominal or ordinal scale

Numeric – interval or ratio scale

No measurement involved

Types of PT

•  Various types of PT available – based on one or more elements of four different categories:

Qualitative

Quantitative

Interpretive

Pre-measurement

Measurement

Post-measurement

1

2

Types of PT

•  Various types of PT available – based on one or more elements of four different categories:

Qualitative

Quantitative

Interpretive

Sequential

Simultaneous

Pre-measurement

Measurement

Post-measurement

1

3

2

Monitor Performance Report

Sequential participation scheme structure

PT Provider

Test Item

Test Result

Test Result

Test Result

Monitor Performance

Monitor Performance

Lab 1

Lab 2

Lab 3

PT P r o v i d e r

Evaluation of

Results

Lab 1

Lab 2

Lab 3

NMi

Simultaneous scheme structure

PT Provider

Test Items

Participating Laboratories

Test Results

PT Provider

Evaluation of

Results Report

Participating Laboratories

Monitoring Performance

Types of PT

•  Various types of PT available – based on one or more elements of four different categories:

Qualitative

Quantitative

Interpretive

Single

Continuous

Sequential

Simultaneous

Pre-measurement

Measurement

Post-measurement

1

3

2

4

Use of PT by laboratories

•  Basic use of PT for a laboratory is:

–  to assess its performance for the conduct of specific measurements or calibrations

•  Results and information received from participation will provide laboratories with:

–  Confirmation that the laboratory’s performance is satisfactory

–  An indication that there are potential problems and that corrections should be made

•  However, the use of PT should be much wider

–  Many additional benefits from PT participation

Benefits of PT to laboratories

Benefits of PT

Performance Comparison

Independent Check

Performance comparison

Benefits of PT to laboratories

Benefits of PT

Method/ Procedure Evaluation

Performance Comparison

Independent Check

Method/instrument comparison

Benefits of PT to laboratories

Benefits of PT

Method/ Procedure Evaluation

Performance Comparison

Monitoring Trends

Independent Check

Monitoring trends

Benefits of PT to laboratories

Benefits of PT

Third Party Demonstration

Method/ Procedure Evaluation

Performance Comparison

Monitoring Trends

Independent Check

Training and Education

Maximising the benefits of PT

•  To obtain the optimum benefit from PT participation, laboratories should: –  treat PT scheme test items as routine samples (i.e. do not

give them any special treatment) –  vary the analyst testing the test items –  vary the instrumentation used for the test items

•  To learn from PT participation, laboratories should: –  use the scheme results to give feedback to laboratory

procedures and staff; both positive and negative –  use the scheme results constructively –  be honest; the scheme is there to help –  Education not regulation

23

Selecting the right PT

Selecting the right PT scheme

•  Selection of a PT scheme is critical if the benefit is to be maximised

•  All laboratories need to develop an adequate PT participation strategy –  Aim to participate in relevant PT schemes, at an appropriate frequency

for the laboratory’s circumstances

•  Five key questions 1)  What level of PT and frequency do I need? 2)  Do any PT schemes exist for the technical competence required? 3)  Is the PT scheme relevant? 4)  Is the PT provider competent i.e. do they operate to ISO/IEC 17043? 5)  Is the selected PT scheme independent of any manufacturing or

marketing interests in equipment, reagents or calibrators?

Strategy of PT participation

•  Important to consider a number of different aspects

•  Other QA measures implemented: -  Use of CRMS -  Comparisons by independent techniques -  Participation in validation or characterisation studies -  Use of internal QC measures -  Other laboratory comparisons

•  Level of risk: -  No measurements undertaken -  Turnover of technical staff -  Staff experience/knowledge -  Known stability/instability of measurement technique -  Significance and final use of data

• Different types of PTs can be used

•  Difficulty of PT participation e.g. technical characteristics of the measurement, low number of laboratories etc

•  Legislative requirements for frequency of type of participation

Fit for purpose PT schemes

•  Need to decide which scheme is fit for purpose –  Often a scheme will not have a perfect fit, so need to find the

best fitness for purpose

•  A number of factors to consider

–  Test Items

–  Distribution

–  Participants

–  Results

–  Reports

–  PT Provider

Test items

‘Real’ sample Simulated sample

Real sample:

•  Familiar format

•  Similar challenges to routine samples

Simulated sample:

•  Homogenous

•  Stable

Test items

•  Parameters: –  Routinely tested parameters –  Standard reporting units –  Available at an appropriate frequency

•  Concentrations: –  Appropriate for the range in which the

laboratory is working –  Significantly different levels will not

reflect on routine testing at the laboratory’s normal levels

Distribution

•  Distribution dates available

•  Distributions monthly, bi-monthly, quarterly, 6-monthly or annual

•  Flexible participation available

•  Fit-for-purpose

Participants

•  National/international –  Languages –  regional offices –  Transport

•  Participating laboratories –  Private/public –  Test methods used

Results

•  Deadlines

•  Methods of returning results

•  Confidentiality

•  Statistical approach –  Available to participants –  Easy to understand –  Consistent –  Summary statistics –  Graphs –  Trend analysis

Reports

•  Speed of reporting

•  Language of documentation

•  Information in reports –  scores –  graphs –  tables –  technical feedback –  method information

•  Format of report e.g. electronic version or paper copy

PT Provider

•  Experience

•  Scope

•  Competence –  Complies with ISO/IEC

17043

34

Interpretation of PT results

Performance evaluation

•  Limited value unless performance is evaluated

•  Evaluate after each round and also over time

•  Interpretation of PT performance concerns all management levels

•  PT is not about “passing” or “failing” – it is about learning from the results

Review of single PT round performance

•  Results of each PT round should be evaluated regardless of performance

–  A satisfactory result may not mean good performance –  Unsatisfactory performance in one round does not indicate

that the laboratory is not competent

•  Important to consider a number of factors: –  The overall results in the round

–  Test method performance

–  Test item factors

–  Bi-modal result distribution

–  PT Scheme factors

Trends in PT performance

•  Trends are valuable in preventive and corrective actions, and root cause analysis

–  Performance over time

–  Common groups of determinands

–  Performance of methods

–  Performance of analysts

Need for an investigation

•  All laboratories will occasionally have unsatisfactory or questionable PT results

•  A laboratory should establish its own criteria for when an investigation is necessary

•  The depth of investigation will depend on a number of factors:

–  Criticality of the method

–  Frequency of unsatisfactory results

–  Evidence of a bias

Causes of unsatisfactory performance

•  Can be analytical and non-analytical

•  Both are equally serious –  the result reported is the important factor for a laboratory

•  Analytical Errors –  Calibration –  Instrument problems –  Extraction/clean-up –  Interferences –  Method performance –  Analyst

•  Non-Analytical Errors –  Calculation –  Transcription –  Units –  Reporting format

PT – in conclusion

 PT is an essential tool to all laboratories making measurements

 PT participation brings many benefits

 PT enables measurements to be monitored and improved

 PT is a very powerful quality assurance tool

underpinning laboratory accreditation

PT drives laboratory quality

PT Reliable Measurements

Reliable Decisions