18/08/2011

1

0.860.82

0.73

0.62

0.28

0.03 0.01

0.17

0.08

0.49

0.8

0.68

0.07

0.14

0.020.01

0.05 0.090

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Dec‐06 Jun‐07 Dec‐07 Jun‐08 Dec‐08 Jun‐09 Dec‐09 Jun‐10 Dec‐10

KPI

All analyte KPI Indicator analyte KPIBest

Worst

Participant 1Chemical Pathology QAP

Participant 2Chemical Pathology QAP

6

9 9

6

8

3

5

4

5

2 2

1

3

2

4 3

2 1 1 2 1 4 3 2 2

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Concordant Minor Discordance Discordant

2008_1  2008_2 2008_3  2009_1 2009_2 2009_3 2010_1 2010_2 2010_3Average Median Score

Median Score 70% 92% 93% 68% 86% 55% 62% 62% 67% 73%

% Rank 3% 45% 50% 11% 57% 1% 1% 2% 3% 3%

Participant 3Anatomical Pathology QAP

18/08/2011

2

Participant 4Cytopathology QAP

2004 2005 2006 2007 2008 2009 2010

Major error 0 3 0 3 0 0 1

Unacceptable response 0 0 0 0 1 0 3

No slides 2444 2772 2821 3266 632

PM 1 2.50 2.78 4.08 2.27 2.37

PM 2b 1.6 1.4 2.8 1.1 0.4

PM 3a 25.0 60.0 70.0 69.2 100.0

PM 3b 33.3 46.2 36.4 22.2 100.0

PM 4 0.0 8.0 2.4 12.0 7.7

Participant 5Cytopathology QAP

2004 2005 2006 2007 2008 2009 2010

Major error 1 0 0 0 1 0 0

Unacceptable response 1 0 0 0 1 1 2

LBC major 0 0 0 0 0 1 1

LBC unacceptable response

0 0 1 1 0 0 1

No slides 54529 53891 50464 51385 50579 50909

PM 1 1.08 0.97 0.78 0.94 1.25 1.1

PM 2b 0.4 0.6 0.6 0.5 0.9 0.6

PM 3a 79.4 75.7 77.6 81.7 64.3 76.8

PM 3b 59.4 60.0 53.8 63.6 57.0 63.2

PM 4 1.0 1.7 3.2 0.5 2.3 1.1

Participant 6Haematology QAP - Haemoglobinopathy

Participant 7Haematology QAP - Haemoglobinopathy

18/08/2011

3

Participant 8Haematology QAP - Morphology

2001

• In 2001 the Commonwealth Department of Health and Ageing sought proposals to undertake an evaluation of Australian pathology laboratory accreditation arrangements.

• This was the first comprehensive evaluation since the introduction of accreditation in 1986. This research resulted in the 2002 Corrs Chambers Westgarth report ‘Evaluation of the Australian Pathology Laboratory Accreditation Arrangements’.

18/08/2011

4

Evaluation of the Australian Pathology Laboratory Accreditation Arrangements

for the 

Commonwealth Department of Health and Aging

REPORT

17 July 2002

Chapter 5. External Quality Assurance Programs

• Recommendation 5.1That the DHA and the HIC seek the cooperation of the RCPA QAP to establish explicit external quality assurance performance criteria, initially in chemical pathology and gynaecological cytology, and a mechanism for the RCPA QAP to identify relatively poorly performing laboratories.

• Recommendation 5.2That RCPA QAP and other external quality assurance providers regularly submit to NATA reports identifying laboratories that are poorly performing according to these agreed performance criteria.

KPI Project• From this recommendation, in 2004/2005 the KPI project was

established by the RCPA and RCPA QAP.

• Key Performance Indicators were setup for Chemical Pathology and Cytopathology with the hope that it could be used as a mechanism to identify unacceptable laboratory performance.

• In 2006/2007 a collaboration was established between RCPA, RCPA QAP and NATA.

• Peer Review Committees were established by NATA to review a limited amount of KPI data.

KPI Project conclusions

• The KPIs and Peer Review Committee process can identify laboratories with poor EQA performance but this does not necessarily equate to poor ratings at on-site NATA assessment.

• The KPIs for Chemical Pathology have continued to be sent to laboratories every 6 months.

• A guide for laboratories on which results should be reviewed was established and distributed to laboratories.

• Cytopathology - KPIs are not a valid tool to use for identifying poorly performing laboratories and therefore it was recommended that they not be implemented in the existing format.

18/08/2011

5

Department of Health and Aging

0.13

0.55

0.13

0.01

0.09

0.01

0.24

0.75

0.03

0.18

0.96

0.080

0.050.01

0.53

0.67

0.020

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Dec‐06 Jun‐07 Dec‐07 Jun‐08 Dec‐08 Jun‐09 Dec‐09 Jun‐10 Dec‐10

KPI

All analyte KPI Indicator analyte KPI

Chemical Pathology QAP

Worst

Best

EQA is one aspect of accreditation

Accreditation

EQA

18/08/2011

6

‘The role of External Quality Assurance in identifying laboratory performance’

• Review four years of Chemical Pathology KPI data retrospectively

• Develop and evaluate KPIs for :– Anatomical Pathology– Transfusion Medicine– Cytopathology

• Develop triggers of concern using the external quality assurance data

• Trial the protocols to determine whether early indicators of unacceptable performance can be developed around the regular EQA that laboratories perform as part of the accreditation process.

• Establish mechanisms to use EQA data to help monitor quality of test kits

Review process

• Established a methodology / scoring system

• Retrospective data analysis

• Set criteria– Test performance– Participation

• Reviewed criteria against individual reports

• Assessed fit to framework (workability)

1

12

8

10

54

8 8 87

19

11

14

9 8

1213 14

22

25

2223

1917

18 19

35

29

36

26 26

21

2322

27

46

0

5

10

15

20

25

30

35

40

45

50

Dec‐06 Jun‐07 Dec‐07 Jun‐08 Dec‐08 Jun‐09 Dec‐09 Jun‐10 Dec‐10

Percentage returned Chemical Pathology KPI

Freq <50% returned Freq <60% returned Freq <70% returned Freq <80% returned

3

15

1

106 8 6 8 88

19

2

1813 12 12

1715

24

42

9

32

24 2534

28 28

39

64

24

50

3843

4743

54

137

148

119

128 125

159

172 170

162

0

20

40

60

80

100

120

140

160

180

200

Dec‐06 Jun‐07 Dec‐07 Jun‐08 Dec‐08 Jun‐09 Dec‐09 Jun‐10 Dec‐10

Percentage late Chemical Pathology KPIFreq >50% late Freq >40% late Freq >30% late Freq >20% late Freq >10% late

18/08/2011

7

12

01 1 1 1 1

02

5 54

35

87 7

1311

15 15

7

21

17

2122

36

25

44

28

21

41

37

41

49

0

10

20

30

40

50

60

Dec‐06 Jun‐07 Dec‐07 Jun‐08 Dec‐08 Jun‐09 Dec‐09 Jun‐10 Dec‐10

Percentage amended Chemical Pathology KPIFreq >20% Freq >15% Freq >10% Freq >5% Anatomical Pathology

0.00

0.50

1.00

1.50

2.00

2.50

3.00

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

100% 94% 81% 72% 67% 50% 45% 30% 19% 9%

Average Discordan

t Sum

Average Med

ian Score

Overall Participant Rank

General 2007 – 2010 excluding nonparticipation (433 participants)Average score over all surveys Average sum over all surveys

Participation 2007 –2010

2007Survey 1

2007Survey 2

2007Survey 3

2007Survey 4

2007Survey 5

2008Survey 1

2008Survey 2

2008Survey 3

2009Survey 1

2009Survey 2

2009Survey 3

2010Survey 1

2010Survey 2

2010Survey 3

Completed surveys 291 302 293 301 282 298 295 265 301 303 286 276 279 270Survey Nonparticipation 21 12 18 13 32 23 26 47 24 27 44 33 47 56Case Nonparticipation 1 1 3 3 2 2 3 11 3 4 5 8 6 5

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

x1 x2 x3 x4 x5 x6 x7 x8 x9Case 32 3 3 0 0 0 0 0 0Survey 104 52 21 6 2 1 1 0 1

0

20

40

60

80

100

120

Nonparticipation ‐ number of times

Non-participation

18/08/2011

8

Gynaecological Cytopathology

2007 2008 2009 2010

x1 Major Error 11 7 7 9

x3 Unacceptableresponse in 12 months 3 1 0 1

x1 PerformanceMeasure outside national standard

11 9 11 ‐

x1 Non return of QAP results 0 0 1 0

x1 Non return of PerformanceMeasure 0 0 0 0

• Number of participants outside criteria

Draft Framework

4. Criteria: Participants falling outside the criteria set for acceptable performance in the next survey or no response has been received within four weeks from the second letter sent, will be referred to the RCPA QAP 

Performance Review Committee. 

Action: A copy of the follow‐up letter or letter, EQA results and participants 

response is sent to NATA.

Action: The Chairperson or nominated Pathologist will contact the Medical 

Director directly. 

3. Criteria: Persisting unacceptable performance is referred to the Program Performance Review Committee.

Action: A copy of the letter, EQA results and participant response is 

sent to NATA.

Action: A letter is sent to the participant and nominated 

supervisor(s) from the Performance Review Chairperson. 

2. Criteria: Identification of results falling outside the criteria set for acceptable performance. 

Action: Initial letter sent to participant by the Program 

Manager.

1. Enrol in QAP.

Participants Results, letter a

nd re

spon

se re

ferred

 to 

NAT

A

Participants Results m

onito

redfor o

ne year

Results re

view

ed and

 actione

d by

 participant w

ithin 4 weeks

5. Findings and discussions with the manufacturer, their sponsor or TGA referred to RCPA QAP Review Committee.

4. Manufactoring company or their sponsor informed of findings by the Performance Review Committee Chairperson

3. Results outside the criteria set for acceptable performance of an Invitro Medical Device reviewed by the Program Performance Review Committee (IVD representative invited)

2. Results outside the criteria set for acceptable performance of an Invitro Medical Device identified by the QAP Program

1. Enrolment in QAP

Referred

 to TGA

Where to from here

• Participant consultation

• Letter to participants

• Letter from Programs re: criteria

• Pilot 2012

18/08/2011

9

Survey – have your say! Acknowledgments

• Department of Health and Aging - QUPP

• RCPA QAP Program Managers

• Penny Petinos