Exploring the Medical Laboratory Quality ToolBox-A Michael A Noble MD FRCPC Clinical Microbiology...

Exploring the Medical Laboratory Quality ToolBox-A

Michael A Noble MD FRCPCClinical Microbiology Proficiency Testing program

Program Office for Laboratory Quality ManagementUniversity of British Columbia

Vancouver BC

Management

by theMoment

Management

by theMomentYou can try

to fix them…

Management

by theMomentBut th

ey never go away!

So the better approach is…

An systemic approach of organization, plan, review, and

action gives you the best chance of success.

The Quality Toolbox

The Quality Toolbox

Lean

RISKFMEA

Six Sigma

Metrics

Surveys

Statistics

The Quality Toolbox

January 2009 7

What are quality tools?

ISO 9001: 2000

Six Sigma

LEAN

A quality tool is a supplement or component part of a quality program that usually will not stand alone

but can enhance the total quality system

Tools in the Toolbox

• Mega-Tools– Lean– Six Sigma – Priority Matrices and

Risk Assessment

• Tools/Techniques– Balanced Scorecards– Brainstorming– Control Charting– Flow Charting– Quality Indicators– Surveys

Quality Indicators are Metrics(measured process information)

• Determine quality of services.• Highlight potential quality concerns, • Identify areas that need further study and

investigation, and • Track changes over time. 

A really good, inexpensive

reference book

Amazon.com$30.00

Keeping ScoreUsing the Right Metrics to Drive World Class Performance

1996

Many organizations spend thousands of hours collecting and interpreting data. However many of these hours are nothing more than wasted time because they analyze the wrong measurements, leading to inaccurate decision making.

– Mark Graham Brown.

Characteristics of Good Metrics

Timedshort and long term

Timedshort and long term

Engagingall levels

Engagingall levels

Balancedfull cycle

Balancedfull cycle

Actionableaction oriented

Actionableaction oriented

Interpretablespecific

Interpretablespecific

Achievablecontained

Achievablecontained

Measurableobjective

Measurableobjective

GoodMetricsGood

Metrics

Seven Steps to Successful Indicators

Do not start data collection until these are addressed

1. Objective

2. Methodology

3. Limits

4. Interpretation

5. Limitations

6. Presentation

7. Action plan

Developing IndicatorsObjective What are you trying to measure?

1. Why am I collecting this information? Be specific

Methodology How to capture the data1. What data needs to be captured2. Who (or what) to capture the data3. How often to capture the data4. Is it achievable (time, resources, revenue)?

Limits Can I preset levels for:1. Acceptable, Concern, Unacceptable, Critical

Presentation Graphic or Text

Interpretation 1. What does it mean?2. Does it reflect on YOUR quality?3. Can I compare it?4. Can I trend it?

Limitations 1. Unintended variables

2. What does it not mean?

Action Plan 1. What will I do if it indicates acceptable performance?

2. What will I do if it does not?

Developing IndicatorsObjective What are you trying to measure.

1. Why am I collecting this information? Be specific

Methodology How to capture the data1. What data needs to be captured2. Who (or what) to capture the data3. How often to capture the data4. Is it achievable (time, resources, revenue)?

Limits Can I preset levels for:1. Acceptable, Concern, Unacceptable, Critical

Presentation Graphic or Text

Interpretation 1. What does it mean?2. Does it reflect on YOUR quality?3. Can I compare it?4. Can I trend it?

Limitations 1. Unintended variables

2. What does it not mean?

Action Plan 1. What will I do if it indicates acceptable performance?

2. What will I do if it does not?

Setting Relevant Limits and Ranges

• Set Objectively• Validate by Study• Clinical Relevancy• Customer Expectation• Matched Benchmarks• Regulation

60 minutesRelevant or Easy?

Assessing Quality Indicators• Importance Potential for Improvement

• Scientific Acceptability Reliability and Validity

• Feasibility Implementation and cost

• Usefulness Comprehensive

Having Quality Quality Indicators

IQLM Indicator List

1. Diabetes monitoring (system) 2. Hyperlipidemia screening (system)

• Test Order Accuracy and Appropriateness (pre-analytic)

• Patient Identification (pre-analytic)

• Adequacy and Accuracy of Specimen Information (pre-analytic)

• Blood Culture Contamination (pre-analytic)

• Accuracy of point-of-care testing (analytic) • Cervical cytology/biopsy correlation (analytic) • Critical Values Reporting (post-analytic) • Turnaround time (post-analytic) • Clinician satisfaction (post-analytic) • Clinician follow-up (post-analytic)

CLMA Survey Pre-examination Phase Indicator List

•Ordered test is appropriate for patient care

•Patient consent appropriately collected

•Test utilization by clinician for best patient care

•Physician written order with every specimen

•Cost/benefit assessment for laboratory test menu

•Patient identification and its accuracy

• Preparation of patient for specimen collection

•Appropriate specimen container•Timing of specimen collection•Phlebotomy success•Specimen integrity •Specimen quantity•Specimen transportation•Accuracy of specimen identification

•Condition for specimen storage

CLMA Survey Examination Phase Indicator List

• Quality Control• EQA-external quality

assessment • Time to first result

availability • Specimen

contamination

• Laboratory injuries or accidents

• Competency of testing personnel

• Vacancy of technical staff

CLMA Survey Post-examination Phase Indicator List

• Result reporting accuracy• Adequacy of information

for interpretation of laboratory tests

• Report delivery turnaround time

• Consistency of critical values reporting

• Result interpretation by physician

• Patient’s satisfaction with laboratory services

• Patient’s satisfaction specifically with phlebotomy services

• Physician’s satisfaction with laboratory services

Contmination Rate: Blood Culture Sets

0.0%

1.0%

2.0%

3.0%

4.0%

1 2 3 4 5 6 7 8 9 10 11 12 13 14

Time Period

Per

cent

2002-2003 2003-2004 2004-2005

Contmination Rate: Blood Culture Sets

0.0%

1.0%

2.0%

3.0%

4.0%

1 2 3 4 5 6 7 8 9 10 11 12 13 14

Time Period

Perc

ent

2002-2003 2003-2004 2004-2005

Limits: Below 2%

Interpretation: Meeting accepted limits all the time

Limitations Definition may include some true infections and may miss others

Presentation: Linear time graph

Action plan: Identify and educate blood collector group.If stable for 2 years, then consider dropping from routine

Objective: to ensure that blood culture results reflect sepsis.

Methodology: Count single bottle positives of common skin flora/ total sets

Quality Indicators and Timing

Use an indicator only as long as

it providesyou with

useful information. Don’t get tied to

your indicators

Underfill Blood Collections (As a percent of collections per site)

0.0

2.0

4.0

6.0

8.0

10.0

12.0

14.0

16.0

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39

RF2

RF1

ER

ICU

Phlebotomists

Limits: Below 2% (?)

Interpretation: Wards with inexperienced collectors have problems

Limitations Some frail and elder people have very weak veins and may be impossible to collect

Presentation: Linear time graph

Action plan: Identify and educate blood collector group.

Objective: to ensure that blood culture are properly

filled.

Methodology: Count underfilled bottles / total bottles collected

Underfill Blood Collections (As a percent of collections per site)

0.0

2.0

4.0

6.0

8.0

10.0

12.0

14.0

16.0

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39

RF2

RF1

ER

ICU

Phlebotomists

Caution about patient outcome indicators

Theoretically, outcomes best assess quality, but they are the most difficult to measure – too many confusing variables

• Age, underlying conditions, therapy, circumstance

– require high volumes of detailed data – need long collection periods

David HsiaMedicare Quality Improvement Bad Apples or Bad Systems? JAMA. 2003;289:354-356.

Quality Indicators, Done Well, Will Consume More Time Than

You Have

FACT:

•Set Priority•Set Limits•Drop Non-Productive Activity•Target: 10-12

Don’t be an

Indicators Glutton

Computer Nonsense Metrics[urine culture] * [glucose] * [INR]

[NUPA hr] * [Telephone minutes] X100

Just because a computer can

calculate a value, doesn’t mean that

it should.

The BIG SECRET for Quality Indicator Team

Engage the folks who do the work, because they know what they do!

Risk management is activity directed towards assessing,

mitigating (to an acceptable level) and monitoring of risks to an enterprise.

Risk Managementhelps definePRIORITIES

and helpsPREVENT ERROR

Tough Decisions 1

• A group of 10 laboratories over 300 square miles considers centralizing all tests to a single facility.

Improved cost efficienciesSimplified process controlCloser oversightImproved utilization managementImproved Patient Care

Reduced clinical-lab interfaceExtended pre-examination phase

Test delaysImpeded Patient care

Tough Decision 2

• A laboratory considers laboratory redesign based on LEAN analysis.

Improved workflowImproved time efficienciesSolve ergonomic challengesImproved cost efficienciesImproved Patient Care

Expensive renovationsConstruction interruptions

Staff retrainingEquipment specificityTest menu specificity Impeded Patient care

Tough Decision 3

• Laboratory considers implementing a quality management team.

Better information for better decisionsBetter process controlContinual Improvement ProcessImproved Patient Care

Staff ReallocationIncreased immediate costs

Quality Commitment Impeded Patient care

Tough Decision 4

• A laboratory proposes the need for an on-site level 3 (increased biosafety level) microbiology laboratory.

Improved biosafetyReduced laboratory infectionsFaster diagnosisImproved Patient Care

Construction costsOperating costs

Potential Bioterror targetImpeded Patient care

Risk and Priority and Prevention are Relative Terms

• Risk tolerance• Risk transfer

• Risk reduction• Risk prevention• Risk avoidance

• Risk aversion

CO

MF

OR

TACTION

PR

IOR

ITY

Risk Management Documents for the Medical Laboratory

• ISO 14971:2007Medical devices -- Application of risk management to medical devices

• ISO 20993:2006 Biological evaluation of medical devices -- Guidance on a risk-management process

• ISO WD TS22367:2007Medical laboratories -- Reduction of error through risk management and continual improvement

Linking Quality and Risk Managements

Introduction to ISO TS 22367

“Risk management framework has been described as being of

steps, planning for risk, identifying risk and its impacts,

developing risk-handling strategies, and monitoring for risk

control. These steps are consistent with management

requirements as described in ISO 15189:2003 including

identification and control of non-conformities, establishment of

preventive and corrective actions, performance of internal

audit and management review and continual improvement.

It is the objective of this technical report to link these in the

context of the medical laboratory.”

Linking Prevention and RiskISO TS 22367

• Assessment of risk of deviations from standard• The quality manager should establish and maintain a process

for identifying incidents associated with deviations from standards requirements, estimating and evaluating the associated risks to patient care, and controlling these risks and monitoring effectiveness of the control.

• The process should include a prospective risk assessment method for processes considered as high risk.

• Assessment of potentially high risk processes may be based upon previous audit, survey, experience, or evidence-based literature on procedures where a failure may lead to significant safety risk to patients.

Linking Prevention and RiskISO TS 22367 (Continued)

– The quality manager should identify a team of people to study the selected process.Note 1: The team should have personal knowledge of the process.Note 2: The team should be comprised of people with different levels and types of knowledge.

– The team should organize a thorough investigation of the process to include:a) each step of the processb) how each step of the process may failc) how each failure at each step of the process may affect patient safetyd) the rank of severity of each failure mode effecte) the most critical failure mode effects f) potential root causes of the most critical failure mode effectsg) procedures to address the potential root causes.

– The analysis of this Failure Mode Effects Analysis should form the basis of a Prevention Action Plan.

Occurrence – Severity Grid(Qualitative)

Remote Rare Common Frequent

Insignificant

Mild

Moderate

Severe UNACCEPTABLERISK

ACCEPTABLERISK

ACCEPTABLERISK

ACCEPTABLERISK

DEBATABLERISK

UNACCEPTABLERISK

UNACCEPTABLERISK

DEBATABLERISK

UNACCEPTABLERISK

UNACCEPTABLERISK

ACCEPTABLERISK

ACCEPTABLERISK

DEBATABLERISK

ACCEPTABLERISK

DEBATABLERISK

DEBATABLERISK

Occurrence – Severity Grid(Semi-Qualitative)

Remote (0.2)

Rare (0.4)

Common(0.8)

Frequent (0.9)

Insignificant(0.1)

Mild(0.2)

Moderate (0.8)

Severe(0.9)

UNACCEPTABLERISK

ACCEPTABLERISK

ACCEPTABLERISK

ACCEPTABLERISK

DEBATABLERISK

UNACCEPTABLERISK

UNACCEPTABLERISK

DEBATABLERISK

UNACCEPTABLERISK

UNACCEPTABLERISK

ACCEPTABLERISK

ACCEPTABLERISK

DEBATABLERISK

ACCEPTABLERISK

DEBATABLERISK

DEBATABLERISK

Severity – Occurrence Grid

• Numerical Values– Literature Benchmarks– Consensus– Experience

• Regardless of source, values should be both verified, and validated to fit the FMEA being performed.

HIV Testing Risk EvaluationCanadian versus American Risk Analysis

Remote (0.2)

Rare (0.4)

Common(0.8)

Frequent (0.9)

Insignificant(0.1)

Mild(0.2)

Moderate (0.8)

Severe(0.9)

UNACCEPTABLERISK

ACCEPTABLERISK

ACCEPTABLERISK

ACCEPTABLERISK

? UNACCEPTABLERISK

UNACCEPTABLERISK

DEBATABLERISK

UNACCEPTABLERISK

UNACCEPTABLERISK

ACCEPTABLERISK

ACCEPTABLERISK

DEBATABLERISK

ACCEPTABLERISK

DEBATABLERISK

DEBATABLERISK

Quality Approaches Look Alike

Strategy Event Steps

Corrective Action Plan

PLAN MONITOR EXAMINEREMEDIATE

AND CORRECT

CONTROL

Preventive Action Plan

PLAN IDENTIFY EXAMINEDEVELOP STRATEGY

MONITOR

Risk Management

PLAN IDENTIFY EXAMINEDEVELOPSTRATEGY

MONITOR

Six Sigma DEFINE MEASURE ANALYZE IMPROVE CONTROL

Failure Mode Effects Analysis

• FMEA is a systematic method of studying failure.

• 1940s US Armed Forces1960s Aerospace (Apollo Space program)1970s Ford Motor Company (post-Pinto)1990s semiconductors, food service, plastics,

software, and diagnostics equipment.• Documented FMEA is a REQUIREMENT in the

automotive, in-vitro diagnostics, and pharmaceutical industries.

October 1, 2008

FMEA: Failure Mode Effects Analysis

In a process:– where are points that the process is likely to

break down?– what is the probable occurrence of the

different possible break-downs?– what are the likely severity and consequences

of the different possible break-downs?– What is the priority list for addressing the

different possible break-downs?

FMEA: Failure Mode Effects Analysis

• FMEA– Failure Mode Effects Analysis

• FMECA– Failure Modes Effects and Criticality Analysis

• FTA– Fault Tree Analysis

• HACCP– Hazards Analysis and Critical Control Points

• PDPC– Process Design Program Chart

A Medical Laboratory FMEA

PhysicianRequisition

SampleCollection

Sample Set-up

SampleAnalysis

SampleTransport

A Medical Laboratory FMEA

PhysicianRequisition

SampleCollection

Sample Transport

Sample Set-up

SampleAnalysis

Good Sample

Proper ContainerSite Identification

Proper Packaging

Courier competent and available

Equipment and stainsworking properly

Workload

Time issues

Expertise

Labelling

A Medical Laboratory FMEA

PhysicianRequisition

SampleCollection1.5

Sample Transport0.5

Sample Set-up1.9

SampleAnalysis1.3

Good Sample

Proper ContainerSite Identification

Proper Packaging

Courier competent and available

Equipment and stainsworking properly

Workload

Time issues

Expertise

0.5

0.5 0.2

0.8

0.3

0.2

0.6

0.5

0.5

0.30.4

Labelling0.9

SEVERITY

A Medical Laboratory FMEA

PhysicianRequisition

SampleCollection0.45

Sample Transport0.8

Sample Set-up0.25

SampleAnalysis1.05

Good Sample

Proper ContainerSite Identification

Proper Packaging

Courier competent and available

Equipment and stainsworking properly

Workload

Time issues

Expertise

0.5

0.05 0.3

0.1

0.4

0.4

0.1

0.5

0.5

0.050.05

Labelling0.1

OCCURENCE

Pre-Examination Grid

October 1, 2008

Remote 0.2

Rare 0.5

Common 0.8

Frequent>1

Insignificant(0.2)

Mild(0.5)

Requisition Transport

Moderate (0.8)

Severe(>1)

Set-up Collection Analysis

A warning about FMEA

• If done as a structured “top-down” exercise, many critical break points will be missed.

• If done as an unstructured “bottom-up” exercise, the team will get bogged down in minutiae

FMEAs should ALWAYS be:✔Directed✔Team Efforts✔Planned✔Structured

The BIG SECRET for The FMEA Team

Engage the folks who do the work, because they know what they do!