QUALITYIMPROVEMENTCriticalappraisalofQIliterature

Dr.JenniferTwissMSc,MDFRCPC

Disclosure

•  Ideclarenopersonalorfinancialconflictofinterestinthispresentation

•  Therewillbenodiscussionofunlicenseddrugsortherapiesduringthispresentation

Objectives

•  ThevalueofQIresearchinthepracticeofevidencedbasedmedicine

•  AppreciatethecommondifficultiesofQIresearch

•  UseofcurrentframeworkstoassessthequalityofpublishedQIresearch

•  ReviewstudydesignsthatleadtohighqualityQIresearch

PollEV•  PleasesignintoPOLLEVwithyourcellphones

•  TextJENNIFERTWIS625to37607oncetojoin

WhatisQualityImprovement?•  QUALITYIMPROVEMENTINHEALTHCARE•  Together,makingcarebetter:supportingandpromotingsubstantialandsustainablepositiveimprovementsincare–  It’slookingforopportunitiesforimprovement,implementingevidencebasedmedicinepractices,tryingnewapproaches,andadoptingthemiftheywork.

•  Systematic,data-guidedactivitiesdesignedtobringaboutimmediateimprovementsinhealthdeliveryinparticularsettings

•  Improvingthequalityofcareofpatientsisafundamentalobligationofhealthcareproviders–  TheQIprocessinvolvesevaluatingandlearningfromexperience

ModelforImprovement

LangleyGL,MoenR,NolanKM,NolanTW,NormanCL,ProvostLP.TheImprovementGuide:APracticalApproachtoEnhancingOrganizationalPerformance(2ndedition).SanFrancisco:Jossey-BassPublishers;2009

HumanSubjectsResearch QualityImprovement

Purpose developorcontributetogeneralizableknowledge implementknowledge,assessaprocessorprogramasjudgedbyacceptedstandards

StartingPoint independentofroutinecareandintendedtoansweraquestionortestahypothesis

integraltoongoingmanagementsystemfordeliveringhealthcare

Design followsarigidprotocolthatremainsunchangedthroughouttheresearch

adaptive,iterativedesign

Benefitsmightormightnotbenefitcurrentsubjects;intendedtobenefitfuturepatients

directlybenefitsaprocess,systemorprogram;mightormightnotbenefitpatients

Risks mayputsubjectsatrisk doesnotincreaserisktopatients

ParticipantObligation

noobligationofindividualstoparticipate responsibilitytoparticipateascomponentofcare

Endpoint Answeraresearchquestion improveaprogram,processorsystem

Analysis Statisticallyproveordisprovehypothesis compareprogram,processorsystemtostandards

AdoptionofResults

littleurgencytodisseminateresultsquickly resultsrapidlyadoptedintolocalcaredelivery

Publication/Presentation

investigatorobligedtoshareresults QIpractitionersencouragedtosharesystematicreportingofinsights

Tableadaptedfrom:https://irb.research.chop.edu/quality-improvement-vs-research

QualityImprovementisDifferent

•  Focusonstudying– Howtoimplementbestpracticesindifferentcontexts

– Applicationoftherapiesknowntoworkindifferentpopulations

– Howproventherapiesfunctioninanuncontrolledenvironment

QualityImprovementisDifferent•  Iterativedesign

– Theimplementationstrategychangesovertimebasedontheoutcomeofsmalltestsofchange

– Multiplechangesoccurtolookateffectononeultimateoutcome

BillimoriaZetal.J.Perinat.Med.2013;41(4):455-460

QualityImprovementisDifferent•  Innovation

– Adaptationtoachangingcomplexenvironment– Changeandadaptationconductedusingdataandstructure

SkiboMetal.JournalofPerinatology(2017)37,1341-1345

BataldenPB,DavidoffF.QualSafHealthCare2007;16:2-3

QualityImprovementResearch

•  Studyofthenetbenefitofexistingtherapiesindifferentenvironmentsandpopulations– Changebehavior–  Implementationofacceptedclinicalpractices– Changesinthedeliveryofhealthcare– Practicalproblemsolving–  Innovationandadaptation

FeaturesofQualityImprovementResearch

•  Descriptionofthemotivationforchange•  Rationale

“Overrecentdecadestherehasbeenasignificantreductioninthecongenitalheartdiseasemortalityrate.Asaresult,increasingemphasishasbeenplacedonreducingmorbidityandidentifyingmarkersofcaredeliverysuchashospitalLOSandincidenceofhospitalreadmission.Thepaediatriccardiacpopulationisatparticularriskforin-hospitaldeteriorationandsubsequentreadmissiontothecardiacintensivecareunit(CICU).”

StoreyJetal.BMJQualSaf2018;27:66-72

FeaturesofQualityImprovementResearch

•  Descriptionofthemotivationforchange–  HospitalreadmissiontopediatricCICUassociatedwithmortality

•  Assumptions–  Preventionofreadmissions

•  Nodatalookingatpreventingreadmission–  Higherrateofreadmissions

•  Internaldatademonstratetherearereadmissions,nothighnobenchmarkdata

–  Standardizedtransferofcare•  Notexisting

–  Standardizedmonitoringbasedonrisk•  Notexisting

StoreyJetal.BMJQualSaf2018;27:66-72

FeaturesofQualityImprovementResearch

•  Descriptionofthemotivationforchange•  Hypothesis1

– Subtleclinicalchangesoccurandgounrecognizedduetolackofpatientspecificmonitoringleadingtomoreurgentandcomplexinterventions

•  Hypothesis2– ComplicancewithstandardizedcareprocessbasedonriskorconditionwouldreduceinhospitalreadmissionsbacktotheCICUwithin48hoursoftransfer

StoreyJetal.BMJQualSaf2018;27:66-72

FeaturesofQualityImprovementResearch

•  Descriptionofthecontext•  Physical

“HutzelWomen’sHospitalisanurban,academicwomen’shospitalwitha55-bedlevel3inbornNICUandspecialcarenurseryalongwithahigh-riskmaternal-fetalmedicineservice.Thereareabout5500deliverieseachyear.ThedeliveryareasareconnectedtotheNICUbyanenclosedbridge,whichtakes1–3mintotraverse.Deliveryofapre-matureneonate(<32weeks’gestationalage)isattendedbyaneonatalfellow,anursepractitioner,arespiratorytherapist,andtworesidents.Aneonatologyattendingphysicianattendsdeliveriesofinfantsat<32weeks’gestationduringthedayshifts,andaneonatalnurseattendssomedeliveriesanticipatedtoneedextensiveresuscitation.”

BillimoriaZetal.J.Perinat.Med.2013;41(4):455-460

FeaturesofQualityImprovementResearch

•  Descriptionofthecontext•  Cultural

“OurorganisationhasamaturequalityimprovementinfrastructurewithinourCenterforHealthSystemsExcellence.Hospitalfacultyandstaffaretrainedinimprovementscienceviamultiplecourseofferings.”

StoreyJetal.BMJQualSaf2018;27:66-72

FeaturesofQualityImprovementResearch

•  Descriptionofthecontext•  Resources

“Qualityimprovementconsultantsanddataanalystsprovideassistanceformanyimprovementprojects.”

StoreyJetal.BMJQualSaf2018;27:66-72

FeaturesofQualityImprovementResearch

•  Datacollectiontodrivetheimprovementefforts•  Systematic

– Onepretestandtwoposttestmeasurements(3and12months)

–  3ICUsreceivedtrainingandcomparedwithamatchedcontrolunit

–  Educationdeliveredtoallgroupsofstaff– Usinganevaluationframework– Mixedmethoddesignusingquestionnaires,observationsandpatientoutcomedata

KemperPFetal.BMJQualSaf2016;25:577-587

QIPitfalls

•  Generalizability– Contextdependent– Lessabletoproduceconsistentresults

QIResearchPitfalls

•  Complex– Multipleinterventions– Multipletimeperiods– Mayrequirefinancialbackingandpersonnel

DifficultiesinQIResearch

•  Highriskofbias– Factorssuchasenvironment,resourceinfluenceresults

– Difficulttoidentifyallfactorsthatmayhavecontributedtopublishedeffect

DifficultiesinQIResearch

•  Heterogeneousapproach– Qualitativeandquantitativedata– Mixedmethodstudies– Designmethodsnotalwaysrigorouslyfollowed

EVALUATINGQILITERATURE

TOOLS

DeterminingQualityofaQIStudy•  Interventionsshouldconsidercurrentstandardsofcareorevidencebasedpracticerecommendations

•  QIworkissubjecttobiasesandconfounders–  Applicabilitytoyourcontext–  Changesinpracticeovertime(drift/creep)–  Variationintrendsofpatientoutcomesovertime

•  Followupperiodhastobelongenough–  Sustainabilityplansareimportanttodescribe

•  Alloutcomesneedtobeidentified–  Unintendedconsequences–  Resourceutilization–  Cost

SQUIRE2.0

•  StandardsforQualityImprovementReportingExcellence

•  CreatedtoguidequalityofQIstudies•  ProvideaframeworkforreportingQIstudies

– Usuallyapplicabletosystemlevelchangetoimprovequalitysafetyandvalueofhealcare

–  PublishedstandarddirectingwrittenframeworkofQIwork

•  Maybeadaptedtosuitdifferentmethodologies•  AgeneralguidetowritingupQIwork

OgrincG,etal.BMJQualSaf2016;25:986-992.

SQUIRE2.0•  TITLEANDABSTRACT•  Title:Indicatethatthemanuscriptconcernsaninitiativetoimprovehealthcare

•  Abstract:– Provideadequateinformationtoaidinsearchingandindexing

– Summarizeallkeyinformationfromvarioussectionsofthetextusingastructuredsummary

•  Eg.Background,localproblem,methods,interventions,resultsandconclusions

OgrincG,etal.BMJQualSaf2016;25:986-992.

SQUIRE2.0•  INTRODUCTION:WHYDIDYOUSTART?•  Problemdescription:

–  Natureandsignificanceoftheproblem•  Availableknowledge:

–  Summaryofwhatiscurrentlyknownabouttheproblemincludingrelevantpreviousstudies

•  Rationale:–  Informalorformalframeworks,models,conceptsandortheoriesusedtoexplaintheproblem,anyreasonsorassumptionsthatwereusedtodeveloptheintervention,andreasonswhytheinterventionwasexpectedtowork

•  Specificaims:–  Purposeoftheprojectandofthisreport

OgrincG,etal.BMJQualSaf2016;25:986-992.

SQUIRE2.0•  METHODS:WHATDIDYOUDO?•  Context:

–  Contextualelementsconsideredimportantattheoutsetofintroducingtheinterventions

•  Interventions:–  Descriptionoftheinterventioninsufficientdetailthatotherscouldreproduceit

–  Specificsoftheteaminvolvedinthework•  StudyoftheInterventions:

–  Approachchosenforassessingtheimpactoftheintervention

–  Approachusedtoestablishwhethertheobservedoutcomeswereduetototheintervention

OgrincG,etal.BMJQualSaf2016;25:986-992.

SQUIRE2.0•  METHODS:WHATDIDYOUDO?•  Measures:

–  Measureschosenforstudyingprocessesandoutcomesoftheinterventionsincludingrationaleforchoosingthem,theiroperationaldefinitionsandtheirvalidityandreliability

–  Descriptionoftheapproachtotheongoingassessmentofcontextualelementsthatcontributedtothesuccess,failure,efficiencyandcost

–  Methodsemployedforassessingcompletenessandaccuracyofdata•  Analysis:

–  Qualitativeandquantitativemethodsusedtodrawinferencesfromthedata

–  Methodsforunderstandingvariationwithinthedata,includingtheeffectsoftimeasavariable

•  Ethicalconsiderations:–  Ethicalaspectsofimplementingandstudyingtheinterventionandhow

theywereaddressed,includingbutnotlimitedtoformalethicsreviewandpotentialconflictofinterest

OgrincG,etal.BMJQualSaf2016;25:986-992.

SQUIRE2.0

•  RESULTS:WHATDIDYOUFIND?•  Results:

–  Initialstepsoftheinterventionandtheirevolutionovertime,includingmodificationsmadetotheinterventionduringtheproject

–  Detailsoftheprocessmeasuresandoutcome–  Contextualelementsthatinteractedwiththeintervention

–  Observedassociationsbetweenoutcomes,interventionsandrelevantcontextualelements

–  Unintendedconsequencessuchasunexpectedbenefits,problems,failuresorcostsassociatedwiththeintervention

–  DetailsaboutmissingdataOgrincG,etal.BMJQualSaf2016;25:986-992.

SQUIRE2.0•  DISCUSSION:WHATDOESITMEAN?•  Summary:

–  Keyfindings,includingrelevancetotherationaleandspecificaims

–  Particularstrengthsabouttheproject•  Interpretation:

–  Natureoftheassociationbetweentheinterventionsandtheoutcomes

–  Comparisionoftheresultswithfindingsfromotherpublications

–  Impactoftheprojectonpeopleandsystems–  Reasonsforanydifferencesbetweenobservedandanticipatedoutcomes,includingtheinfluenceofcontext

–  Costsandstrategictradeoffs,includingopportunitycosts

OgrincG,etal.BMJQualSaf2016;25:986-992.

SQUIRE2.0•  DISCUSSION:WHATDOESITMEAN?•  Limitations:

–  Limitstothegeneralizabilityofthework–  Factorsthatmighthavelimitedinternalvaliditysuchasconfounding,

biasorimprecisioninthedesign,methods,measurement,oranalysis

–  Effortsmadetominimizeandadjustforlimitations•  Conclusions:

–  Usefulnessofthework–  Sustainability–  Potentialforspreadtoothercontexts–  Implicationsforpracticeandforfurtherstudyinthefield–  Suggestednextsteps

•  OTHERINFORMATION•  Funding:

–  Sourcesoffundingthatsupportedthework.Roleifanyofthefundingorganizationinthedesign,implementation,interpretationandreporting

OgrincG,etal.BMJQualSaf2016;25:986-992.

SQUIRE2.0

•  Allowsfor– Transparency– Descriptionoftheentireprocess– Accuracyofreporting– Reportingofthe‘doing’and‘studying’ofQIwork

•  Doesnotaddress– Howtoassessqualityofevidenceprovidedinthearticle

Users’GuideforanArticleAssessingQualityImprovement

•  Insightintohowtoassessthequalityofthearticle•  EmphasizesspecificproblematicfeaturesofQIarticles

–  STUDYDESIGN–  BIASES

•  Confounders,clustereffects,timetrendeffects•  Generalizabilityandspread•  Sustainability•  Benefits,hassles,costs

–  DATA•  Dataquality,definitions,rigorousmeasurement•  Size,precisionandeffectoftheresults•  Outcomesandbalancingmeasures

Fanetal.JAMA2010;304(20):2279-2287

Fanetal.JAMA2010;304(20):2279-2287

QIDesignsThatWork•  SteppedWedgedesign

–  Sequentialrolloutofinterventionstudyunits(phases)overdefinedperiodsoftime

–  Randomandsequentialcrossoverofclustersfromcontroltointervention

–  Allpatientsreceivetheinterventionovertime–  Canrandomizetheorderofinterventions

•  Step–  Dataandoutcomesmeasureatthepointwherethepatiententerstheintervention

•  Wedge–  Observeddifferencesinoutcomeincontrolvsintervention,thencansaythechangeisattributabletotheintervention

SteppedWedgeDesign•  Withindividualrecruitmentandwithoutconcealmentof

allocation(blinding)–  Atriskofselectionbias–  Effectoftheinterventionmightbeconfoundedwithtemporaltrends

•  Eachclustercontributesbothexposedandunexposeddataandthereforecanactasitsowncontrol

•  Timeisassociatedwithpotentialforchangesinpracticeandnewevidencesoshouldbeadjustedforintheanalysis

•  Risingtideeffect–  Promptingoftheinterventionorsentimenttowardsoverallimprovementmayimpacttheeffectside

HemmingKetal.BMJ2015;350:h391

Fig 1 Schematic illustration of the conventional parallel cluster study (with variations) and the stepped wedge study.

K Hemming et al. BMJ 2015;350:bmj.h391

©2015 by British Medical Journal Publishing Group

QIDesignsThatWork•  Interruptedtimeseriesdesign

– Quasi-experimentaldesign–  Evaluateslongitudinaleffectsofinterventionsthroughregressionmodelling

•  Determineswhethertheintervention’seffectispresentbeyondtheunderlyingtrend– Wellthoughtout/definedtimelinesforinterventions

•  Multiplemeasurementsbeforeandafter– Allowsadeterminationofregularvariationandtrends

•  Interventioncanbestoppedandstartedmultipletimesorsingularly

KontopantelisEetal.BMJ2015;350:h2750

Helderetal.BMCHealthServicesResearch2013,13:417

QIDesignsThatWork

•  ControlledBefore-Afterstudies,uncontrolledBefore-Afterstudies

•  Prospectivedesign•  Specificdatacollectedandoutcomesmeasuredbeforeinterventionandagainafterimplementation

•  Differencescanbeassumedtoberelatedtotheintervention–  Difficulttoidentifyappropriatecontrolgroup–  Subjecttoconfoundersthatmaynotbemeasuredorunderstood

–  Unabletoconfirmrelationshipascausal

Fanetal.JAMA2010;304(20):2279-2287

Biases•  Clustereffect

–  Considerationforeffectofenvironmentorlocationofpracticethatinfluencestheoutcome

–  Improvedwhenabletocompareeffectsindifferentlocationsortypesofpractices

•  Commonbiases–  Attritionbias

•  Losstofollowup–  Allocationbias

•  Unclearallocationofpatientstointervention–  Selectiveoutcomereporting

•  Importantoutcomesomittedfromresults–  Contamination

•  Crossover,adoptionofthepracticeovertime

DataQuality•  Unitsofanalysis

–  Processandoutcomemeasures–  Clearlydefined,rationale

•  CollectionaspartoftheQIstudy–  Rigorousmethodsofdatacollection–  Reviewofdatacollection–  Trainingfordatacollectionandabstraction

•  Reportingofmissingdata–  Shouldreportsensitivityanalysesifmissingdataproportionislarge

•  Balancingmeasures–  Unintendedconsequences–  Effectsonotheraspectsofcare

Fanetal.JAMA2010;304(20):2279-2287

DataQuality

•  Followup•  CriticalinthesettingofQIinterventions•  Determinesthesustainabilityoftheinterventionandthepotentialapplicabilityinothersettings

DataQualityExample•  AcommunicationtooltoimprovesituationalawarenessandpromoteproactivecareisimplementedinaNeonatalICU

•  Carriedoutoverastudyperiodof6months,andfollowedupagain3monthsaftertheactiveintervention

•  Itdemonstratesadecreaseinneedfor‘rescue’therapiessuchassalinebolusesandinotropesduringtheactivestudyperiod

•  TheauthorspublishinareportthattheirinterventionwasasuccessandthisapproachshouldbeadoptedbyNICUsasastandardofcare

Generalizability•  WhenevaluatingQIresearchforyourownpractice,must

considerthecontext•  Aspectsofcontextinclude

–  Physicalaspectsàcircumstances,environment,equipment,skill,location

–  Culturalàmotivation,training,appetiteforchange,providers,leadership,organizationalclimate

–  Resourcesàfinancial,stakeholderbuyin•  Thearticleshouldrevealanunderstandingof

–  Why,WhenandWheretheQImethodsusedworkedmosteffectively

–  Assumptionsaroundthenatureoftheinterventionasapplicabletothecontexttheywereused

OtherQIResearchEvaluationTools

•  MUSIQ– ModelforUnderstandingSuccessinQuality

•  ConceptualmodelthatcanbeappliedtoQIresearchtoidentifyfactorsthatleadtosuccessinQIresearch.

•  Canbeused–  ProactivelyindesigningQIstudies–  Systematicwaytodescribeandevaluatethecontextualfactorsimpactingonaproject’ssuccess

–  Toguidethecollectionandanalysisofdata•  Identified25contextualfactorsthatinfluencethesuccessofQIresearch

KaplanHetal.BMJQualSaf2012;21:13-20

KaplanHetal.BMJQualSaf2012;21:13-20

OtherQIResearchEvaluationTools•  PromotingActiononResearchImplementationinHealth

Services(PARIHS)–  Proposesthatcertaincontextualelementsarerequiredinthesuccessfulimplementationofevidencebasedpracticeimprovements

•  Theoreticalframeworksuggeststhatimplementationofresearchintopracticeisafunctionof–  NatureoftheEvidence(E)–  Context/Environmentintowhichitisimplemented(C)–  NatureofhowtheprocessisFacilitated(F)

•  TheseelementsshouldbedescribedinhighqualityQIstudiestodemonstratetherigorrequiredofsuccessfulQIresearch

KitsonAetal.QualityinHealthCare1998;7:149-158

KitsonAetal.QualityinHealthCare1998;7:149-158

AdditionalConsiderations•  Interpretationofthedatashouldinclude

–  Impactofthechangeontheintendedoutcome–  Commentonthemechanismbywhichtheinterventionfacilitatedthechange

–  Impactuponthesystemitself–  Costevaluation(financial,workforce)–  Factorslimitingvalidityandotherlimitations– Assessmentofusefulness,andotherapplicationsofthework

–  Sustainability–  Futureimplications

Summary•  PublishedresearchinthefieldofQIinhealthcareshouldbe

subjecttorigorousframeworkstoensureaccurateandcomprehensivereportingoffindings–  Users’guidestothemedicalliteratureHowtouseanarticleaboutqualityimprovement

–  Squire2.0•  ImplementationoffindingsfromQIresearchshouldtake

intoaccount–  Theoriginalcontextinwhichtheimplementationoccurred–  Culturalfactors–  StudyDesign–  DataQuality–  Riskofbias