Systema(c review Maj.Picha Suwannahitatorn, MD PhD

Department of Parasitology

Phramongkutklao College of Medicine

Overview

•  Over2,000,000ar*clesarepublishedinover20,000journals,annuallyinthebiomedicalliterature.

•  Noscien*ficstudystandsalone-itisonepieceofajigsaw.

2

Framework

•  Whatpreciseques*onsdrivesthereview?• Whatistheaverageeffect?• Whobenefitsmost?• Willthetreatmentworkhere?

•  Whatstudiesshouldweinclude?• Allstudies?• Stra*fiedbykeycharacteris*cs?• Expertselec*on?

•  Shouldfindingsbeadjustedforpublica*onbias?• Trackdownallstudies• Sta*s*caladjustment

3

What is evidence-based medicine (EBM)?

•  Theconscien*ous,explicitandjudicioususeofcurrentbestevidenceinmakingdecisionsaboutthecareoftheindividualpa*ent’(SackeO,1996).

4

Two types of knowledge

• MichaelPolanyi(1964)madeatheore*cal/philosophicaldis*nc*onbetweentwotypesofknowledgeusednotonlybypeopleineverydaylifebutalsobyhealthcareprofessionalsduringprac*ce.•  Tacitknowledge•  Explicitknowledge

5

Tacit knowledge

•  Subconsciouslyunderstoodorapplied.• Difficulttoar*culate.• Developingfromdirectac*onandexperience.•  Sharedthroughconversa*on,story-tellingetc.

6

Explicit Knowledge

• Canbepreciselyandformallyar*culated.•  Easytocodify,document,transfer,share,andcommunicate.

•  Evidence-basedmedicineiscallingforexplicitknowledge.

7

Defini(on

•  Systema*creview•  Areviewofaclearlyformulatedques*onthatusessystema*candexplicitmethodstoiden*fy,select,andcri*callyappraiserelevantresearch,andtocollectandanalysedatafromthestudiesthatareincludedinthereview.

• Meta-analysis•  Theuseofsta*s*caltechniquesinasystema*creviewtointegratetheresultsofincludedstudies.

8

Why carry out a systema(c review?

•  Thevolumeofscien*ficknowledgerelevanttohealthcareisvast.• Asinglestudyaddsdatatotheknowledgebase,butisrarelydefini*ve.•  Studiesuseavarietyofmethods,areofvariablequality,andmayappeartohavecontradictoryfindings.• Asystema*creviewefficientlyintegratesvalidinforma*onandprovidesabasisforra*onaldecisionmakingthatisbasedonthetotalityoftheavailableevidence.

9

Outcome

• Resolveconflic*ngevidence•  Explainvaria*onsinstudyresults• Answerques*onswheretheanswerisuncertain• Confirmtheappropriatenessofcurrentprac*ce

• Primaryaimofasystema*creviewistosummariseandhelppeopleunderstandtheevidence.

10

Steps in a systema(c review

•  Formulatethereviewques*on• Writetheprotocol•  Searchforandselectprimarystudies• Assessstudyquality• Abstractrelevantdata• Analysedata•  Interpretresults• Writereport

11

Contents of the protocol

• Backgroundinforma*onandobjec*ves•  Inclusion/exclusioncriteriaforstudies•  Searchstrategyforiden*fyingstudies• Methodsfordataabstrac*on• Assessmentofmethodologicalquality•  Sta*s*calmethodstobeused

12

PICO model

• Par*cipants

•  Interven*ons

• Comparisons

• Outcomes

13

Case I

•  Inves*gatetheeffectofEstrogenReplacementTherapyonOsteoporosisandBreastorEndometrialCancerinPost-MenopausalWomen?

14

Case I

•  Par%cipants

•  Interven%ons

•  Comparisons

•  Outcomes

15

Case II

• A28yearoldmalepresentswithrecurrentfurunculosisforthepast8months;theseepisodeshavebeentreatedwithdrainageandseveralcoursesofan*bio*cs,buttheyhavekeptrecurring.Heasksyouifrecurrencescanbeprevented?

• Amongpa*entswithrecurrentfurunculosis,doestheuseofprophylac*can*bio*cs,comparedtonotreatment,reducetherecurrencerate?

16

Case II

• Par%cipants

•  Interven%ons

• Comparisons

• Outcomes

17

Search strategy

•  Searchelectronicdatabases•  Usekeywordsin*tleandabstract•  Standardisedsubjecttermspar*culartothatdatabase

• Handsearch•  Manualpage-by-pagesearchoftheen*rejournal

• Conferenceproceedings•  Abstractsandothergreyliterature

18

Search strategy

• Checkreferencelistsofotherreviewar*cles• Printversionsofelectronicdatabases•  Forcita*onsbeforetheelectroniconesifitisthoughttheremayberelevantearlystudies

•  Iden*fyunpublishedstudies•  Maybeimportanttominimisebias,butnoeasywaytoobtaininforma*onaboutstudiesthathavebeencompletedbutnotpublished

19

Database

• Medline-world’slargestgeneralbiomedicaldatabase,indexes~1/3ofallbiomedicalar*cles• CINAHL-forNursingandalliedhealthstudies• Psycinfo–psychology/psychiatryrelated•  Embase–medicalandpharmacological• CochraneLibrary-fortherapiesandinterven*ons,providessystema*creviewsoftrialsofhealthcareinterven*onsandaregistryofcontrolledtrials

20

Cochrane Collabora(on

•  TheCochraneCollabora*onisaninterna*onalorganisa*onthataimstohelppeoplemakewell-informeddecisionsabouthealthcarebypreparing,maintainingandpromo*ngtheaccessibilityofsystema*creviewsoftheeffectsofhealthcareinterven*ons.

21

The Cochrane Library

•  ThisisthemainoutputoftheCollabora*on,updatedquarterlyanddistributedonanannualsubscrip*onbasisondisk,CD-ROMandviatheInternet.Itcurrentsincludesseveraldifferentdatabases:•  TheCochraneDatabaseofSystema*cReviews•  TheDatabaseofAbstractsofReviewsofEffec*veness•  TheCochraneControlledTrialsRegister•  TheCochraneReviewMethodologyDatabase•  TheCochraneCollabora*onsec*on

22

An effec(ve search strategy

•  Example:Whatisthemortalityreduc*onincolorectalcancerasaresultofperforminghaemoccultofthestool(faecaloccultbloodtest)screeninginwell-appearingadults?

23

An effec(ve search strategy •  Iden*fykeywords/concepts• PICOformat: •  Popula*onofinterest•  Interven*on•  Comparison•  Outcomeofinterest

Mortality

Colorectalcancer

Screening

Outcome

Interven*on

Popula*on

24

Organising studies

•  Keeptrackofallstudiesiden*fiedinthesearchinareferencemanagementsystemsuchasProCite,EndNote,ReferenceManager.•  Checkforduplicates

•  Astudymayonlybeincludedinananalysisonceotherwisebiaswillbeintroduced,thereforeimportanttoonlyincludeeachstudy(notreport)once

•  Searchcanbe*meconsumingandchallengingbutitisveryimportant,asystema*creviewcanonlybegoodifitincludesallrelevantstudies

25

Assessment of study quality

• Qualityofasystema*creviewlargelydependsonthequalityoftheprimarystudiesincludedinthereview• Qualityusuallyassessedduringdataabstrac*on• Qualityassessmentofstudiesmaylimitbias

26

Assessment of study quality

•  Factorstobeassessedarethoserelatedto•  Applicabilityoffindings•  Validityofindividualstudies(clearindica*onsofbias:incompletedata,selec*verepor*ng)

•  Certaindesigncharacteris*csthataffectinterpreta*onofresults(noclearsamplesize,senng,randomisa*on)

27

Possible biases

•  Selec*onbias•  Systema*cdifferencesbetweenthegroupsthatarecompared•  Systema*cdifferencesbetweenthoseselectedforstudyandthosethataren’t

•  AOri*onbias•  Systema*cdifferencesbetweencomparisongroupsinwithdrawalsorexclusionsofpar*cipantsfromtheresultsofastudy(mostcommon:par*cipantwithdrawal)

•  Detec*onbias•  Systema*cdifferencesbetweencomparisongroupsinhowoutcomesareascertained,diagnosedorverified(differentdiagnos*ccriteria)

28

Possible biases

•  Recallbias•  Arisesfrommistakesinrecollec*ngeventsbothbecauseoffailuresofmemory,andlookingatthings“withhindsight”andpossiblychangedviews(par*cularlycasesandcontrolsrememberingthingsdifferently)

•  Confounders•  Whichareimportantandaretheymeasuredandcontrolledforappropriately?

•  Repor*ngbias•  Causedbyonlyasubsetoftherelevantdatabeingavailable,eitherpublica*onbiasorsubgroupanalysis

29

Publica(on bias

• Publica*onbiasiscausedwhenonlyasubsetoftherelevantdataisavailable•  Failuretoincludeallrelevantdatamayover(orunder)es*matetheeffectofexposure/interven*on• Nullornon-significantfindings(especiallyinsmallstudies)arelesslikelytobereported/publishedthansta*s*callysignificantfindings

30

Data extrac(on and elimina(on •  Developedformtoabstractrelevantdetailsfromeachstudy

•  2reviewersindependentlyabstractedandcompared~250papers

•  Duplicatestudiesiden*fiedandexcludedbycomparing•  Authors•  Timeperiod•  Country•  Numbersofcasesandcontrols

•  Databaseof98studiesforanalysis

~ 600 papers

277 papers

298 papers

187 papers

111 papers

98 unique studies

Search

Preliminary screen of abstracts for data

21 Reference searching

Excluded

Included in meta-analysis

Abstraction

31

Meta-analysis

• Whenshouldwedoameta-analysis?• Whenmorethanonestudyhases*matedatreatmenteffect.• Whenthereareminimaldifferencesincharacteris*csacrossstudies.• Whentheoutcomehasbeenmeasuredinthesameway.• Whenthedataineachstudyareavailable.

32

Methods

• Oncethesetofstudieshavebeeniden*fied,foreachstudycalculatesummarysta*s*cs.•  Then,calculateaweightedaverageacrossthestudies,usingtheinverseofthevarianceasweights[recallthevarianceisthestandarderrorsquared]•  Thisgivesmoreweighttostudieswithsmallvariance(ietheinforma*vestudies)andlessweighttostudieswithhighvariance

33

What is Heterogeneity?

• Variabilityines*matesoftreatmenteffectsbetweenstudies•  SignificantHeterogeneitysuggeststhattrials(studies)arenotes*ma*ngasinglecommoneffect–possiblyduetodifferencesinpa*ents,interven*on,senngandoutcomes.

34

Types of Heterogeneity

• ClinicalHeterogeneity•  Varia*oninpar*cipants,interven*ons,outcomes,studydesigns

• Methodological•  Varia*oninmethodsusedinstudiesegalloca*onconcealment

•  Sta*s*cal•  Varia*onintreatmenteffectsandresultsofstudies

35

Assessment of Heterogeneity

•  Fixedeffects-assumesallstudiesarees*ma*ngthesametruemean-alldifferencesobservedareduetosamplingvaria*on• Randomeffects-assumesthestudiesthemselvesmaydiffer.Thisintroducesa‘between-studies’variancecomponent.Thisneedstobeincorporatedintothecalcula*onsoftheweightedmeanandvariance,andleadstowiderconfidenceintervals.

36

Assessment of Heterogeneity

•  Therearetwoapproachestoassessmentofheterogeneity;1.  Hypothesistes*ng-involvescalcula*onoftheQ

sta*s*ctotestwhethervaria*onbetweenstudiesismorethanonewouldexpectbychance.

2.  Es*ma*on-theI2sta*s*csprovidesameasureofthepropor*onofvaria*onduetothebetweenstudiesvaria*on.

37

Assessment of study quality

•  No“goldstandard”forthe“true”validityofastudy•  Qualityscale

•  Eachiteminascalegetsascore•  Overallqualityscorebyaddingupscoresofeachitem•  Complexandnotlikelytobetransparenttousersofthereview

•  Simpleapproach–usecertain“objec*ve”criteriatodecidewhetherastudyisgood,averageorpoorwithrespecttopoten*albias:

38

Scoring system

•  Developedtoassessrelevanceandqualityofstudies•  Simplescoringsystemminimisedsubjec*vityofqualityassessment•  Studiessplitinto3groups

•  Score1–studydoesnothavegooddesign(i.e.missinginforma*ontoohigh;responseratelow;noconsidera*ongiventoage)

•  Score2–studyofgooddesign,butinsufficientcontrolforconfounding

•  Score3–gooddesignandadequatecontrolforconfounding

39

Dealing with heterogeneity

•  Ifnoheterogeneityexists,afixedeffectsmodelcanbeusedtopooltheeffectes*mates•  Ifsomeheterogeneityexists,usearandomeffectsmodeltopooltheeffectes*mates•  Iftoomuchheterogeneityexists,itmightnotbeappropriatetopoolthestudies•  Usesubgroupanalysisormeta-regressiontoexplorepossiblecausesofheterogeneity

•  Becareful,subgroupanalysiscanbepronetorepor*ngbias

40

Test for heterogeneity

•  • Qhasanapproximateχ2distribu*ononI-1degreesoffreedom(Istudies)•  Lowpower• Combinethistestwithyourownknowledgeofthestudies

( )2ˆ∑ −= yywQ ii

41

Forest plot

• Mostcommonwayofpresen*ngtheresults•  Eye-ballexamina*onofresults• Getideaofbetweenstudyvariability• Plonngsymboltomarkthepointes*mateforeachstudyispropor*onaltoit’sprecision• Mostprecisees*mateshavethelargestplonngsymbols

42

Pointes*mateofeffectofstudy

Sizeofboxpropor*onaltotheprecisionofeachstudy

Widthofwhiskers=95%CI

Summaryes*mate

Widthofdiamond=95%CI

Centreofdiamond=Pooledpointes*mate

43

44

Things to concern..

•  Regressionsareowennon-linear,buttreatedaslinear.(Thedangerofextrapola*on).•  Effectsmaybemul*variateratherthanunivariate(e.geffectsofsmokingonlungcancer,ignoringpersonality).•  Qualityofstudies.Whatistheeffectofexclusions?•  Addingappliesandoranges•  Inappropriatecombina*onofstudiesduetoreviewerignorance(e.geffectoftherapyonbehaviour).•  Thetheory-directedapproach;individualstudiesmaybebiasedtowardsestablishingtheirhypotheses.

45

Advantages of a systema(c review

•  Useofexplicit,systema*cmethodslimitsbias(systema*cerrors)andreduceschanceeffects,thereforeprovidingmorereliableresults

•  Consistentresearchresultscanbeappliedacrosspopula*ons,senngsandsmalldifferencesintreatment(e.g.dose)

•  Inconsistenciesinresultsbetweenstudiescanbeiden*fiedandnewhypothesesmaybegeneratedaboutpar*cularsubgroups

•  Meta-analysiscanprovidemoreprecisees*matesthanthosederivedfromsinglestudies

•  Methodsareexplicitandareopentoscru*ny,sothatotherscanseewhatwasdoneandques*ontheresults

46

Summary

• Managingunmanageableamountsofinforma*on.•  Synthesis/comparisonofscien*ficevidenceisimportantinitsownright.•  Scien*ficefficiency:doweneedyetanotherstudy?•  Generalisa*onfromspecifictobroaderpopula*ons.•  Assessmentofconsistency•  Assessmentofinconsistency•  Increasedpower/precisionofes*mates•  Increasedaccuracyduetosystema*cversusnon-systema*creview.

47