Systematic Review:Workshop 1

26 July 2007

www.warwick.ac.uk/go/chrisbridle/sr

Chris Bridle, PhD, CPsychol Institute of Clinical Education Warwick Medical School University of Warwick

Tel: +44(24) 761 50222

Email: C.Bridle@warwick.ac.uk www.warwick.ac.uk/go/hpsych

Workshop Overview

Systematic review 70 mins

Break 15 mins

Programme structure 10 mins

Protocol development 45 mins

Questions -

Why is research synthesis needed?

2 million articles published in 20,000 health journals pa

Equivalent to a pile of paper 900 metres high

If you tried to stay current by reading two articles each day, within 1 year you would fall 55 centuries behind!

Put another way, if you try to read everything of possible relevance, you would have to read 5,500 articles per day.

Seeing is believing ...‘What burns me up is that the answer is right here somewhere, staring us in the face …

… if only Chris B was here!

381m

Expert Advice?

What and When?

What? Thalidomide

When? Before 1960

The evidence was there, but the methods for

making it accessible and meaningful were not

OR (fixed) ORStudy 95% CI (fixed) [95% CI]

101 3.47 [0.35, 14.51] 102 5.10 [0.24, 18.96] 103 2.43 [0.21, 12.71] 104 3.40 [0.35, 13.43] 105 2.43 [0.43, 11.69]

Total (95%CI) 3.06 [1.13, 8.31]

Heterogeneity: Chi²=0.23, df=4 (P=0.92)

Effect: Z=2.20 (P=0.03)

.1 0.2 0.5 1 2 5 10

Favours treatment Favours controlTreatment <Favours> Control (Fewer <Adverse Events> More)

◊

Don’t forget the slides handout!

Why we need research synthesis

Vast quantity of research Single trials rarely definitive

Hidden effects Research and practice built upon

biased &/or unfounded assumption

Need for research synthesis

How can research evidence be synthesized?

Chalk and Cheese

You Decide

Traditional Review: Take a simmering topic,

extract the juice of one argument, add the essence of one filling cabinet, sprinkle liberally with your own publications and sift out the work of noted detractors

A non-scientific method prone to bias in evidence retrieval, ‘evaluation’, synthesis and interpretation

Systematic Review:

A review of the evidence on a clearly formulated question that uses systematic and explicit methods to identify, evaluate and synthesise relevant primary research

A scientific methodology that is reproducible and amenable to rigorous evaluation

Formulate Review Question

Develop Protocol

Search, Identify & Select Evidence

Data extraction

Quality Assessment

Data Synthesis & Interpretation

Conclusions & Recommendations

Write Review Report

Key stages inconducting asystematic

review

Formulating a Review Topic / Question

Example question:

Effects of interventions for overweight children

This is a bad question! It is much too broad, would take years to answer, and

results unlikely to be meaningful

Needs to be narrower, i.e. more focussed What effects? Effects of what exposure? Among who? Compared to what / who? Etc., etc., etc.

Focussing the Review

Answerable questions include:

A description of the population P

An identified intervention I

An explicit comparison C

Relevant outcomes O

Overweight children

Community walking

Orilstat

Weight loss

Community walking versus orilstat for weight loss among overweight children

PICO Components

Identify the PICO components:

Effectiveness of homeopathy

Stage-based interventions for smoking cessation

Psychosocial interventions for prevention and treatment of childhood obesity

Effectiveness of atypical antipsychotics for treatment of mania in bipolar disorder

The Review Protocol

The review protocol is the first major milestone of any systematic review

Provides a rigid, well-specified plan for how each stage of the review will be conducted – a roadmap

Helps to avoid or minimise bias at each stage of the review – but only if followed rigidly

Should be sent for external peer review, e.g. advisory group comprising researchers, practitioners, users, etc.

Should be published, e.g. project webpage, research register, or peer-reviewed journal

Comprehensive Literature Search

Necessary to reduce bias:

Publication, language, and geographical

Search media:

Electronic, manual, and personal

Publication type:

Journal, conference, dissertation, grey, and ongoing

From question to search terms

For each PICO component identify and record:

Abbreviations, e.g. CBT, TTM, MS, CFS

Synonyms, e.g. CBT - Theory or Therapy

Related terms: cognitive theory/therapy, behavior theory/therapy

Transatlantic differences, e.g. pressure sores / decubitus ulcers

Spelling, e.g. behavior or behaviour

Set limits? Publication, e.g. published;

language; date, etc.

Study, e.g. sample size, follow-up time, type of outcome assessment, etc.

Evidence reliability, e.g. study design; specific criteria …

Hierarchy of evidence

Systematic Review

RCT

Cohort Studies

Case-Control Studies

Case Reports

Cross Sectional Surveys

Expert Opinion

Anecdotal

Experimental Study

Observational Studies

You, … and me

Drunk Bob

+ +

P

oten

t ial F

o r B

ias

- -

Question – Evidence Relationship

Question Type Relevant Evidence

Intervention - SR, RCT, Cluster RCT

Prognosis - Cohort study

Etiology - Cohort, case-control

Harm - Cohort, case-control

Diagnosis - Cross-sectional, case-control

Experience - Qualitative research

Building a Search Strategy

Record everything!* Search history, e.g. database, host and date Search strategy, e.g. terms and combinations Search results

Design a search-term table Enter search terms in relevant column Combine search terms for each column with ‘or’ = more Combine column results with ‘and’ = less

* See example

Preparing to be flooded

How will you manage the many (thousands) of

references and abstracts your search has identified?

Bibliographic software Endnote RefWorks ProCite RevMan

Study Selection

What references to retrieve? Study information limited to Abstract or Title Assess potential relevance of studies against inclusion / exclusion

criteria (i.e. P.I.C.O) – retrieve relevant full text papers

What studies to include? Use full text papers to assess the relevance of studies against the

study inclusion / exclusion criteria - PICO

The remaining papers are those relevant to your review

Keep precise records from start to finish Account for all references (studies)

Example: Flow chart of study

papers through a review

Study papers are to a review, what participants

are to a trial

Data Extraction

Recording of information presented in primary studies Ranges from superficial reporting to systematic recording Systematic extraction / recording is important for data

synthesis and interpretation

Important to strike the right balance between reporting / recording too much or too little information Determined by the ‘nature’ of the review and its intended

use / audience

Extracting Study Data

Important data to extract: Participants: demographic / disease characteristics

Intervention: description of intervention and control

Analysis: statistical method, number of Ps, ITT analysis

Results: be selective and sensible, e.g. primary outcome

Authors’ conclusions: word-for-word reporting of conclusions

Reviewer’s comments: validity of results, calculation of summary (effect) statistic, etc.

Participants Intervention Results Conclusion / Comment

N Randomised: 290 (I=150, C=140)

Age: m=43.

Gender: 30% female

Type: UK Community (Patient)

Recruitment: Non-smoking related attendance at GP surgery

I: 3 x 30 min weekly stage-based, group MI with take-home intervention pack.

C: GP advice

Provider: GP Nurse

Setting:Surgery

Follow-up:2 month

Outcome:

Abstinence 3 wks, self-report

Dropout:

82 (I=53, C=29)

N Analysed:

208 (I=97, C=111)

Abstinence:

31 (I=19, C=12) (p<0.05)

Reviewer:

ITT OR=1.54 (95% CI, 0.63 to 4.29)

Brief, stage-based MI with take-home material is an effective smoking cessation intervention.

Comment:

High attrition (I, OR=2.09) and ns effect with ITT

Unknown inclusion criteria

Unclear tailoring (group-MI)

Follow-up too short – 80% relapse/failure to 12 months

Authors’ conclusions are inconsistent with data

Data Extraction: Example

Tabulated Data: Example

StudyP’s Exposure Context Control

Outcome (Abstinence)

Summary effect OR (95%CI)

Validity

Smith, 2003

290, UK GP patients

Group MI + written advice

Nurse, GP surgery, 3 pw

Usual careSelf-report at 2

months1.54

(0.63,4.29) Poor

Jones, 2004

600, UK community

Group MIResearcher, community centre, 2 pw

No intervention

Biochemical validation at 12

months

1.03 (0.33,1.22)

Good

Davis, 2005

100, UK students

Stage-basedWritten material

No intervention

Self-report at 2 months

2.54 (1.33,4.89)

Poor

Scott, 2006

60, UK GP patients

Individual MICounsellor, home visit,

1pw

No intervention

Self-report at 1 month

1.87 (1.12,3.19)

Poor

Tabulated data are usually presented in the text of the report / paper

Critical Appraisal

Estimates potential for bias Extent to which results attributable to factors other than

named intervention

Independent application of a priori criteria Criteria depend on study design Common criteria for RCTs:

Adequate randomisation; Concealed allocation; a priori sample size calculation; Blinding; Baseline comparability; Adequate follow-up; Intention-to-treat analysis

Importance of Criteria

Quality Item % Overestimation ofintervention effect

Flawed randomisation 41

Unclear randomisation 30

Lack of blinding 17

No a priori sample size calculation 30

Failure to use ITT analysis 25

Trials with poor quality of reporting 20

Khan et al (1995)Moher et al (1998)

Uses of Quality Assessment

Determine the strength of conclusions

A threshold for inclusion

Potential source of heterogeneity

As weights in statistical analyses – unwise!

Recommendations for improving future research

Data Synthesis

Quantitative: statistically combine results of several studies into a

single numerical estimate of effect, e.g. meta-analysis Only used for (some) quantitative data

Qualitative: a narrative summary and synthesis of primary studies

Used for both quantitative and qualitative data

Selecting a Method of Synthesis

Decision based on degree of heterogeneity Variation within the group of included studies

Types of heterogeneity: Clinical: participants, interventions and outcomes, e.g.

age, intervention intensity, outcome definition, etc.

Methodological: how studies were conducted, e.g. study design, study quality

Statistical: measured intervention effect – you should calculate a summary statistics for each study yourself

Workplace exercise intervention for mild depression

Assessing Statistical Heterogeneity

Visual inspection Overlap of CIs

Chi-square statistic Significant p<.01 Power, too much, too little Statistic Vs degrees of freedom

Meaningless question Is heterogeneity present? Of course it is! We want to know how much heterogeneity is present, where,

and is it important?

Dealing With Heterogeneity

Subgroup analyses Comparison of separate meta-analyses based on

subgroups of the studies

Sensitivity analyses Repeating meta-analyses with amended criteria to

determine robustness of review findings

Subgroup analyses: Mortality results of zidovudine trials in HIV, stratified by infection stage (early or late).

Sensitivity analyses: Case-control studies relating residential exposure to electromagnetic fields to

childhood leukaemia, stratified by methodological quality.

Caution

Subgroup and sensitivity analyses need to interpreted with caution

Strong conclusions are often drawn from subgroup / sensitivity analyses –misleading and dangerous

Results based on indirect comparisons

Any differences are observational in nature

Prone to bias and confounding

Narrative Synthesis

Used for both quantitative and qualitative data but no formal method of synthesis

Studies meaningfully grouped and explored for potential sources of effect heterogeneity

Participants: age, gender, ethnicity, SES

Interventions: setting, provider, intensity, duration

Outcomes: definition, measured, collection, follow-up

Methods: quality, design, sample size, protocol adherence

Conclusions

Must be based solely on the evidence reviewed

Should highlight implications for research, practice, policy, decision makers, etc.

May be graded according to the strength of the evidence

May indicate a time period within which the findings of the review are relevant

Formulate Review Question

Develop Protocol

Search, Identify & Select Evidence

Data extraction

Quality Assessment

Data Synthesis & Interpretation

Conclusions & Recommendations

Write Review Report

Scoping Search

Develop Research Bid

Advisory Group

Advisory Group Review Drafts

Reasons for Exclusion

Independently Assessed

Check for accuracy

Independently Assessed

Investigate Heterogeneity

Core Systematic Review StagesR

evie

w P

roce

sses

- I

np

uts

Review

Ou

tpu

ts

PublishProtocol

Publish Review

Critical Appraisal of SRs

Question Is the review question clear, specific and answerable?

Search Have attempts to identify relevant evidence been sufficiently comprehensive?

Evaluation Have included studies been critically appraised against appropriate criteria?

Synthesis Is the method of synthesis appropriate, and have potential sources of heterogeneity been investigated?

Conclusions Do conclusions reflect both the quality and quantity of evidence?

Process Is the review process independent and transparent so as to limit potential bias?

Summary

Systematic reviews are important:

Meet the need for research synthesis

Use scientifically rigorous methods

Provide evidence-based answers to meaningful questions

Identify gaps and weaknesses in evidence base

Improve research quality

Enhance effectiveness of clinical practice

Strengthen bids for research funding

Good Guidance

Centre for Reviews & Dissemination: Report 4 www.york.ac.uk/inst/crd/report4.htm

Cochrane Collaboration: Reviewer’s Handbookwww.cochrane.org/resources/handbook/index.htm

Cochrane Behavioral Medicine: Recommended paperhttp://www.cochranebehavmed.org/

WMS Systematic Review Training Materialswww.warwick.ac.uk/go/chrisbridle/sr

Any questions?

Now, or later

Chris Bridle PhD, CPsychol Institute of Clinical Education

Warwick Medical School University of Warwick

Tel: (024) 761 50222 Email: C.Bridle@warwick.ac.uk

Web: www.warwick.ac.uk/go/chrisbridle

Programme Overview

Date Review Workshop Intervening Tasks & Objectives

July

Planning: Developing a review protocol; Formulating a review question / objective; Methods for conducting the review

Develop protocol with completed background, specified objective/question; Outline methods.

Submit draft for assessment, amend and submit to expert for comment.

Sept

Conducting: Search strategy; Data extraction; Quality assessment (Data synthesis; Heterogeneity; Interpretation)

Update protocol methods (search, extraction template, assessment criteria) and submit.

Conduct review as specified in approved protocol, submit draft review report (data synthesis plan).

Oct

Reporting: Report structure; Manuscript guidelines; Dissemination strategy; Targeted publication; (Data synthesis)

Synthesise data; Write draft report; Identify peer-review jrnl; Obtain manuscript guidelines.

Submit draft report and dissemination strategy for internal review and respond to feedback.

Nov

Disseminating: Publishing the review; Target audiences; Dissemination media; Responding to peer-review; Medical education

Present completed review to peers; Reflect on programme experience / evaluation & feedback.

Write full report for internal peer-review, submit for publication and go to the pub to celebrate.