Cochrane Systematic Reviews & Cochrane Oral Health Group

Luisa FernandezAnne-Marie Glenny

Cochrane Oral Health Group, University Dental Hospital of Manchester

Emma TavenderCochrane EPOC Satellite, Melbourne Australia

Aims

To discuss the role of Cochrane systematic reviews and meta-analyses

Cochrane Oral Health Group – progress, undertaking a systematic review with the group & looking to the future

Cochrane Systematic Reviews & Oral Health

Anne-Marie GlennyCochrane Oral Health Group, University Dental Hospital of Manchester

What are systematic reviews?

• The process of systematically locating, appraising and synthesising evidence from scientific studies in order to obtain a reliable overview

• Aim to find all studies addressing the review’s question using an objective and transparent process

Why are they important ?

• Reduce large quantities of information into manageable portions

• Formulate policy and develop guidelines• Efficient use of resources• Increased power/precision• Limit bias and improve accuracy

Cumulative 0.5 1 2Year RCT Pts I I I I I I I I I I I

1 231960 2 85

1965 3 149

4 1381970 7 1783

10 2544 11 26511975 15 3311 17 3929 22 5452

23 57671980

27 6125 30 63461985 33 6571 43 21050 54 22051 65 47185 67 475311990 70 48154

Ro

uti

ne

Sp

ecif

ic

Rar

e/N

ever

Exp

erim

enta

l

No

t M

enti

on

ed

21

8

1 10

1 2

2 8

7

8

1 12

1 8 4

1 7 3

5 2 2 1

15 8 1

6 1

p<.01

p<.001

p<.00001

Odds Ratio (Log Scale)

Favours Treatment Favours Control

Textbook/ReviewRecommendations

A. Thrombolytic Therapy

Systematic review

• Structured process involving several steps:1. Well formulated question

2. Comprehensive data search

3. Unbiased selection and abstraction process

4. Validity assessment of papers

5. Synthesis of data

1. Well formulated question

• Participants

• Interventions (Exposure)

• Comparisons

• Outcomes

1. Well formulated question

• Participants• Interventions• Comparisons• Outcomes

Pit and fissure sealants versus placebo for the prevention of dental caries in permanent teeth in children and adolescents

Inclusion criteria

• Participants

• Interventions

• Comparisons

• Outcomes

• Study design/methodological quality

What type of study design?

How effective is paracetamol at reducing pain?

Does smoking increase the risk of oral cancer?

STRONG Experimental studies/ clinical trialsRandomised controlled trialsNon-randomised controlled trials

Observational studies

CohortsCase-controlsCross-sectional surveysCase seriesCase reports

WEAK Expert opinion, consensus

2. Search strategy

• Needs to be as comprehensive as possible• Consider

– Electronic databases (Cochrane Controlled Trials Register, Medline, Embase)

– Reference lists

– Handsearching

– English language/non-English language

– Sources of ongoing and/or unpublished studies

Reporting biases

• Statistically significant ‘positive’ results are; - more likely to be published

- publication bias

- more likely to be published rapidly- time lag bias

- more likely to be published in English- language bias

- more likely to be cited by others- citation bias

3. Unbiased selection and data abstraction process

• Selection of relevant papers

3. Unbiased selection and data abstraction process

• Selection of relevant papers

• Data abstraction/extraction

Data extraction is:

• Time consuming

• Often subjective

• Prone to error

• Often difficult

3. Unbiased selection and abstraction process

• Predefined data abstraction form

• Independently and in duplicate

4. Validity assessment

• Can be used;– As a threshold for inclusion of studies– As a possible explanation for differences in

results between trials– In sensitivity analyses– As weights in statistical analysis of the results

Quality assessment tools

• Composite scales - assign numerical value to individual items to provide overall estimate of quality – problematic

• Component approach - assesses relevant methodological aspects individually (e.g randomisation, blinding, drop-outs) - preferred

4. Validity assessment

• Process should be conducted independently by at least two reviewers

• Results of the quality assessment should be reflected in the analysis

5. Study synthesis

• Appropriate pooling– qualitative (narrative)– quantitative (meta-analysis)

– inappropriate when data are sparse or heterogeneity exists

– Clear presentation of individual studies included in the review

Meta-analysis

• The process of using statistical methods to combine the results of different studies.

• The aim is to integrate the findings, pool the data, and identify the overall trend of results

(Dictionary of Epidemiology, 1995)

What is a meta-analysis?

Optional part of a systematic review

Systematic reviews

Meta-analyses

When can/should you do a meta-analysis?

• When more than one study has estimated an effect• When there are no differences in the study

characteristics that are likely to substantially affect outcome

• When the outcome has been measured in similar ways

• When the data are available (beware when only some data are available)

Summary statistic for each study

• Calculate a single summary statistic to represent the effect found in each study

- usually displayed with 95% confidence intervals (CI)

Weighting studies

• More weight to the studies which give us more information– More participants– More events– Lower variance

• Weight is proportional to inverse variance

Displaying results graphically

• Forest plots– Commonly used

there’s a label to tellyou what the comparisonis and what the outcomeof interest is

Alderson 2002

At the bottom there’sa horizontal line. This is the scale measuringthe treatment effect.Here the outcome is deathand towards the left thescale is less than one,meaning the treatmenthas made death lesslikely.

Take care to read whatthe labels say – things tothe left do not always meanthe treatment is better thanthe control.

Alderson 2002

The vertical line in themiddle is where thetreatment and control have the same effect – there is no differencebetween the two

Alderson 2002

For each study there is an id

The data foreach trial are here, divided into the experimental and control groups

This is the % weightgiven to thisstudy in the pooled analysis

Alderson 2002

•Each study is given a blob, placed where the data measure the effect.•The size of the blob is proportional to the % weight •The horizontal line is called a confidence interval and is a measure of how we think the result of this study might vary with the play of chance. •The wider the horizontal line is, the less confident we are of the observed effect.

The label above the graph tells you what statistic has been used

The data shown in the graph are also given numerically

Alderson 2002

The pooled analysis is given a diamond shapewhere the widest bit in the middle is located at the calculated best guess (point estimate), and the horizontal width is the confidence interval

Note on interpretation

If the confidence interval crosses the line of no effect, this is equivalent to saying that we have found no statistically significant difference in the effects of the two interventions

Alderson 2002

Heterogeneity

• Clinical heterogeneity – differences in trial characteristics

• Statistical heterogeneity - the variability in the reported effect sizes between studies

– how similar are the results?

– are the differences among the results of the trials greater than could be expected by chance alone?

Heterogeneity

Odds ratio (95% CI)

1 1

Odds ratio (95% CI)

Heterogeneity

• Chi-squared test of heterogeneity

• P<0.1 demonstrates statistically significant heterogeneity– may not be appropriate to pool data

Subgroup analyses

• Where it is suspected in advance that certain features may alter the effect of an intervention

Example – women – a particular age group– those with a specific disease subtype

Subgroup analysis

• Often misleading-

• Is there indirect evidence in support of a difference?

• Did the hypothesis about the difference precede rather than follow the analysis?

• Is the subgroup analysis one of a small number of hypotheses tested?

Sensitivity analysis

• Does result change according to small variations in the data and methods?– Choice of treatment effects or method for

pooling– Inclusion/exclusion of dubious data– Inclusion/exclusion of trials

A common sensitivity analysis is to repeat the analysis taking out lower quality trials

Useful websites;

http://www.shef.ac.uk/~scharr/ir/netting/(comprehensive list of E.B.P. websites and links)

http://www.york.ac.uk/inst/crd– CRD manual on how to conduct a systematic review– DARE - database of abstracts of reviews of effectiveness

http://www.cochrane.org– reviewer’s handbook– Cochrane Library (abstracts only)

Cochrane Oral Health Group – looking to the future

Emma TavenderCochrane EPOC Satellite, Melbourne Australia

Aims

• Progress of the group

• Process of undertaking a systematic review with the group & support available

• Challenges for the future

• Questions & answers

Cochrane Oral Health Group Include all RCTs of oral health broadly conceived to include

the prevention, treatment and rehabilitation of oral, dental and craniofacial diseases and disorders

Achievements• 624 members from 40 countries

• Panel of referees

• Set up Specialised Register of Trials (21,000)

• Handsearching programme

• Annual Evidence Based Dental Practice Course

• NIDCR funding for oral cancer reviews

Oral Health Group Reviews

0

10

20

30

40

50

60

70

80

Cochrane Library Issue

Nu

mb

er o

f Pu

blic

atio

ns

Updates

Reviews

Protocols

68 Protocols

54 Reviews

21 Updates

Growth of Oral Health Group Trials RegisterJanuary 1998 – January 2006

3946

5801

7756

12248

1476

918

245 201

13208

92

Reports of Trials

Jan-98Jan-99Jan-00Jan-02Jan-03Jan-04Jan-05Jan-06

Editorial Process• Register title

• Prepare protocol

• Editorial and external review of protocol

• Protocol entered on Cochrane Library

• Identify trials

• Complete systematic review

• Peer review of systematic review

• Systematic review entered on Library

• Update the review regularly (every 2 years)

Editorial support• Register title

– Check for overlap– Find co-reviewers with similar interests– Assigned Contact Editor

Editorial support• Prepare protocol

– Reviewers Handbook – RevMan software/support at editorial base– Protocol training workshop (free of charge)– Online learning materials– Help with developing search strategy (TSC)

Editorial Support• Identify trials

• Complete systematic review– Search the OHG Trials Register

– Analysis workshop (free of charge)

– Statistical & methodological support from editorial base

– Help with translations

Editorial support• Update the review regularly

– Help with identifying new trials– Procedures for changing lead author

Challenges for the future

• Prioritisation of reviews

• Diagnostic reviews

• Reviews of no trials – including other levels of evidence?

• Umbrella reviews

• Funding

• Dissemination

Challenges for the future

• Prioritisation of reviews– Increasing number of reviews/overlap– Usefulness – answering the right questions for

clinicians, consumers, funders, policy makers?– NIDCR funding of oral cancer reviews– OHG Symposium – identify questions/topics– Future – ways of identifying priorities for

consumers etc

Challenges for the future

• Diagnostic reviews– Important area within dentistry– Cochrane Diagnostic Reviews of Test Accuracy

Initiative– 4 phases: development of materials, piloting,

implementation & publication– Piloting phase – 10 test review teams– Training & implementation 2007

Challenges for the future

• Reviews of no trials – including other levels of evidence?– Scope of OHG includes RCTs and Quasi RCTs– Reviews eg. Endocarditis included other levels

of evidence– OHG currently developing guidance for

reviewers on how to deal with reviews of no trials and when to include other levels

Challenges for the future

• Umbrella reviews– Combine the findings of a number of related

reviews so information is in one document– Incorporate summary of findings tables– RevMan 5 released in 2007 will include this

feature– Cochrane Umbrella Reviews Working Group

Challenges for the future

• Funding– 5 years funding from DoH, UK– Covers salary of staff, administration at

editorial base– NIDCR grant to undertake series of oral cancer

reviews– NICE, Italian Govn., vCIOH– Alternative methods?

Challenges for the future

• Dissemination– The Cochrane Library– Journal articles & by products – Courses/conferences– Incorporation in guidelines (HTA, NICE,

SIGN)– Improve visibiltiy and utilisation of reviews

Any questions?