How can we improve the quality of Cochrane Reviews

(in 8 minutes)?

David Tovey

What is quality?

?

Valid

Relevant

Accessible

Timely

Moving from process to end point

Moving from process to end point

What is a “good enough” review?

“You should see the rubbish that is submitted to us..”

“And the authors are volunteers....so what can we do?”

Hmmm....what would she do?

What does these chaps think about improving quality?

What’s the plan?

Agree standards

Set standards

Measure against standards

Act

And training....

Results: 3 priorities

• Presenting effects

• Significance and non significance

• Harms

Absolute and relative effects

“If Cochrane reviews continue to express results solely in [relative] terms, they will continue to mislead clinicians, reporters, and the general public in just the way the pharmaceutical and vaccine companies would like.”

“****s were significantly more effective for [outcome] than placebo (OR 3.20, 95% CI:2.85, 4.27).”

“ A recommended approach is to re-express an odds ratio or a risk ratio as a variety of NNTs across a range of assumed control risks...”

“If a relative effect measure ..is chosen for meta-analysis, then a control group risk needs to be specified as part of the calculation of an ARR or NNT”

No evidence of an effect?

“Combined data from the ... studies revealed a non-significant difference between groups”

“There was a trend towards an increase in risk of [outcome]”

“Review authors are advised not to describe results as not statistically significant or non significant”

0 Favours interventionFavours control

0 Favours interventionFavours control

0 Favours interventionFavours control

Harms“A Cochrane review that considers only the favourable outcomes of the interventions....will lack balance and make the intervention look more favourable than it should..

Different types of studies may be needed to evaluate different outcomes”

Harms“A Cochrane review that considers only the favourable outcomes of the interventions....will lack balance and make the intervention look more favourable than it should..

Different types of studies may be needed to evaluate different outcomes”

“Observational studies are almost always necessary to assess harms adequately”1

1Chou, Aronson, Adkins et al 2010

Conclusions

• Review quality is of paramount importance (Strategic review recommendation #1)

• Quality is multi-faceted

• We need to work together to identify, agree and implement explicit review quality standards – Across review groups (warranted and unwarranted

variation)