New methodological standards for Cochrane reviews

first output from the MECIR project

edit

Rachel Churchill

Co-ordinating Editor representative on Steering Group

School of Social and Community Medicine, University of Bristol

Julian Higgins

Methods Groups representative on Steering Group

MRC Biostatistics Unit, Cambridge, UK

& Centre for Reviews and Dissemination, University of York

Outline

• Background and purpose of MECIR

• Development of methodological standards for conduct of intervention reviews

• The first set of standards

• Some examples

• Next steps and discussions around implementation

Background

• Coordinated by MARS (Methods Application and Review Standards) – Co-convened by Julian Higgins and Rachel Churchill to

facilitate interaction between Methods Groups and CRGs; supports Editor in Chief and CEU

• Coordinating Group from MARS and the CEU (Rachel Churchill, Julian Higgins, Jackie Chandler, David Tovey and Toby Lasserson)

MECIR

• Stands for Methodological Expectations of

Cochrane Intervention Reviews

• Purpose is – to specify methodological expectations for Cochrane

protocols, reviews, and updates of reviews on the effects of interventions

– to ensure that these methodological expectations are supported and implemented across the Collaboration

Development process

MECIR Working Groups

1. Developing a question and deciding the scope of the review

2. Searching for studies3. Selecting studies and

collecting data4. Assessing risk of bias in

studies5. Analysing data and

undertaking meta-analyses

6. Interpretation and presenting results

Coordinating Group

1. WGs submittedlong list of items

2. WGs asked to consider Mandatory itemsvsHighly Desirablevs Good practiceandRationale for each

Collaboration-wide consultation

• All registered entities (16/6/11 – 8/8/11)

• 51% responded - CRGs (35/53) - MGs (8/15) - Centres (5/14) - Centre branches (2/15) - Fields (5/11) - CEU, TWG, Author rep

Consultation process

• Further work by Coordinating Group, before extensive consultation and feedback from all Cochrane entities

• All registered entities (16 June ‘11 – 8 August ‘11)

• 51% entities responded– CRGs (35/53)– MGs (8/15)– Centres (5/14)– Centre branches (2/15)– Fields (5/11)– CEU, TWG, Author rep

Consultation feedback (Aug – Oct 2011)

• High levels of support for most items

• Decision around status item description, expectation and rationale (merging or omitting as appropriate) and provided associated Handbook reference for each standard

• 100 standards, reduced to 80

• Provided responses to comments and final decision regarding status, description, and rationale for each item (see Annex 2)

www.editorial-unit.cochrane.org/mecir

Summary of MECIR standards• 80 standards for the conduct of CIRs structured

around: – Setting research question to inform the scope of the

review– Setting the eligibility criteria for including studies in the

review– Selecting outcomes to be addressed – Planning the review methods – Searching for studies – Selecting studies into the review – Collecting data from included studies – Assessing risk of bias in included studies – Synthesizing the results of included studies – Summarizing the findings – Reaching conclusions

Some examples

Example of one we got right

Pre-defining clear and unambiguous criteria for interventions and comparators (standard 7)

(Mandatory)Agreed/Disagreed:

• CRGs: 30 / 0• Centres: 7 / 0• Fields: 4 / 0• Methods Groups: 6 / 0

• Left unchanged

Define in advance the eligible interventions and the interventions against which these can be compared in the

included studies

Example of one we had to work on

Excluding studies without useable data (standard 40)

[Draft] (Mandatory)Agreed/Disagreed:

• CRGs: 23 / 5• Centres: 6 / 1• Fields: 4 / 0• Methods Groups: 6 / 0• Three suggestions to change to Highly Desirable

Do not exclude studies on the basis of there being no usable data (either because the data are inconvenient

format or because they are not reported at all)

Excluding studies without useable data (standard 40)

Hugely important

Clarify that this relates to the formal “exclusion” of studies from the review

We will not implement this in our group

RevMan will protest if a study is ‘included’ with no data at all

agree if the outcomes of interest were measured

as an outcome

somewhat confusing

should be rephrased

Excluding studies without useable data (standard 40)

(Mandatory)

Include studies in the review irrespective of whether measured outcome data are reported in a ‘usable’ way.

Some key ones

Some key conduct standards

• Consider important potential adverse effects: Mandatory• Clarify role of outcomes: Mandatory• Search trials registers: Mandatory• Double (independent) data extraction: Mandatory

• Risk of bias tables: Mandatory• Summary of findings tables: Highly desirable• The five GRADE considerations (study limitations, consistency

of effect, imprecision, directness and publication bias) to assess the quality of the body of evidence: Mandatory

The new stuff

Standards not currently in the Handbook

• Rerun or update searches for all relevant databases within six months before publication of the review or review update, and screen the results for potentially eligible studies: Mandatory, BUT

• Incorporate fully any studies identified in the rerun or update of the search within six months before publication of the review or review update: Highly Desirable

• Compare magnitude and direction of effects reported by studies with how they are presented in the review, taking account of legitimate differences: Mandatory

• Consider in advance whether issues of equity and relevance of evidence to specific populations are important...: Highly desirable

Other MECIR outputs

• Guidance on implementation of standards for updates to be produced– to be produced

• Reporting standards (for protocols, reviews, updates)– in draft– Working Groups to be invited to input– consultation: Collaboration and external (users)

• Good practices and Common errors– in draft– will continually evolve

Facilitating implementation

A mixture of • Distribution to entities and individuals• Handbook amendments

– prominent incorporation of standards throughout– extension/clarification of guidance as appropriate

• Training materials (Training Working Group)• Check lists for authors, editors, peer reviewers

– building on existing author checklist (Editorial Resources Committee)

• Modifications to RevMan• Baseline audit

Concluding remarks

• MECIR conduct standards are (almost all) clarifications of existing Handbook guidance

• Adopting the standards should ensure that Cochrane intervention reviews are consistently of high quality

• We hope you will support and implement them– and contribute to development of further expectations,

and future evaluations of them all• The standards will be updated over time• For more, see www.editorial-unit.cochrane.org/mecir• Finally, THANK YOU to all individuals and entities who have

contributed so helpfully so far to the development of the expectations...

Particular thanks to Working GroupsDoug Altman Mohammed Ansari (Methods lead, WG1)Sally Bell-SyerPatrick BossuytDeborah Caldwell Christopher Cates Rachel Churchill (Co-Eds lead, WG4)Mike Clarke (Co-Eds co-lead, WG2)Jan Clarkson (Co-Eds co-lead, WG6)Philippa DaviesMarina Davoli (Co-Eds lead, WG1)Ruth Foxlee Chantelle GarrittyDavina Ghersi (Co-Eds co-lead, WG2)Julie Glanville (Methods co-lead, WG2)Peter HerbisonJulian HigginsSophie Hill (Co-Eds lead, WG3)Toby LassersonEdith Leclercq Carol Lefebvre (Methods co-lead, WG2)Jessie McGowan

Rachel MarshallRuth Mitchell Donal O’Mathuna Anna Noel-Storr Georgia Salanti (Methods lead, WG5)Doug Salzwedel Margaret SampsonJelena SavovicHolger Schünemann (Methods lead, WG 6)Ian ShemiltNandi Siegfried Jonathan Sterne (Methods lead, WG4)Britta Tendal (Methods lead, WG3)David Tovey Peter TugwellLucy TurnerClaire Vale Julia WaltersHelen Worthington (Co-Eds lead, WG 5 and Co-Eds co-lead, WG6)Janelle Yorke