Systematic Reviews Practicalities and Realities

Alison Brettle, Research Fellow (Information)

Salford Centre for Nursing, Midwifery and Collaborative Research

University of Salford

Session Overview

Practical overview of systematic reviews and what you need to conduct one

Discussion

What is a systematic review?

An overview of primary research studies conducted according to explicit and reproducible methodology

A rigorous method of summarising research evidence

Shows what we know and don’t know about a topic area

Provides evidence of effectiveness (or not) by summarising and appraising relevant evidence

Systematic reviews aim

To find all relevant research studies (published and unpublished)

To assess each study on basis of defined criteria

Synthesise the findings in an unbiased way

Present a balanced and impartial summary of the findings taking any flaws into consideration

Systematic review models

Medical/Health care Cochrane Collaboration, NHS Centre for

Reviews and Dissemination Usually includes “high quality” research

evidence – RCTs Often includes meta-analysis (mathematical

synthesis of results of 2+ studies that addressed same hypothesis in same way)

Social care/Social Sciences SCIE, EPPI Centre, Campbell Collaboration Often include wider range of studies including

qualitative Often narrative synthesis of evidence

Systematic review process

Define/focus the question Develop a protocol Search the literature (possibly 2 stages scoping

and actual searches) Refine the inclusion/exclusion criteria Assess the studies (data extraction tools, 2

independent reviewers) Combine the results of the studies to produce

conclusion– can be a qualitative or quantitative (meta-analysis)

Place findings in context – quality and heterogeniety of studies, applicability of findings

Task

You are part of a systematic review team and this is your first meeting. Your task is to draft a protocol by the end of the meeting then present to the other groups

Components of a protocol

Background/context leading to question Review question Inclusion/Exclusion criteria Methods for

Identifying evidence Selecting studies Extracting studies Assessing quality of studies Synthesising the results Disseminating the findings

Lessons learned in systematic reviews

Lindsey Dugdill, Alison Brettle, Claire Hulme, Serena McCluskey, Andrew Long

Workplace physical activity interventions: a systematic review of the research evidence

Background

Policy Increased PA important for

prevention of chronic illness Trend on a worldwide and UK scale

that PA levels are declining

Objectives To identify which types of workplace PA

intervention were effective in changing PA behaviour for different sectors of the workforce and different types of workplace

To ascertain which aspects of intervention design and delivery contributed to effectiveness and contributory motivators, barriers and facilitators

Lesson 1. Focus the question

More focussed the question, easier to undertake literature search and decide on relevant studies

Need to maintain a balance between a clearly defined topic and a meaningful and relevant question

Can help scope the literature to help develop the question

Refine question as part of review process

Focussing the question

Options: empirically led -> literature scoping ->

narrowing down topic on the basis of ‘what is there’ in the evidence base

theoretical and conceptual -> concept mapping, explore the meaning given to the concepts, embracing user and carer notions of desired outcomes (based on previous research) themselves central to a ‘social care’ perspective

Funder/policy driven

Lesson 2. Scope the literature

Provides an indication of literature available and what needs to be examined

Helps focus the question Helps clarify initial inclusion and

exclusion criteria

Lesson 3. Write a protocol

Plan what you are going to do in the review

Set out the background and objectives

Outline the resources you will use Establish inclusion/exclusion criteria How will data be extracted (what

will be extracted) How will you synthesise literature

Literature search

12 electronic databases 1996-2007Selected websites of known organisationsCitation tracking

Could also use hand-searching of journals “grey” literature

Lesson 4. Use multiple sources of information

Databases only part of the picture Handsearching identifies further

references Citation tracking adds more references SMI review – 96 studies (73 databases,

23 handsearching/citation tracking) Counselling – 47 studies (aprox 10

handsearching, 4 citation tracking)

Lesson 5. Refine your search plan as you carry out the literature searches

Theory Develop your search plan Everything that follows flows

smoothly and easilyPractice Modify your approach as the search

progresses Use spiral approach to refine Ongoing discussions within team

Search spiral

Scoping (initial database searches) Refining (indepth searches of range

of sources) Confirming (hand searches, citation

tracking)

Inclusion

Intervention aimed to increase PA Aimed at employed adults Initiated or endorsed by employer Outcome measures included a

measure of PA Evidence of behaviour change English Published post 1996 Took place in UK, Australia, NZ, Europe

Exclusion

Self employed or unemployed adults

Adults needing specialist advice regarding PA

No evidence of behaviour change or PA measure

Insufficient detail reported in study Cost effectiveness Located in US or Asia

Lesson 6. How will you refine the inclusion/exclusion criteria?

Tighter the criteria = less papers to review

BUT will your review draw any meaningful

conclusions? Focussing the question v quality of

studies

Overview of search and appraisal process

38 papers representing 33 studies Each paper appraised by 1 reviewer, using a

standardised protocol, 20% sample reviewed by 2 reviewers and each checked as put on database

Quality of papers ++ high, + good, - poor evidence (using protocols/guides provided by NICE)

Categorised by intervention and quality rating Summary table Conclusions on the whole derived from those

classified as high or good

Appraisal

Lesson 7: Know what you want to find out

Select your critical appraisal/data extraction tool carefully

Make sure it captures the elements that you wish to write about in your final report

Think what data need to extract Narrative review v meta analysis Quality of studies – what evidence are you

going to include? How are you going to assess the quality

Do you need to refine the inclusion/exclusion criteria as you go along?

Findings: published systematic reviews

Inconclusive review level evidence that workplace PA interventions were effective in increasing PA

Findings: Stairwalking

7 studies (1 high, 2 good, 4 poor) 4 demonstrated that posters and

signs can increase stair use in the short term

Little evidence of effectiveness

Findings: Walking interventions

4 studies (2 good, 2 poor) 3 used pedometers (objective

measure) All 4 also used self reported counts Evidence that workplace walking

interventions can increase daily step counts

Findings: Active travel

3 studies (1 good, 2 poor) Evidence from 1 study that a

walking and cycling to work campaign using written materials can increase walking to work in economically advantaged women

Findings: multi component interventions

16 studies (2 high, 1 good) Wide range of interventions Difficult to attribute which bit of the

intervention was causing the effect Limited evidence that counselling

has effect on workplace PA

Limitations

Quality of evidence Exclusion of studies from US and

Asia Most of studies took place in large

public sector workplaces Reliance on self report measures

Lesson 9. Set up a system to keep track of the review process

Use a package such as reference manager or endnote to keep track of references

Large reviews need method of keeping track of data extraction, exel, access or specialised software

Update at every stage of the process Compiles references at the end Need a hard copy system too for the actual

papers and a system for keeping track of results If you do not know where you are up to – it is

unlikely that your review will be systematic

System to manage process – the physical

Putting material into endnote Direct export from many sources From a file Direct input

How are you going to manage the hard copies

What about your notes? What systems do you use?

Reducing and organising

Using groups and adding notes to Endnote

Using a screening tool (once you’ve ditched the really irrelevant ones)

Data extraction/Critical Appraisal

Tools Many widely available Can adapt to suit your own study Could design own Assessing quality – systems available – be

explicit if you are going to do this Storing the data

Template in word/Table in word Excel spreadsheet Other?

Useful resources – systematic reviews

Cochrane Collaboration http://www.cochrane.org/ http://www.cochrane.org/docs/irmg.htm

Centre for Reviews and Dissemination http://www.york.ac.uk/inst/crd/

handbook for conducting systematic reviews, http://www.york.ac.uk/inst/crd/methods.htm Searching for systematic reviews http://

www.york.ac.uk/inst/crd/revs.htm EPPI-Centre – Stages of a review

http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=89 SCIE - The conduct of systematic research reviews for SCIE

knowledge reviews http://www.scie.org.uk/publications/details.asp?

pubID=111