What is a Rapid Evidence Assessment? What’s involved?Evidence Base Camp 2013

Levin WhellerPractice Development TeamResearch Analysis and Information Unit

This session

• Evidence reviews: The what, why and how…

• Evidence reviews in practice:Examples of recent Rapid Evidence Assessments

Evidence reviews:the what, why and how…

Why review evidence?

“The sheer amount of potential research evidence in most substantive areas of social science and public policy… make[s] it almost impossible to keep abreast of the research literature in any one area”.

Davies, 2003

Methods for reviewing evidence…Review type Systematic? Timeframe

Literature Review No 1-8 weeks

Scoping Review No 1-8 weeks

Rapid Evidence Assessment

Yes 2-6 months

Systematic Review Yes 8-12+ months

Multi-arm systematic review

Yes 12+ months

Review of Reviews Yes Variable

Literature reviews

Look! The breadcrumbs lead here, this MUST be the answer!

Literature reviews

AHAHAHAHA! I have tricked you into only

reviewing only *some* of the

available evidence!

Are we happy for professionals to only have some of the evidence when making decisions?

We need to look at all the evidence…• Antman et al, 1992.• Study comparing recommendations for treating heart

attacks based on literature reviews with recommendations based on a systematic meta-analysis.

• Literature reviews often failed to mention important advances or exhibited delays in recommending effective preventive measures.

• In some cases, treatments that have no effect on mortality or are potentially harmful continued to be recommended by several clinical experts.

Systematic Reviews

Systematic reviews

Taken from: Mazerolle, L., Bennett, S., Davis, J., Sargeant, E. and Manning, M. (2013) Procedural Justice and Police Legitimacy: A Systematic Review of the Research Evidence. Campbell Systematic Reviews 2013:1.

Overall, legitimacy interventions resulted in a large, significant increase in positive perceptions of police.

Policy opinions of systematic reviews

• Of those that had heard of them, some had concerns:

– Timeliness

–Relevance

–Usefulness• Some had difficulty distinguishing them from

literature reviews, even when explained

Campbell S et al (2007) Analysis for policy: evidence-based policy in practice

Rapid Evidence Assessments

• Use systematic principles and the same process as a systematic review…

• …but make compromises given available time and resources

• Pragmatic and transparent approach

The process (in a nutshell)

1. Draft search terms

2. Draft sift criteria

3. Sift received abstracts

4. Request relevant papers

5. Read and ‘grade’ papers

6. Write it up (‘synthesis’)

Some key principles

• Demonstrate consistency in searching/ sifting

• Document search and sift process– Process should be transparent and repeatable

• Specify required quality of evidence – Systematic reviews only?– Pre-post studies only?– All ‘empirical’ papers?

• Be explicit/ transparent about the limitations of the approach

Some examplesWhat it is like in practice?

One practical example• Review of Police Leadership and

Training commissioned by the Home Secretary

• CC Peter Neyroud needed the best evidence he could get on “What works in training and behaviour change?”

• Three weeks to deliver an evidence review– Not a full REA, but…– More than literature or scoping

reviews– Used systematic principles– Due to time limit, search

restricted to evaluations and systematic reviews only

Searching (Training)Search terms1. training OR learning OR development 2. AND evaluat* OR assess* OR what

works OR impact3. AND systematic review

Limitations• Searched 11 databases and 2

websites • English language only System

atic review

Evaluation

Training

Initial search identified 1,015 abstracts to sift

Sifting (Training)• Q1: Is the study about

adult training, learning, or development?

• Q2: Is the study: An evaluation (at least pre & post level) OR a systematic review?

• Secondary sift to remove papers related to inappropriate populations and specific medical conditions

Original references

1,015

After first sift

Less duplicates

After second sift

Available from NPL/ BL in time

for inclusion

38

32

22

10

Read, Appraise and Synthesise papers – What works in training?

What works (Good practice)

Strong evidence in healthcare shows training that is integrated into routine practice is more effective at changing individual’s attitudes and behaviour than traditional classroom based approaches.

Strong evidence from education shows that collaborative continuous professional development is effective in improving pupil outcomes (learning and behaviour) and the practice, attitudes and beliefs of teachers

What doesn’t work (Bad practice)

There is systematic review evidence that classroom-based training alone is not necessarily an effective way to improve practitioner’s skills or to change their behaviour

What’s promising (Promising/noteworthy practice)

There is some evidence that simulation-based training may have some advantage over more traditional classroom methods

What’s unknown There is a lack of clear evidence on the effects of reflective practice; portfolio learning; problem based learning; and learning technologies and virtual learning.

Limitations• Time– Availability of papers (10/22)– Unable to pilot search terms

• Available databases– Only those available to the NPL

• English language only

• Literature focussed in different areas– Almost nothing on policing– Papers mostly from healthcare

Organisational change and business improvement

• Forces are adopting business improvement techniques to examine current practices and explore scope to change processes to release savings

• Techniques include QUEST, CI, Lean, Six Sigma, Kaizen, etc.

• So – are these techniques the answer?

• Is that magic potion? Or is it snake oil?

Rapid Evidence AssessmentStep 1

Systematic search(11,960 abstracts)

Step 2Quality assess & critically appraise

(181 empirical studies)

Step 3Synthesise findings

(41 studies with useful findings)Draw conclusions

Example of the two searches in the organisational change and business improvement REA

Organisational change and business improvement

Question Approach Abstracts identified

Included papers

What works in delivering organisational change?

Rapid review of reviews

797 5

What works in organisational change and business improvement?

REA 11,163 36

Total of approx 12,000 abstracts; 181 full papers were requested; 41 were included…

Why so many papers?• Better search terms needed?• Dodgy descriptions of papers in abstracts?• Problems with searching?

Organisational change – what works?

• Potential success factors for organisational change

Leadership (and having a clear strategy)

Resources (financial, personnel and training)

Organisational culture and structure

Staff feelings of active participation (and related employee empowerment and strong teamwork)

Communication

…and prior experience of implementing a quality improvement programme

Organisational change: potential success factors Leadership

• stability of supervision throughout implementation • direct support from supervisors - ‘on-the job’ training • staff involvement in decisions • transformational leadership behaviour = reduced employee

cynicism

Engagement

• staff active participation in decision making & ‘room to experiment’

• degree to which staff understood rationale for change• communication found to influence self reports of job

performance

Lessons learned...• Piloting your searches is critical (we’ll do this tomorrow) to give

you an idea of the size of the job, and if there is much available material

• It’s important to focus on the end result and how will the findings be used – make it relevant…

• Recognise (and accept) there are limitations of the approach

• Make sure you are researching the right question

• Follow the key principles:– Demonstrate consistency in searching/ sifting– Document search and sift process– Specify required quality of evidence – Be explicit about the limitations of the approach– REAs should be replicable

The process (for Evidence Base Camp)

Stage Who When

Draft search terms Review teams EBC Day 2

Draft sift criteria Review teams EBC Day 3 (Nov)

Sift received abstracts

Review teams EBC Day 3 (Nov)

Request full papers NPL and researchers

After EBC Day 3 (Nov)

Read and grade full papers

Review teams EBC Day 4/5 (Mar)

Synthesise evidence

Review teams EBC Day 4/5 (Mar)

Terms Results

Anywhere in the article

“Domestic violence” 597,000

“Domestic abuse” 32,500

Title only

“Domestic violence" 23,700

“Domestic abuse” 1,220

Published since 2000 (title only)

“Domestic violence" 13,200

“Domestic abuse” 822

Adding ‘randomised controlled trial’

“Domestic violence" 0

“Domestic violence" 0

Adding ‘experiment’

“Domestic violence" 32

“Domestic abuse” 0

Adding ‘systematic review’

“Domestic violence" 7

“Domestic abuse” 0

• Search terms are hugely important.

• Tomorrow is all about developing search terms and seeing the impact of using different terms.

• We will be running live pilot searches in your groups with College librarians.

Useful links

Civil service REA toolkit:http://www.civilservice.gov.uk/networks/gsr/resources-and-guidance/rapid-evidence-assessment

Campbell Collaboration (social interventions, e.g. crime and justice)

http://www.campbellcollaboration.org/

Cochrane Collaboration (medical interventions)

http://www.cochrane.org/

EPPI centre (education)

http://eppi.ioe.ac.uk/cms/