EMMIE training workshop

Evidence appraisal in crime prevention

Agenda for workshop

12.30 Introduction to EMMIE and rationale of approach (Shane Johnson)

12.45 Evidence appraised to date

13.00 The coding instrument

13.30 Exercise

14.30 The EMMIE narratives (and meta-synthesis)

15.00 Finish

Profile of (original) reviews

Focus of reviews

326 reviews

~125 on single

intervention topics

~60 interventions

in total

+12 reviews in production

Profile of (single intervention) reviews

Reviews appraised so far

1. CCTV

2. Lighting

3. Multi-systemic therapy

4. Alcohol ignition interlocks

5. Sobriety checkpoints

6. CPTED (retail robbery)

7. Neighbourhood Watch

8. Music making interventions

9. Electronic monitoring

10. Increased police patrols

Experience of applying EMMIE

• Most reviews don’t use the language of EMMIE

– Different fields have very different reporting conventions

• The evidence is generally weak on effect, and often on other

dimensions

– But need to remember that reviews rely on primary study evidence

• BUT, weak evidence on effect doesn’t undermine other dimensions

– I.e. reviews can be strong on moderators or implementation

• Appraising quality is subjective, so we automated the scoring

Experience of applying EMMIE

Codebook has constantly been challenged and refined

• Effect: meta-analyses conducted in various ways

• Mechanism: presented (or not) in many different ways

• Moderators: working out a priori / post hoc can be challenging

• Implementation: teasing this out from MM tricky sometimes

• Economics: evidence rare on this

The coding instrument

• EMMIE-E - relates to the ‘evidence’ that emerges from reviews

• EMMIE-Q – relates to the ‘quality’ of it

• Both are needed for prospective users to gauge what is (not) known

and a level of confidence associated with findings

• Coding instrument appraises both of these for each dimension of

EMMIE

Exercise

• 4 groups

• Identify information on EFFECT and one other dimension

• 45 mins reading, discussing and annotating

• 15 minutes for group discussion

The EMMIE narratives (and meta-synthesis)

• Turning the coding into an accessible format

• Quality assurance

• The narratives

Summarising the effect

Rating Interpretation

X X Overall, evidence suggests an increase in crime

X X ! Overall, evidence suggests an increase in crime (but some

studies suggest a decrease)

X X Overall , no evidence to suggest an impact on crime (but some

studies suggest an increase)

X X No evidence to suggest an impact on crime

X X Overall, evidence suggests no impact on crime (but some

studies suggest either an increase or a decrease)

X X Overall, evidence suggests no impact on crime (but some

studies suggest a decrease)

X ! Overall, evidence suggests a decrease in crime (but some

studies suggest an increase)

X X Overall, evidence suggests a decrease in crime

Summarising EFFECT-Q

Star rating for Effect Q Associated text

★★★★ The review was sufficiently systematic that

most forms of bias that could influence the

study conclusions can be ruled out.

★★★★ The review was sufficiently systematic that

many forms of bias that could influence the

study conclusions can be ruled out.

★★★★ Although the review was systematic, some

forms of bias that could influence the study

conclusions remain.

★★★★ Although the review was systematic, many

forms of bias that could influence the study

conclusions remain.

★★★★ Text to reflect specific review

Star rating for

Moderator Q

Associated text

★★★★ Collection and analysis of relevant data relating

to theoretically grounded moderators and

contexts

★★★★ Theoretically grounded description of relevant

contextual conditions

★★★★ Tests of the effects of contextual conditions

defined post hoc using variables that are at

hand

★★★★ Ad hoc description of possible relevant

contextual conditions

★★★★ No reference to relevant contextual conditions

that may be necessary

Summarising MODERATOR-Q

Meta-synthesis

• Synthesis methods needed for integrating reviews

• Two overarching decision rules for narratives:

1. For each EMMIE element use the HIGHEST QUALITY (highest Q score) scoring

review to populate the EMMIE-E and EMMIE-Q scores

2. For each piece of information make it clear which source is being referred to

• Use sub-group analysis FROM ALL REVIEWS to work out the ‘inside’

cross and tick

– i.e. evidence of statistical reduction or backfire under certain conditions)

– Use the HIGHEST QUALITY meta-analysis to populate the overall effect

Thanks everyone