Critical Appraisal Overview

Tues 10th May

Critical Appraisal

Master class

Critical Appraisal: WTF?

(well that’s fab)

Introductions

Elspeth Hill• MRes Medical Sciences• PhD Education Student• Journal Reviewer• Cochrane Author• Nervous wreck

James Durrand• BSc Pathology• MB ChB (nearly!)• Academic Foundation Post• RSM Rep• Massive geek

You?

Outline

1. Evidence-based medicine

2. Principles of critical appraisal

3. Jargon busting

4. Clinical studies

5. Lab stuff

6. Qualitative work

Evidence-Based Medicine

What is it?

Structure of EBM

Question

•What do I need to know?

Evidence

•What is known about this topic?

•What is in the literature?

Critical Ap

praisal

•Evaluation of the evidence

•What do we actually know?

Applicatio

n

•Try your suggestion based on evidence

Monitor

•How did that work out?

Critical Appraisal

Should I believe this paper?

ClinicalShould I treat my patients differently based on this paper?

EducationShould we include this in the medical school curriculum?

Lab ScienceShould we divert resources to investigate this further?

3 Keys of Appraisal

1. Validity

2. Applicability

3. Impact

ValidityValidity:How far has this paper actually answered the question they set out to answer?

Question: Does endovascular emergency AAA repair reduce mortality?

Answer:Endovascular emergency AAA repair is associated with reduced blood loss.

ApplicabilityApplicabilityDoes the evidence from this paper apply to my patients?

Trial TitleCleverstatin reduces cardiac events in male patients with type II diabetes. A randomized controlled trial

Population1000 patients in Helsinki, Finland

Should I change my Wigan patients to the more expensive cleverstatin?

ImpactImpactHow much of a difference would this make?

Trial TitleHigh flow oxygen increases mortality in acute myocardial infarction. A randomised controlled trial.

RecommendationPatients should no longer be given maximal oxygen therapy in acute MI.

I am the SHO in A&E: a man presents with crushing chest pain – am I really going to withhold oxygen?

How to approach a paper!

Case Study

First impressions

What journal?

•BMJ vs The Antarctic Journal of Homeopathic Resuscitation

•Impact factors

•Conflict of Interests

•Track Record

•Abstract?

Research Question

•What is the research question at the centre of this paper?

•Is it an important question?

•Did I learn anything from reading this paper?

Introduction

. Outlines and references preceding work and ‘the story so far’.

. Points out weaknesses and gaps in preceding work.

. Justifies need and usefulness of the study.

. Puts the paper in context.

Methods

What kind of a study is it?

Meta-analysis or Systematic

Review

Multiple RCTs or 1 really good one

Single RCTs

Cohort studies

Case-control studies/case series

Expert opinion

Expert opinion

Thoughts of one person who has seen many cases

Case Reports/Series

What happened with one patient?What happened with a few patients?

Case-control studies

Risk Factors

Time

Cohort studies

Time

Chewing Gum Lung Cancer

Smoking

A B

C

Confounding factors

Randomized controlled trials (RCTs)...Randomised…– minimise bias by randomly assigning subjects to groups

...Controlled… – compare your intervention to a control, where the only difference is

the intervention

…Trial… – you don’t know what’s going to happen

The Risk of Bias

Selection bias

Treatment bias

Observer bias

‘Systematic differences between the two groups’

Minimising bias

•Blinding

•Double blinding

•Triple blinding

•Losing will to live?

Results

•Groups at baseline

•Intention to treat

•Selective outcome reporting

•Subgroup analyses

•Side effects

DiscussionWhat does this mean?How was the data interpreted?What does this mean for practice?Does it fit with other studies?

Have they acknowledged/justified the flaws?

Conclusion

Is this a reasonable conclusion?Does it answer the research question?

Lab ResearchWhat is the research question?

Description – observationClarification – mechanism

Very similar to clinical studies, but watch out for relevance

cell biology vs animal studies vs human tissue vs human trials

• Does it follow from previous research – theoretical justification

• n numbers, selective reporting• are the results meaningful – does it matter

if the CRP is high if all the animals died?• good modeling – young animals do not

represent aged obese smoking Salford residents

• adequacy of controls – is that really the only difference

• outcome assessment

• are the controls normal• are the interventions different, how was

this assessed• is there another explanation

• how does it fit with existing research• do their conclusions support extrapolations

Qualitative/Education

Scientific methods - leaves on trees

Did they need to use words?•Well-problematized topic•Clear research questions – relevance to the real world?•Appropriate methodology - RCT in education•Clear recipe methods•Limitations and conclusions

Big problemSo many qualitative methods, but 90% is description, very little is justification, even less clarification

Take home messages

1. take nothing at face value

2. healthy skepticism

3. anyone can do it!

Further ReadingBooks:

How to read a paper. Greenhalgh T. BMJ books. Blackwell publishing

The doctors guide to critical appraisal. Gosall N, Gosall G. Pastest books

Bad Science. Ben Goldacre

Papers:

Why Most Published Research Findings Are False. John Ioannidis

Courses:

Research methods and critical appraisal- www.rsm.ac.uk (events)