Modelling Stroke in the Laboratory - Separating Fact from Artefact The impact of sources of bias in animal models of neurological disease, and what we

Modelling Stroke in the Laboratory - Separating Fact

from ArtefactThe impact of sources of bias in animal

models of neurological disease, and what we should do about it

Malcolm MacleodSenior Lecturer, Centre for Clinical Brain Sciences

University of Edinburgh

1026

1026 interventions in experimental stroke

O’C

olli

ns

et

al A

nn N

euro

l 2006

1026603


Tested in focal ischaemia

O’C

olli

ns

et

al A

nn N

euro

l 2006

1026883374


Effective in focal ischaemia

O’C

olli

ns

et

al A

nn N

euro

l 2006

1026883550

97 18


Tested in clinical trial

O’C

olli

ns

et

al A

nn N

euro

l 2006

1026883550

97 171 3


Effective in clinical trial

O’C

olli

ns

et

al A

nn N

euro

l 2006

Where are we going wrong?

• Are animal experiments falsely positive?

• Have clinical trials tested the

conditions of maximum efficacy?

… and what, if anything, does this

mean for models of other diseases?

Back

gro

und

Control half dose full dose

Infa

rct V

olum

e

0

50

100

150

200

250

300

10-120 M 10-60 M

Animal data in stroke• There are huge

amounts of often confusing data

• Systematic review can help to make sense of it

• If you select extreme bits of the evidence you can “prove” either harm or substantial benefit

• However, if you have a precise and highly significant overall effect, then it is probably real

Hypothermia101 publications277 experiments

3353 animals

Bett

er

Wors

e

van d

er

Worp

et

al B

rain

2007

-100

0

100

200

Potential sources of bias in animal studies

• Internal validity

– Low sample size

• External validity– Publication bias – Are the models we use good models?

• Co-morbidities

Problem Solution

Selection Bias Randomisation

Performance Bias Allocation Concealment

Detection Bias Blinded outcome assessment

Attrition bias Reporting drop-outs/ ITT analysis

Cro

ssle

y e

t al, S

troke

200

8

Internal Validity NXY-059

Macl

eod

et

al, S

troke

2008

9 publications29 experiments

408 animalsImproved outcome by 44% (35-53%)

Internal ValidityHypothermia

van d

er

Worp

et

al B

rain

2007

Randomisation

Yes No

Blinded outcome

assessment

Yes No



Internal Validity Stem Cell based therapies

• Infarct Volume

• Neurobehavioural score:

Jen L

ees,

un

publis

hed





RandomisationStem Cell based therapies

Jen L

ees,

un

publis

hed

Blinded outcome assessmentStem Cell based therapies

Jen L

ees,

un

publis

hed

What does this mean?Modelling the efficacy of tPA

“Standard”HealthyMale Rat

No randomisationHalothane anaesthesiaQuantify infarct volume

with TTC

25%

“Co-morbid”Hypertensive

Male Rat

No randomisationHalothane anaesthesiaQuantify infarct volume

with TTC

12%

“Randomised”Hypertensive

Male Rat

RandomisedHalothane anaesthesiaQuantify infarct volume

with TTC

0%

Em

ily S

en

a,

In P

repara

tion

Reported Efficacy 36%

Corrected Efficacy <0%

Control half dose full dose

Infa

rct V

olum

e

0

50

100

150

200

250

300

Comparing interventions Modelling efficacy under standard

conditions

nosd

onor

sno

otro

pic

estrog

ens

tPA

FK50

6tir

ilaza

dnico

tinam

ide

minoc

yclin

eno

sinhibito

rm

elat

onin

nxy0

59

othe

r th

rom

bolyt

icsIL

1-RA

hypo

ther

mia

Stan

dard

ised

effi

cacy

-10

0

10

20

30

40

50

60

70

Type II error, β, (1-power)

• The risk of falsely concluding that a biological effect is not present because the study was not large enough reliably to detect such differences.

• The smaller the experiment, the greater the risk of a Type II error

• Small studies with increased risk of Type II errors – waste animals, time and money– may lead to avenues of research being closed

down inappropriately.

Chances that data from any given animal will be non-

contributory

Number of animals Power % animals wasted

4 18.6% 81.4%

8 32.3% 67.7%

16 56.4% 43.6%

32 85.1% 14.9%

assume simple two group experiment seeking 30% reduction in infarct volume, observed SD

40% of control infarct volume

Chances of wasting an animal

Number of animals per group

0 10 20 30 40

% a

nim

als

was

ted

0

20

40

60

80

100

How does stroke compare?

Randomisation

Blinded Outcome

Assessment

Sample Size

calculation

Stroke 36% 29% 3%

MND 31% 20% <1%

PD 12% 15% 0%

EAE 8% 15% <1%

Glioma

14% 0% 0%

Sen

a et

al 2

007

TiN

S;

Am

aras

ingh

et

al,

J N

’Onc

in p

ress

Efficacy of Dopamine agonists in suppressing induced rotational activity following unilateral

6-OH-DA lesioning

Fer

guso

n et

al,

in p

repa

ratio

n

Sup

pres

sion

of

rota

tiona

l act

ivity

Quality items scored

BetterWorse

Pre

cisi

on

29 publications 109 experiments

1596 animals Improved outcome by 31% (27-35%)

External Validity Publication Bias for FK506

Macl

eod

et

al, JC

BFM

200

5

External ValidityPublication bias

Sena e

t al, a

ccepte

d f

or

ESC

200

8

All

stud

ies

Nic

otin

amid

eT

hrom

boly

ticN

XY

-059

NO

S d

onor

Nos

Inhi

bito

rF

K50

6H

ypot

herm

iaM

elat

onin

Est

roge

nsT

irila

zad

IL1-

RA

Effi

cacy

0

20

30

40

50

60ObservedAdjusted

991 publications

External Validity Hypertension in studies of NXY-059

Macl

eod

et

al, S

troke

in

pre

ss

7% of animals studied had hypertension

77% of patients in SAINT II had a history of hypertension at study entry

External Validity Hypertension in studies of tPA in

experimental stroke

Pere

l et

al B

MJ 2007

Comorbidity

“Normal” HBP

Effi

cacy

-2%25%

113 publications 212 experiments

3301 animals Improved outcome by 24% (20-28)

Summary

• Certain aspects of the design of animal experiments probably do lead to the over-statement of neuroprotective efficacy

• A substantial publication bias is present

• Neuroprotective efficacy may be substantially lower in animals with relevant co-morbidities

26%

Publication bias

Randomisation

Co-morbidity

bias

32%

Reported efficacy

How much efficacy is left?

20% 5%

Quality of Translation tPA and tirilazad

• Both appear to work in animals

• tPA works in humans but tirilazad doesn’t

• Time to treatment: tPA:– Animals – median 90 minutes– Clinical trial – median 90 minutes

• Time to treatment: tirilazad– Animals – median 10 minutes– Clinical trial - >3 hrs for >75% of patients

Sena e

t al, S

troke

2007;

Pere

l et

al B

MJ 200

7

tPA: Effect of time to treatment on efficacy

Pere

l et

al B

MJ 2007

; La

nce

t 2

004

AnimalStudies

Systematic Review

AndMeta-analysis

• how powerful is the treatment?

• what is the quality of evidence?

• what is the range of evidence?

• is there evidence of a publication bias?

• What are the conditions of maximum efficacy?

Clinical Trial

Summarising data from animal experiments

ST

AIR

VI:

pos

sibl

e de

velo

pmen

ts

A toolkit for effective translation

• Clear, rigorous SOPs for all aspects of experimental design

• On-line tools for– Sample size calculation– Random allocation to group

• Curated data repository– ? List of interventions tested (~ stroketrials.org)– ? Details of individual experiments (~ GENBANK)

• Development of experimental methods and funding streams to support multi-centre animal studies

• Adoption of CONSORT statement for animal stroke studies

ST

AIR

VI:

pos

sibl

e de

velo

pmen

ts

Mac

leod

et

al,

Str

oke/

JCB

FM

/IJS

in

pre

ss• Sample size calculation• Animals used• Inclusion and exclusion criteria• Randomisation• Allocation concealment• Reporting of animals excluded from

analysis• Blinded assessment of outcome• Reporting potential conflicts of interest and

study funding

Acknowledgements

Emily SenaPeter Sandercock

H Bart van der Worp

David HowellsTori O’CollinsGeoff Donnan

Nicolas CrossleyUlrich Dirnagl

Laura GrayPhilip Bath

Pablo PerelIan Roberts

Documents

Modelling Stroke in the Laboratory - Separating Fact from Artefact The impact of sources of bias in animal models of neurological disease, and what we