Summarising the evidence from animal models of neurological disease: Are there any babies in the...

Summarising the evidence from animal models of neurological disease: Are there any babies in the bathwater?

Malcolm MacleodUniversity of Edinburgh

• Lots of drugs seem to work in animal models of stroke

• Some of these drugs have been around for ages, and are probably safe

• Rather than spend billions developing a novel drug, why not test something we already know works in animals?

Where we started …

What do we already know works in animals?

• Lots of individual promising studies• Little in the way of research

synthesis

So …• Review the field systematically• Pick a target and synthesise the

evidence

1026

1026 interventions in experimental stroke

1026603

1026 interventions in experimental stroke

Tested in focal ischaemia

1026883374

1026 interventions in experimental stroke

Effective in focal ischaemia

1026883550

97 18

1026 interventions in experimental stroke

Tested in clinical trial

1026883550

97 171 3

1026 interventions in experimental stroke

Effective in clinical trial

Outline

• Review the internal validity of animal studies

• Review the external validity of animal studies

• Explore the potential uses of this approach– Mechanisms research– Biomarker validation– Bibliometrics

Methodological approach

• Written review protocol – hypothesis, inclusion and exclusion criteria, analyses prespecified

• Standardised search strategy – 3 online databases, conference abstracts, dual screening

• Data extraction to bespoke database/analysis tool• Random effects weighted mean difference

stratified meta-analysis or meta-regression• Publication bias by Funnel plot, Egger regression,

Trim and Fill

There is nothing new under the sun …

“…you will meet with several observations and experiments which, though communicated for true by candid authors or undistrusted eye-witnesses, or perhaps recommended by your own experience, may, upon further trial, disappoint your expectation, either not at all succeeding, or at least varying much from what you expected”

Robert Boyle (1693)Concerning the Unsuccessfulness of Experiments

Focal cerebral ischaemia

• Emily Sena• David Howells• Tori O’Collins• Bart van der Worp• Philip Bath

Internal validity …

• the extent to which an experiment accurately describes what happened in that model system

• may be confounded bySelection Bias (Randomisation)Performance Bias (Allocation Concealment)Detection Bias (Blinded outcome assessment)Attrition bias (Reporting drop-outs/ ITT analysis)False positive report bias (Adequate sample sizes)

NXY-059 in animal stroke models

9 publications, 29 experiments, 408 animals 44% (35-53%) improvement in outcome

External validity

• Has the intervention been tested under a range of circumstances similar to those which might be encountered in clinical practice?

• Are the data you know about representative of all data? (what is the likelihood of publication bias)

Hypertension in studies of NXY-059 in experimental stroke

Hypertension:– 7% of animal studies– 77% of patients in the

(neutral) SAINT II study

Hypertension in studies of tPA in experimental stroke

Comorbidity

“Normal” BP

Effi

cacy

-2%25%

Infarct Volume:– 113 publications– 212 experiments– 3301 animals– Improved outcome by 24% (20-28)

Hypertension:– 9% of animal studies– Specifically exclusion criterion in (positive)

NINDS study

The importance of Time to Treatment

• Both tPA and tirilazad appear to work in animals

• tPA works in humans but tirilazad doesn’t

• Time to treatment: tPA:– Animals – median 90 minutes– Clinical trial – median 90 minutes

• Time to treatment: tirilazad– Animals – median 10 minutes– Clinical trial - >3 hrs for >75% of patients

Publication bias in experimental stroke

• Only 11/525 publications (2.2%) reported no significant treatment effects

• Trim and Fill suggested ~16% (214/1573) of experiments remain unpublished

• Best estimate of magnitude of problem – Observed efficacy 31.3% (29.7-32.8)– Adjusted efficacy 23.8% (22.2-25.5)

Publication bias

Randomisation Co-morbidity

bias

Reported efficacy

24%32% 18% 4%

There are multiple drivers of bias

Dopamine agonists in PD models

• Evelien Rooke• Hanna Vesterinen• Kieren Egan• Emily Sena

Systematic review

Internal validity in PD models

Blinded outcome assessment Composite quality

Experimental allergic encaphalomyelitis

• Hanna Vesterinen• Emily Sena

Publications identified in review

Outcomes reported

Efficacy – for what it’s worth

Internal validity

External validity

Demyelination

Axon loss

Neurobehaviour

31%

32%

14%

Relationship between endpoints

Transgenic models of AD

• Kieren Egan

Systematic Review

8495

409

8086 Excluded

381 Full publications

28 Abstracts

284 papers have histologicaldata extracted

69 Papers have data on the MWM extracted

Study Quality

RandomisationBlinded

Outcome AssessmentSample

Size calculation

Stroke 36% 29% 3%

MND 31% 20% <1%

AD 14% 22% 0%

PD 16% 15% <1%

EAE 8% 16% <1%

Glioma 14% 0% 0%

Effect Size Amyloid beta 40, All fractions (Standard Deviations)

-12 -8 -4 0 4 8 12 16 20 24 28

Eff

ect

Siz

e A

ß42

, A

ll fr

actio

ns (

Sta

ndar

d D

evia

tions

)

-8

-4

0

4

8

12

16

20

24

28 R2 = 0.74

Is there a relationship between efficacy for Aß 40 and Aß 42?

Amyloid beta 42 = 0.5321 x amyloid beta 40 + 0.3685

Efficacy is higher in younger animals

Time at Administration (Days)

1-10

1

102-

195

196-

335

>335

Eff

ect

Siz

e (S

tan

dar

d D

evia

tio

ns)

0.0

0.5

1.0

1.5

2.0

Summarising data from more fundamental research

• The interval validity of findings– Are they the product of bias?

• The validity of research summaries– Have the authors of “pivotal” reviews

considered all relevant data?

• The external validity of findings– Do these findings hold only in limited

highly controlled situations, or only one species, or are they generally applicable across biology?

Perplexing pathways

The added value of a systematic approach …

• Identifying, and explaining, heterogeneity

• Developing a systematic evidence base for understanding biological pathways

• Providing the components for mathematical models predicting the response to interventions

The role of Th17 cells in the development of EAE

Immunisation EAETh17

Th1

CD4+

- = +

- 6

= 3

+ 6 1

EAE

Th

1

- = +

- 11 1

= 1

+ 2 2

EAE

Th

17

Surrogate outcomes and biomarkers

Surrogate outcomes and biomarkers

Surrogate outcome

Functional outcome

Can animal models help?

• In animal studies– Does structural outcome predict functional

outcome?– Does this relationship hold across

interventions?

• Inclusion criteria: – Reports of the efficacy of a candidate stroke

drug in an animal model of focal cerebral ischaemia

– Structural and functional outcome reported from the same cohort of animals

– Simultaneous measurement of structural and functional outcome

Data

1047

257

16

71

131

71

21

25

110

13

110

12

24

18

26

9

Experiments reporting any

outcome

299TOTAL

58Stem Cells

11Other Thrombolytics

20Nicotinamide

54Thrombolytics

16NOS Inhibitors

8FK506

0NOS Donors

42Hypothermia

4Melatonin

51Growth Factors

3NXY-059

7Enriched Environment

16Tirilazad

8Minocycline

1IL1-RA

Experiments reporting structural and

functional outcomesDrug Group

Raw correlation

Raw correlation coefficient = 0.439

adjusted r2=0.301

∆ F

un

ctio

nal

ou

tco

me

∆ Structural outcome

Better

Better

Worse

Structural Outcome

Coefficient 95% CI

Constant (tPA) 23.9 17.1-30.7

Other lytics 19.3 2.93-35.7

FK506 19.9 1.62-38.1

Tirilazad 21.4 7.91-34.9

Nicotinamide 23.4 6.75-40.2

Hypothermia 26.1 15.9-36.4

NXY 059 29.1 0.43-57.8

Delay to assessment (days) -0.01 -0.02-0.00

% improvement (SO) = 23.9 + f(drug) -0.01*(days to assessment)

Adjusted r2 = 0.24

Functional Outcome

Coefficient 95% CI

Structural outcome (Infarct Volume) 0.47 0.37 to 0.56

Stem cells 10.0 1.9 to 18.1

Hypothermia 11.2 2.4 to 19.9

FK506 18.4 4.2 to 32.6

Minocycline 18.5 5.1 to 31.9

Nicotinamide 19.9 10.1 to 29.7

NOS Inhibitors 20.7 10.4 to 30.9

Tirilazad 22.9 8.1 to 37.9

Delay to assessment (days) 0.58 0.36 to 0.82

Delay to treatment (hours) -0.96 -1.44 to -0.48

% improvement (FO) = 0.47*SO + f(drug) + 0.58*(days to assessment) – 0.96*(hours to drug administration)

Adjusted r2 = 0.56

Summary

Structural Outcome Functional Outcome Independent effects on both

tPA Stem cells FK506

Other lytics Minocycline Tirilazad

NXY 059 NOS Inhibition Nicotinamide

Hypothermia

Delay to treatment (-ve)Delay to assessment

(-ve SO, +ve FO)

Lesion Structural Outcome Functional Outcome

Summary

Structural Outcome Functional Outcome Independent effects on both

tPA Stem cells FK506

Other lytics Minocycline Tirilazad

NXY 059 NOS Inhibition Nicotinamide

Hypothermia

Delay to treatment (-ve)Delay to assessment

(-ve SO, +ve FO)

Lesion Structural Outcome Functional Outcome

Summary

Structural Outcome Functional Outcome Independent effects on both

tPA Stem cells FK506

Other lytics Minocycline Tirilazad

NXY 059 NOS Inhibition Nicotinamide

Hypothermia

Delay to treatment (-ve)Delay to assessment

(-ve SO, +ve FO)

Lesion Structural Outcome Functional Outcome

Summary

Structural Outcome Functional Outcome Independent effects on both

tPA Stem cells FK506

Other lytics Minocycline Tirilazad

NXY 059 NOS Inhibition Nicotinamide

Hypothermia

Delay to treatment (-ve)Delay to assessment

(-ve SO, +ve FO)

Lesion Structural Outcome Functional Outcome

Key findings

• Structural outcome explains around 30% of the variation in functional outcome

• Crucially, this relationship is different for different interventions

• Surrogate outcome measures in Phase II are likely to be intervention specific rather than disease specific

• Nonetheless, this approach might help with rational selection of combination therapies

Summary

• The internal and external validity of animal experiments is limited

• Conclusions from such experiments are confounded by many biases

• There is no reason to believe that more “pathophysiological” experiments are any better

Does journal impact factor reflect validity of the presented work?

• 500 publications in focal cerebral ischaemia

• Modelling of Impact factor• IF = 3.7 + 2.4(Conflict of Interest

Statement) + 1.2(Allocation Concealment):

• adjusted r2 = 0.06

Future directions

• Qualitative systematic research• Quantitative systematic research

– Study quality issues– Evidence based pathways– Strategic research targeting

Are there any babies in the bathwater?

• MRC Trials Methodology Hub• Chief Scientist Office• MS Society• NHS R&D Methodology Program