Summarising the evidence from animal models of neurological disease: Publication bias, poor internal validity, and (perhaps) some efficacy

Malcolm MacleodUniversity of Edinburgh

• Lots of drugs seem to work in animal models of stroke

• Some of these drugs have been around for ages, and are probably safe

• Rather than spend billions developing a novel drug, why not test something we already know works in animals?

Where we started …

Introduction Translation –Internal Validity

Translation –External Validity

Mechanisms Research

Summary

What do we already know works in animals?

• Lots of individual promising studies

• Little in the way of research synthesis

So …• Review the field systematically• Pick a target and synthesise the evidence

Introduction Translation –Internal Validity

Translation –External Validity

Mechanisms Research

Summary

1026

1026 interventions in experimental stroke

Introduction Translation –Internal Validity

Translation –External Validity

Mechanisms Research

Summary

1026603

1026 interventions in experimental stroke

Tested in focal ischaemia

Introduction Translation –Internal Validity

Translation –External Validity

Mechanisms Research

Summary

1026883374

1026 interventions in experimental stroke

Effective in focal ischaemia

Introduction Translation –Internal Validity

Translation –External Validity

Mechanisms Research

Summary

1026883550

97 18

1026 interventions in experimental stroke

Tested in clinical trial

Introduction Translation –Internal Validity

Translation –External Validity

Mechanisms Research

Summary

1026883550

97 171 3

1026 interventions in experimental stroke

Effective in clinical trial

Introduction Translation –Internal Validity

Translation –External Validity

Mechanisms Research

Summary

Outline

• Review the internal validity of animal studies

• Review the external validity of animal studies

• Explore the potential uses of this approach in mechanisms research

Introduction Translation –Internal Validity

Translation –External Validity

Mechanisms Research

Summary

Methodological approach

• Written review protocol – hypothesis, inclusion and exclusion criteria, analyses prespecified

• Standardised search strategy – 3 online databases, conference abstracts, dual screening

• Data extraction to bespoke database/analysis tool

• Random effects weighted mean difference stratified meta-analysis or meta-regression

• Publication bias by Funnel plot, Egger regression, Trim and Fill

Introduction Translation –Internal Validity

Translation –External Validity

Mechanisms Research

Summary

Introduction Translation –Internal Validity

Translation –External Validity

Mechanisms Research

Summary

There is nothing new under the sun …“…you will meet with several observations and experiments

which, though communicated for true by candid authors or undistrusted eye-witnesses, or perhaps recommended by your own experience, may, upon further trial, disappoint your expectation, either not at all succeeding, or at least varying much from what you expected”

Robert Boyle (1693)Concerning the Unsuccessfulness of Experiments

Internal validity …

• the extent to which an experiment accurately describes what happened in that model system

• may be confounded bySelection Bias (Randomisation)Performance Bias (Allocation Concealment)Detection Bias (Blinded outcome assessment)Attrition bias (Reporting drop-outs/ ITT

analysis)False positive report bias (Adequate sample sizes)

Introduction Translation –Internal Validity

Translation –External Validity

Mechanisms Research

Summary

NXY-059 in animal stroke models9 publications, 29 experiments, 408 animals 44% (35-53%) improvement in outcome

Introduction Translation –Internal Validity

Translation –External Validity

Mechanisms Research

Summary

A widespread problem …

Dopamine Agonists in 6-OH model of Parkinson’s Disease

Various interventions in 435 experiments (7170 animals) in EAE

Introduction Translation –Internal Validity

Translation –External Validity

Mechanisms Research

Summary

Internal validity and Journal Impact Factors• Regression model offered:

– Effect size– Variance– Randomisation– Allocation concealment– Blinded assessment of outcome– Sample size calculation– Statement of possible conflict of interest

• Univariate:– Allocation concealment– Blinded assessment of outcome– Statement of possible conflict of interest

• Multivariate: IF = 3.7 + 2.4(Conflict of Interest Statement) + 1.2(Allocation Concealment)

Introduction Translation –Internal Validity

Translation –External Validity

Mechanisms Research

Summary

External validity

• Has the intervention been tested under a range of circumstances similar to those which might be encountered in clinical practice?

• Are the data you know about representative of all data? (what is the likelihood of publication bias)

Introduction Translation –Internal Validity

Translation –External Validity

Mechanisms Research

Summary

Hypertension in studies of NXY-059 in experimental stroke

Hypertension:– 7% of animal studies– 77% of patients in the

(neutral) SAINT II study

Introduction Translation –Internal Validity

Translation –External Validity

Mechanisms Research

Summary

Hypertension in studies of tPA in experimental stroke

Comorbidity

“Normal” BP

Effi

cacy

-2%25%

Infarct Volume:– 113 publications– 212 experiments– 3301 animals– Improved outcome by 24% (20-28)

Hypertension:– 9% of animal studies– Specifically exclusion criterion in (positive)

NINDS study

Introduction Translation –Internal Validity

Translation –External Validity

Mechanisms Research

Summary

The importance of Time to Treatment• Both tPA and tirilazad appear to work in animals

• tPA works in humans but tirilazad doesn’t

• Time to treatment: tPA:– Animals – median 90 minutes– Clinical trial – median 90 minutes

• Time to treatment: tirilazad– Animals – median 10 minutes– Clinical trial - >3 hrs for >75% of patients

Introduction Translation –Internal Validity

Translation –External Validity

Mechanisms Research

Summary

Introduction Translation –Internal Validity

Translation –External Validity

Mechanisms Research

Summary

Publication bias in experimental stroke

• Only 11/525 publications (2.2%) reported no significant treatment effects

• Trim and Fill suggested ~16% (214/1573) of experiments remain unpublished

• Best estimate of magnitude of problem – Observed efficacy31.3% (29.7-32.8)– Adjusted efficacy 23.8% (22.2-25.5)

Introduction Translation –Internal Validity

Translation –External Validity

Mechanisms Research

Summary

Publication bias

Randomisation Co-morbidity

bias

Reported efficacy

24%32% 18% 4%

Introduction Translation –Internal Validity

Translation –External Validity

Mechanisms Research

Summary

There are multiple drivers of bias

Introduction Translation –Internal Validity

Translation –External Validity

Mechanisms Research

Summary

Developing a lexicon for mechanisms research

• The interval validity of findings– Are they the product of bias?

• The validity of research summaries– Have the authors of “pivotal” reviews considered all relevant data?

• The external validity of findings– Do these findings hold only in limited highly controlled situations, or only one species, or are they generally applicable across biology?

Introduction Translation –Internal Validity

Translation –External Validity

Mechanisms Research

Summary

The added value of a systematic approach …

• Identifying, and explaining, heterogeneity

• Developing a systematic evidence base for understanding biological pathways

• Providing the components for mathematical models predicting the response to interventions

Introduction Translation –Internal Validity

Translation –External Validity

Mechanisms Research

Summary

The role of Th17 cells in the development of EAE

Introduction Translation –Internal Validity

Translation –External Validity

Mechanisms Research

Summary

Immunisation EAETh17

Th1

CD4+

- = +

- 6

= 3

+ 6 1

EAE

Th

1

- = +

- 11 1

= 1

+ 2 2

EAE

Th

17

Introduction Translation –Internal Validity

Translation –External Validity

Mechanisms Research

Summary

Introduction Translation –Internal Validity

Translation –External Validity

Mechanisms Research

Summary

Summary

• The internal and external validity of animal experiments is limited

• Conclusions from such experiments are confounded by many biases

• There is no reason to believe that more “pathophysiological” experiments are any better

Introduction Translation –Internal Validity

Translation –External Validity

Mechanisms Research

Summary

Future directions

• Qualitative systematic research• Quantitative systematic research– Study quality issues– Evidence based pathways– Strategic research targeting

Introduction Translation –Internal Validity

Translation –External Validity

Mechanisms Research

Summary