Summarising what we already know – the pivotal role of systematic reviews
Malcolm Macleod
Is there an agreed and accepted definition of systematic reviews?
• Characteristics– A pre-defined question– A pre-defined search strategy– Pre-defined inclusion and exclusion criteria
• May include a meta-analysis, but doesn’t have to
[Are systematic reviews only about the application
of a methodology (bias-reducing principles), or do systematic reviews require a clearly defined topic and
specific question, as well as rigorous inclusion and
exclusion criteria?]
Why bother – what does it add?
1. Less biased research summary– “Positive” findings are more likely in
• Papers compared with Abstracts• High impact journals compared with low impact
journals• English Language verses non-English language
reports• Published versus unpublished findings
The File Drawer problem
0worse better
Characteristics of narrative reviews
• Research summaries based on what comes through your inbox in eTOC alerts do not adequately describe the whole literature
• Data supporting the opinion of the reviewer are more likely to be cited
Why bother – what does it add?
2. Has the work been done before?– Replicating the work of others is an
important part of science– Unknowing replication is a waste of time
(and your money)– Some things may already be known with
sufficient certainty that further experiments are not required (but e.g. positive control)
When and where are systematic reviews most
likely to be carried out and published from now on?
Are they common?
0
10
20
30
40
50
60
70
80
90
1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008
Pubmed search “systematic review” limited to animals conducted 22/1/09
When?
• In drug selection for clinical trial
• In decisions to proceed to clinical trial
• In preparation of grant and ethics applications
• In the planning of research
• In the refinement of the methodologies used in animal experiments
Where, and by whom?
• By people like me interested in developing new treatments
• By basic scientists summarising existing knowledge and planning new experiments
• By people studying the in-use characteristics of animal models of their disease
Publication
• Systematic reviews of clinical trials are widely published – sometimes in specialist journals
• Rigorous systematic reviews are cited about twice as frequently as narrative reviews
• The publishers of the Cochrane Library are exploring the possibility of publishing preclinical reviews
Do we have some examples of systematic reviews in basic
animal research?
• Different purposes …– To survey an area of research– To summarise the effectiveness of a specific
intervention– To examine the impact of potential sources of
bias
To survey an area of research
To summarise the effectiveness of a specific intervention
Hypothermia101 publications277 experiments
3353 animals
Bett
er
Wors
e
-100
0
100
200
To examine the impact of potential sources of bias
NXY 0599 publications29 experiments
408 animalsImproved outcome by
44% (35-53%)
Should the approach of systematic reviews be applied to all areas of
preclinical research?
• It probably could be, but methodologies may need some adaptation according to the specifics of the field
• Horizontal extension– from stroke to related diseases– Parkinsons, MS, Alzheimer’s, Neuro-oncology, Depression
• Vertical extension– Transgenic studies– Pathophysiological studies– In vitro studies
• Its wider application is likely to lead to substantial insights into the characteristics of the models used
Does the application of systematic reviews to
animal research require additional funding, and if
so where should this come from?
• Yes – but not much – Edinburgh group ~ £6k per publication
• This funding should come from those wishing to improve the usefulness of animal models of human disease
Are many reviews currently being published which are not systematic -- and if so
when, where and why?
Who is responsible for ensuring that reviews
are systematic, and how can we encourage this to
happen?
What questions remain about the validity of
research using animals as models of human disease,
and whether animal research has predictive value for human health?