Meta-Analyses: Considerations for Probiotics & Prebiotics ... · envisionable primary research study design ... –Queries (search terms & logic operators) –Methods for tracing

Forum: Meta-analyses

Meta-Analyses: Considerations for Probiotics & Prebiotics Studies

Daniel J. Tancredi, PhD Associate Professor in Residence

Department of Pediatrics UC Davis School of Medicine


Objectives

Provide an overview of systematic reviews & meta-analyses

Discuss with you key considerations & controversies for sponsors of probiotic/prebiotic studies


Key sources of material for talk

Julie Glanville, Sarah King, Francisco Guarner, Colin Hill, Mary Ellen Sanders. Nutrition Journal (2015) 14:16


Key sources of material for talk

Systematic Reviews in Health Care: Meta-Analysis in Context, 2nd Edition

Matthias Egger (Editor), George Davey-Smith (Editor), Douglas Altman (Editor)

March 2001, BMJ Books


Definitions:

“Systematic Review: A review of a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant research, and to collect and analyse data from the studies that are included in the review.

Statistical methods (meta-analysis) may or may not be used to analyse and summarise the results of the included studies.”

Source: http://community.cochrane.org/glossary/


Systematic Reviews: Scientific Virtues

Protocol-driven

Transparent (understandable)

Reproducible

Objective

Rigorous (aims to reduce biases and imprecision)


Narrative vs. Systematic Reviews Feature Narrative Systematic

Question Often broad Focused

Sources & search

Not usually specified, potentially

biased

Comprehensive sources, explicit search strategy

Selection ” Criterion-based,

uniformly applied

Appraisal Variable Rigorous

Synthesis Often qualitative Quantitative

Inferences Sometimes

evidence-based Usually evidence-

based

Source: Cook DJ et al (1998) “Synthesis of best evidence for clinical

decisions” Ch. 1 of “Systematic Reviews: Synthesis of Best Evidence

for Health Care Decisions” (ed. Mulrow C & Cook D). ACP, Philadelphia.


When to do a systematic review

Beginning and end of every RCT!? – Clarke et al. Reports of clinical trials should begin

and end with up-to-date systematic reviews of other relevant evidence: a status report. J R Soc Med. 2007 Apr; 100(4): 187–190.

CONSORT 2010 recommends (does not require) post-RCT SR:

– as an internal validity check (item 20)

– to situate results in context (item 22) • CONSORT 2010 Explanation and Elaboration:

updated guidelines for reporting parallel group randomised trials. BMJ 2010;340:c869

Meta-analysis of RCTs of probiotics for management of infant colic (crying time at 21 days post-intervention)

Valerie Sung et al. BMJ 2014;348:bmj.g2107 ©2014 by British Medical Journal Publishing Group

Previous

Example: Before/After Meta-analysis in RCT

Updated


When to do a systematic review (cont.)

According to EFSA “food/feed safety” guidance, a systematic review

may be useful when

– research question could be answered by an envisionable primary research study design

and practical and worthwhile when – SR on available data could improve precision of

estimated values for parameters of interest

Source: Application of SR methodology to food and feed safety

assessments…EFSA Guidance for those carrying out systematic reviews.

EFSA Journal 2010; 8(6):1637


…also “from” EFSA Q. What are the requirements for meaningful SR results for health benefits of foods or constituents?

A. SR’s (with or without meta-analyses) can be useful if:

1. They ask a clear and specific question

2. They are carried out rigorously – so as to minimise bias and random error

3. Reported well enough to allow assessment of the level of bias in the underlying evidence & in the review process

--Lee Hooper, U. East Anglia In presentation (http://www.efsa.europa.eu/en/events/documents/131120-p07.pdf ) at 2013 EFSA “Technical meeting on the reporting of human studies submitted for the scientific substantiation of health claims” http://www.efsa.europa.eu/en/supporting/doc/569e.pdf)


Don’t forget: RCTs are still primary

• Sponsors must strive for high quality RCTs

• Rationale: The assessment of individual RCTs is the key stage in weighing the “totality of evidence” in favor of a claim.

Sources: Appendix H (Study Quality) of EFSA's "Opinion of the Panel on dietetic products, nutrition and allergies (NDA) on a request from the Commission related to scientific and technical guidance for the preparation and presentation of the application for authorisation of a health claim" and

Table 13a of “Health Canada‘s Guidance Document for Preparing a Submission for Food Health Claims (March, 2009)”

Probiotics for the prevention of Clostridium difficile‐associated diarrhea in adults and children

Cochrane Database of Systematic Reviews 31 MAY 2013 DOI: 10.1002/14651858.CD006095.pub3 http://onlinelibrary.wiley.com/doi/10.1002/14651858.CD006095.pub3/full#CD006095-fig-0003

Another Example

http://onlinelibrary.wiley.com/doi/10.1002/14651858.CD006095.pub3/full#CD006095-fig-0003







How to do a SR: Seven Steps

1. Frame question (& preparing protocol)

2. Systematically locate relevant studies

3. Study selection

4. Data extraction (or assembly for IPDMA)

5. Assess quality of studies

6. Synthesize evidence (via meta-analyses)

7. Interpret findings Source: Glanville J, King S, Guarner F, Hill C, Sanders ME. A review of the systematic review process and its applicability for evaluating evidence for health claims on probiotic foods in the EU. Nutrition Journal (2015) 14:16


Register and publish protocol early!

Pre-specify research question, eligibility criteria & methods to be used for all steps in protocol

Register SR at PROSPERO http://www.crd.york.ac.uk/PROSPERO/

Publish protocol

http://www.crd.york.ac.uk/PROSPERO/





Frame question

Explicit specification of PICOS

– Population, including settings/locations

– Intervention(s), including vehicles/matrices

– Comparison

– Outcome(s), including timing & how measured

– Study types (designs & methodological quality)


Systematically locate relevant studies

Comprehensive (published & unpublished)

Explicit description of

– Sources (trial registries, MEDLINE, EMBASE, Cochrane Controlled Trials Register)

– Queries (search terms & logic operators)

– Methods for tracing citation linkages

– [handsearching key journals & personal communication with experts]


Study Selection

2+ observers check eligibility

Develop strategy to resolve disagreements

2-stages:

– Titles & Abstracts

– Full study

Keep track of inclusion/exclusion decisions for each stage (for PRISMA Flow Chart)

From: Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group (2009). Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med 6(6): e1000097. doi:10.1371/journal.pmed1000097


Data Extraction/Assembly

Develop, pilot test & refine a standard data extraction form

2+ observers

[Replicate published results]

Frank D’Amico, PhD Professor of Mathematics Duquesne University


Assess Quality of Studies

GRADE approach (quality is degree of confidence that effect size estimate is close to true parameter)

For RCTs, assess

random sequence generation/concealment,

Blinding,

Incomplete outcome data,

Selective reporting and other biases

[validation of intervention/comparator composition]

See http://www.gradeworkinggroup.org/index.htm for guidelines from the Grading of Recommendations Assessment, Development and Evaluation Working Group Also see: http://handbook.cochrane.org/chapter_8/table_8_5_a_the_cochrane_collaborations_tool_for_assessing.htm

http://www.gradeworkinggroup.org/index.htm



Probiotics for the prevention of Clostridium difficile‐associated diarrhea in adults and children

Cochrane Database of Systematic Reviews 31 MAY 2013 DOI: 10.1002/14651858.CD006095.pub3 http://onlinelibrary.wiley.com/doi/10.1002/14651858.CD006095.pub3/full#CD006095-fig-0002








Synthesize Evidence

Tabulate results (study-specific)

Examine forest plots

Assess heterogeneity

– Statistical distinction: sample v. population

Consider MA of all trials or subgroups

Perform sensitivity analysis

Examine funnel plots (publication bias)

Make available list of excluded studies

Sackett DL, Glasziou P, Chalmers I. Meta-analysis may reduce imprecision, but

it can’t reduce bias. Unpublished commentary commissioned by the New

England Journal of Medicine, 1997. (see SR in Health Care, p. xiv)


Funnel Plots Example

http://handbook.cochrane.org/chapter_10/figure_10

_4_a_hypothetical_funnel_plots.htm

Symmetrical plot in

the absence of

reporting bias

Asymmetrical plot

in the presence of

reporting bias


Meta-Analysis Meta-analysis may reduce imprecision, but it

can’t reduce biases

Heuristically, a pooled effect size is estimated as a weighted averages of sample effect sizes, resulting in a more precise estimate (with a smaller uncertainty interval)

Assess explainable/unexplainable heterogeneity in effect sizes, including subgroups

Sensitivity analyses to assess robustness

EFSA requires rationale & justification for MA Sackett DL, Glasziou P, Chalmers I. Meta-analysis may reduce imprecision, but

it can’t reduce bias. Unpublished commentary commissioned by the New

England Journal of Medicine, 1997. (see SR in Health Care, p. xiv)

Figure Example from Jack Cuzick’s. ‘Forest plots and the interpretation of subgroups.’ The Lancet 2005 365,Issue 9467

Copyright © 2005 Elsevier Ltd

http://www.elsevier.com/termsandconditions

Meta-analysis of RCTs of probiotics for management of infant colic

Outcome is crying time (min/day) at 21 days post-intervention)

Valerie Sung et al. BMJ 2014;348:bmj.g2107 ©2014 by British Medical Journal Publishing Group

Boxplots of Infant Crying Time in 3 RCTs for Managing Colic By Study Arm and Day of Follow-up

Placebo

L. reuteri DSM 17938


Present & Interpret Findings

Follow PRISMA statement (www.prisma-statement.org) for reporting

(Indeed, follow PRISMA after conceiving SR, to make sure you’re accounting for everything you’ll need!)

Explain departures from PRISMA (per EFSA)


Heterogeneity of causal effects…


Source: J R Soc Med. 2009 May 1; 102(5): 186–194

Copyright © 2009, The Royal Society of Medicine

Three classes of evidence for causal claims in Howick et al’s “The evolution of evidence hierarchies: what can Bradford Hill’s ‘guidelines for causation’ contribute?”

http://www.ncbi.nlm.nih.gov/pmc/about/copyright.html


RCTs can provide direct (probabilistic) evidence of causality

Randomization could yield sufficient evidence to outweigh plausible sources of confounding, even without knowing mechanism.

Intervention Black box Outcome

Control Black box Outcome


Howick et al’s Two Levels of Mechanistic Evidence

1. Internal substudies of causal links in Black box

2. Purported Mechanism of Action demonstrated in external studies

Source: J R Soc Med. 2009 May 1; 102(5): 186

– 194

© 2009, The Royal Society of Medicine

Challenge to Combinability: Black Boxes Depend on Study PICOS

Intervention Black

box(A) Outcome

Control Black

box(A) Outcome

Study A

Intervention Black

box(B) Outcome

Control Black

box(B) Outcome

Study B

The blank “wheel” of the pragmatic–explanatory continuum indicator summary (PRECIS) tool. “E” represents the

“explanatory” end of the pragmatic–explanatory continuum.

Kevin E. Thorpe et al. CMAJ 2009;180:E47-E57

©2009 by Canadian Medical Association

Figure A: PRECIS summary of four unrelated RCTs, for illustration

Kevin E. Thorpe et al. CMAJ

2009;180:E47-E57

©2009 by Canadian Medical Association

NB: Potential

use for modified

PRECIS tools

for SR has

begun (see

PMID:

21474282).

Possible distribution of mechanisms among probiotics

Hill, C. et al. (2014) The International Scientific Association for Probiotics and Prebiotics consensus

statement on the scope and appropriate use of the term probiotic

Nat. Rev. Gastroenterol. Hepatol. doi:10.1038/nrgastro.2014.66


Some rationales for combining strains

Rationale: Common structural or secreted product

Example: All L. bulgaricus & S. thermophilus

Rationale: Common MoA known to be n.&s. for effect (e.g., production of a specific bacteriocin or range of bacteriocins known to be active against a specific pathogen, or induction of immune mechanisms needed for the effect)

Example: L. salivarius strains A & B

Other rationales: common identity, taxonomy, physiological effect (in humans)

Glanville et al. Nutrition Journal (2015) 14:16


Remember

High quality RCTs are key

Well-done SR’s can provide

– more precise estimates of causal effects (narrower uncertainty intervals) and

– assess (some) heterogeneity of treatment effects

Pooling different studies requires scientific basis for the “common black box” assumption


Questions and Comments

[email protected]

Documents

Meta-Analyses: Considerations for Probiotics & Prebiotics ... · envisionable primary research study design ... –Queries (search terms & logic operators) –Methods for tracing