“Evidenced-Based” Behavioral Medicine as Bad as

Bad Pharma

James C. Coyne, Ph.D.University Medical Center

Groningen (UMCG), Groningen, the Netherlands

jcoynester@gmail.com

Introduction to ICBM Symposium Groningen, August 2014

Are findings in behavioral medicine believable?

Nick Brown. Physical health outcomes in positive psychology: Weak evidence for strong claims.

James C. Coyne, Moritz Heene, Gozde Ozakinci.

Unsafe Dependence of Meta Analyses in Behavioral Medicine on Failsafe N.

Daniel Lakens. How likely is it an intervention study will replicate? The pain literature as an example.

John Ioannidis documented many positive findings in the biomedical literature do not replicate and many apparent “discoveries” turn out to be exaggerated or simply false.

Many apparent discoveries are created by a combination of confirmatory bias, flexible rules of design, data analysis and reporting and significance chasing.

“It can be proven that most claimed research findings are false”

Ioannidis, JPA. (2005). Why most published research findings are false. PLOS Medicine 2: 696-701.

Ioannidis, JPA. (2005). Contradicted and initially stronger effects in highly cited clinical research. JAMA 294: 218-228.

Young, NS., Ioannidis, JPA. et al. (2008). Why Current Publication Practices May Distort Science. PLOS Medicine 5: 1418-1422.

Ben Goldacre

Drugs are tested by the people who manufacture them, in poorly designed trials, on hopelessly small numbers of weird, unrepresentative patients, and analysed using techniques which are flawed by design, in such a way that they exaggerate the benefits of treatments. Unsurprisingly, these trials tend to produce results that favour the manufacturer. When trials throw up results that companies don't like, they are perfectly entitled to hide them from doctors and patients.

Efforts at reform

Preregistration of clinical trials make it more difficult to hide negative trials or alter analytic plans after results are known.

Reporting standards ensure more transparent and detailed article so results can be independently validated.

Reporting standards and pre-registration of plans for meta-analyses and systematic reviews.

Making data available for independent reanalysis.

Our modest contribution to reform

Roseman, M, Milette, K, Bero, LA, Coyne, JC, Lexchin, J., Turner, EH, & Thombs, BD. (2011). Reporting of conflicts of interest in meta-analyses of trials of pharmacological treatments. JAMA, 2011;305(10):1008-17.

Roseman, M, Turner, EH, Lexchin, J., Coyne, JC, Bero, LA, & Thombs, BD. (2012). Reporting of conflicts of interest from drug trials in Cochrane reviews: cross sectional study. BMJ, 2012; 345.

We documented with association of source of funding with claimed size of effects.

We proposed considering conflict of interest as a moderator, source of heterogeneity, and another risk of bias in Cochrane risk of bias assessment.

The Cochrane Collaboration agreed.

Cochrane Risk of Bias

Sequence generation.

Allocation concealment.

Blinding of participants, investigators, outcome assessors.

Incomplete outcome data.

Selective outcome reporting.

Other threats to validity.

In 1979 he wrote, "It is surely a great criticism of our profession that we have not organised a critical summary, by specialty or subspecialty, adapted periodically, of all relevant randomized controlled trials”

Prof Archibald Cochrane, CBE (1909 - 1988)

The Cochrane Collaboration isnamed in honor of ArchieCochrane, a British researcher.

Clinical psychology and behavioral medicine

CONSORT adopted later and less consistently than Biomedicine.*

Preregistration of trials is now encouraged, but enforcement is lax.

Conflict of interest less acknowledged, although investigator allegiance pervasive and a risk of bias.

*APS (Association of Psychological Science) journals not yet compliant.

Trickle down of reform

Changes forced upon Pharma slowly and inconsistently reach clinical psychology.

Larger data sets allow exploration of issues such as choice of control groups and investigator allegiance.

Behavioral Medicine adopts changes occurring in clinical psychology later and inconsistently.

Clinical psychology and behavioral medicine

RCT literature dominated by methodologically flawed, underpowered studies obtaining significant results at statistically improbable rate.

Weak control groups, unusual to have active control group.

Flexible rules of design and selective reporting of outcomes chosens after results are known.

Strong investigator allegiance effects.

Evidence-Based Medicine developed to weed out ineffective treatments.

Evidence-Based Behavioral Medicine developed to demonstrate that treatments worked and should be disseminated and reimbursed.

The Two Tribes of EBBM: Dodoists and Skeptical Dogs

Beware of Professional Organization Involvement as a Risk

of Bias

Guidelines for Guidelines Shaneyfelt T. In guidelines we cannot trust. Arch Intern

Med;172:1633-1634.

Institute of Medicine Committee on Standards for Developing Trustworthy Clinical Practice Guidelines (2011). Clinical practice guidelines we can trust. Washington, DC: National Academies Press.

Loblaw, D. A., Prestrud, A. A., Somerfield, M. R., et al. (2012). American Society of Clinical Oncology clinical practice guidelines: Formal systematic review–based consensus methodology. J Clin Oncol, 30(25), 3136-3140.

AHA Advisory on Screening for Depression Not Guideline-Congruent!

Ziegelstein RC, Brett D, Thombs BD, Coyne JC, de Jonge P. Routine Screening for Depression in Patients with Coronary Heart Disease: Never Mind. Journal of the

American Academy of Cardiology. 2009;54(10):886-90.

  Thombs, B. D., Jewett, L. R., Knafo, R., Coyne, J. C., &

Ziegelstein, R. C. (2009). Learning from history: a commentary on the American Heart Association Science Advisory on depression screening. American heart journal, 158(4), 503.

Beware “Official” meta-analyses(Critiqued here)

Coyne JC, Thombs BD, Hagedoorn M. Ain’t Necessarily So: Review and Critique of Recent Meta-Analyses of Behavioral Medicine Interventions in Health Psychology. Health Psychology. 2010;29(2):107-16.

  Coyne, J. C. (2012). Re: Meta-analysis of Efficacy of

Interventions Elevated Depressive Symptoms in Adults Diagnosed With Cancer. Journal of the National Cancer Institute, djs408.

SBM InitiativeSBM Initiative

Meta-analyses generated by professional organizations should receive special critical scrutiny because of tendency to gloss over limits of literature in order to promote the services of their membership.

“We are grateful to the Society of Behavioral Medicine (SBM) for selecting the authorship group. This article is one of three meta-analyses that have been undertaken under the aegis of the SBM Evidence-Based Behavioral Medicine Committee; the other two meta-analyses examine the effects of psychosocial interventions on depression and fatigue among patients with cancer.”

What to watch for in meta analyses commissioned by professional

organizations

Precommitment to conclusion that interventions work and are ready for dissemination and reimbursement.

Ignoring of preponderance of methodologically flawed, unpowered sources of bias.

Broad inclusion of diverse interventions into one category.

Ignoring statistical heterogeneity.

Weak assessment of risk of bias (Cochrane criteria are seldom used).

How This Symposium Came About

EHPS 2013: Improving the Credibility of Health Psychology Intervention Research: Problems and Solutions

Ozakinci, G., Whitehead, R., Sobota, A., Coyne, JC. Inaccurate Abstracts in Health Psychology: The Problem and an Easily Implementable Solution.

Coyne, JC. Too good to be true: Health psychology depends too much on positive underpowered studies.

Picking up collaborators on the Internet

Nick Brown

Moritz Heene

Daniel Lakens