How Do We Know What We Know?

How Do We Know What We Know?

Ivan Oransky, MDExecutive Editor, Reuters HealthCo-Founder, Retraction Watch

@ivanoransky

UTPABioethics: Creating and Challenging Knowledge in Health

Edinburg, Texas, April 2013

Retractions on the Rise

-The Wall Street Journal

How Often Are Studies Wrong?

“Winner Takes All” Incentives

Scientific American, August 2012

“Winner Takes All” Incentives

“The winner-take-all aspect of the priority rule has its drawbacks, however. It can encourage secrecy, sloppy practices, dishonesty and an excessive emphasis on surrogate measures of scientific quality, such as publication in high-impact journals.”

-- Fang and Casadevall, Scientific American

Anonymous Whistleblowers Step Up

http://www.labtimes.org

Blogs Get Aggressive

http://abnormalscienceblog.wordpress.com/

Blogs Get Aggressive

Blogs Get Aggressive

http://md-anderson-cc.blogspot.com

Blogs Get Aggressive

http://www.science-fraud.org/

Journals are Listening

Retraction Watch

Post-Publication Peer Review

Nature (22 Dec 2011) doi:10.1038/480449a

Post-Publication Peer Review

http://crossref.org/

Post-Publication Peer Review

Post-Publication Peer Review

http://precedings.nature.com/

Post-Publication Peer Review

Alt Metrics

http://total-impact.org/

Alt Metrics

http://sciencecard.org/

How Often Are Medical Studies Wrong?

Ioannidis JPA. PLoS Med 2005; 2(8): e124

How Often Are Medical Studies Wrong?

Does The Literature Reflect Reality?

Does The Literature Reflect Reality?

Publish a trial that will bring US$100,000 of profit or meet the end-of-year budget by firing an editor.

-- Former BMJ editor Richard Smith

Positive Publication Bias

Positive Publication Bias

“The overall frequency of positive supports has grown by over 22% between 1990 and 2007, with significant differences between disciplines and countries.”

“…the strongest increase in positive results was observed in disciplines—like Clinical Medicine, Pharmacology & Toxicology, Molecular Biology”

Fanelli, Scientometrics 2012.

Publish All Data?

Publish All Data?

www.AllTrials.net

FDA: “Black-or-White Approval”

FDA: “Black-or-White Approval”

“…abandon the current black-or-white approval process in favor of an incremental, conditional one. In such a process, drugs could be provisionally approved after promising early-stage data, with the FDA retaining the option to revoke that approval later on, should unexpected data come to light.”

“A ‘conditional approval’ approach would grant limited marketing authorization to new drugs after successful Phase II trials.”

http://www.manhattan-institute.org/

Confirmation Biases

“Facts do not accumulate on the blank slates of researchers' minds and data simply do not speak for themselves. Good science inevitably embodies a tension between the empiricism of concrete data and the rationalism of deeply held convictions.”

“…awareness of the systematic errors that can occur in evaluative processes may facilitate the self regulating forces of science and help produce reliable knowledge sooner rather than later.”

-- Kaptchuk, BMJ, 2003;326:1453–5

Believers vs. Snails

-- Kaptchuk, BMJ, 2003;326:1453–5

Contact/Acknowledgements

ivan-oransky@erols.com

http://retractionwatch.com

@ivanoransky

Thanks to Nancy Lapid, Reuters Health