Embed Size (px)
DESCRIPTION
There has been a significant decline of productivity in pharmaceutical R&D • Many different reasons have been discussed, but little attention has been paid to the role of poor reproducibility of biomedical research in general • There are often fundamental doubts about the credibility of research data leading to costly and often futile repetition of biomedical research studies • One of the root causes for the reproducibility problem is the lack of a common quality standard for biomedical research • There have been recent initiatives by the WHO and the FD&C Division of the American Society for Quality (ASQ) to address this issue • The pharmaceutical industry needs to embrace these initiatives in order to reverse the negative productivity trend
Citation preview
The Reproducibility Crises in Biomedical Research and its Impact on Pharmaceutical R&D Productivity
Ülo Palm, MD, PhD, Ülo Palm, MD, PhD,
Senior Vice President Clinical Operations & BiometricsSenior Vice President Clinical Operations & BiometricsForest Research InstituteForest Research Institute
February 11, 2013February 11, 2013
1
Moore’s Law:Moore’s Law:Transistor Count Transistor Count Doubling Every Doubling Every
Two YearsTwo Years
Source: Wikipedia 2
Eroom’s Law:Eroom’s Law:Number Of New Drug Number Of New Drug
Approvals Per Billion US Approvals Per Billion US Dollars Dollars Halved Every Nine Halved Every Nine
YearsYears
Source: Nature Reviews – Drug Discovery: Diagnosing the decline in pharmaceutical R&D efficiency Vol 11, March 2012, page 191 - 200
3
““War On Cancer” Going On For 40 War On Cancer” Going On For 40 YearsYears
• About $200 billion spent on cancer research in the US since 1971*
• More than 1.5 million papers published*But despite some progress in prevention and
treatment• Nearly 1 in 2 men and more than 1 in 3
women diagnosed with cancer during their lifetime**
• Almost 1.5 million cancer cases and 560 000 cancer deaths in the United States in 2009**
• Cancer now the second leading cause of death in the US**
Source: * Fortune Magazine March 22, 2004, ** JAMA, March 17, 2010 - Vol 303, 4
Some Causes Of Declining Some Causes Of Declining Productivity Of Biomedical Productivity Of Biomedical
ResearchResearch
• Biomedical R&D increasingly complex*
• Higher regulatory hurdles *• Pharmaceutical companies too big
to innovate
*Source: Nature Reviews – Drug Discovery: Diagnosing the decline in pharmaceutical R&D efficiency Vol 11, March 2012, page 191 - 200
5
Is There Another Major, Root Cause Of The Is There Another Major, Root Cause Of The Declining Productivity Of Biomedical Declining Productivity Of Biomedical Research?Research?
6
Lack Of Reproducibility In Lack Of Reproducibility In Biomedical Research – Literature Biomedical Research – Literature
And Press:And Press:
“Five of the seven largest published studies addressing cancer prognosis did not classify patients better than chance”
7
Lack Of Reproducibility In Lack Of Reproducibility In Biomedical Research – Literature Biomedical Research – Literature
And Press:And Press:
8
Lack Of Reproducibility In Lack Of Reproducibility In Biomedical Research – Literature Biomedical Research – Literature
And Press:And Press:
November 2010
9
Lack Of Reproducibility In Lack Of Reproducibility In Biomedical Research – Literature Biomedical Research – Literature
And Press:And Press:
“This is one of medicine's dirty secrets: Most results, including those that appear in top-flight peer-reviewed journals, can't be reproduced”
December 2011
10
Lack Of Reproducibility In Lack Of Reproducibility In Biomedical Research – Literature Biomedical Research – Literature
And Press:And Press:
“An empirical assessment of 18 published papers of microarray studies showed that independent analysts could perfectly reproduce the results of only two of the studies…”
December 2011
11
Lack Of Reproducibility In Lack Of Reproducibility In Biomedical Research – Literature Biomedical Research – Literature
And Press:And Press:2012
“Forensic bioinformatics is the art of using raw data and reported results to infer what must have happened to get from one to the other”
12
13
Lancet 2002Lancet 2002
• A new mass spectrometry method a highly sensitive and specific diagnostic tests for ovarian cancer
• NCI launched clinical Proteomics initiative• Companies formed to take assays based on this
method to the clinic• Turned out to be an experimental artifact due to
running of all of the controls before all of the cancers
Source:
Keith A . Baggerly and Kevin R . Coombes Handbook of Statistics in Clinical Oncology, Third EditionAntje Crowley and John Hoering Chapman and Hall/CRC 2012 Pages 605–618
Petricoin EF, Ardekani AM, Hitt BA, Levine PJ, Fusaro VA, Steinberg SM, Mills GB et al. Use of proteomic patterns in serum to identify ovarian cancer. Lancet 2002; 359(9306):572–577
14
Nature Medicine 2006Nature Medicine 2006
• Microarray-based signatures of drug sensitivity derived from cell lines predict patient response to specific chemotherapeutics
• Discover magazine designated this paper as one of the top 100 breakthroughs of 2006
• Large clinical trials started using this methodology• In 2009, independent analysis demonstrated that the
data were wrong due to mislabeling and indexing errors
Source: Keith A . Baggerly and Kevin R . Coombes Handbook of Statistics in Clinical Oncology, Third EditionAntje Crowley and John Hoering Chapman and Hall/CRC 2012 Pages 605–618
Potti A, Dressman HK, Bild A, Riedel RF, Chan G, Sayer R, Cragun J et al. Genomic signatures to guide the use of chemotherapeutics. Nat Med 2006; 12:1294–1300. Retracted January 7,
2011. 15
A few “What If Questions”A few “What If Questions”
16
Impact On Industry Impact On Industry ProductivityProductivity
Amgen*• Published literature described that inhibition of STK33
destroyed cancer cells• Amgen could not replicate the results despite massive
research effortsBayer*• Bayer halted in 2011 nearly two-thirds of its early drug target
projects because in-house experiments failed to match claims made in the literature.
Pfizer***• Pfizer wrote off $750 million in 2012 after results of a study
with Dimebon for Alzheimer, published originally in the journal Lancet*, could not be reproduced
Source: *WSJ, December 2, 2011; ** Fierce Biotech, January 17, 2012 17
What Are The Reasons For What Are The Reasons For The Lack Of Reproducibility The Lack Of Reproducibility
In Modern Biomedical In Modern Biomedical Research?Research?
18
Modern Science Is Based On Modern Science Is Based On Collective IntelligenceCollective Intelligence
• Collective intelligence requires: – a shared body of knowledge, methods,
and techniques, a shared praxis* – a shared and agreed upon quality standard
how to plan, conduct, and report scientific work
*Source: Reinventing Discovery – The new era of networked science by Michael Nielsen, 2012
19
No Well Defined Quality No Well Defined Quality Standard Exists In Biomedical Standard Exists In Biomedical
ResearchResearch
• “Genes were mislabeled due to an off-by-one indexing error”
• “The method didn’t actually work at all; it only appeared to work due to poor bookkeeping”
• “A disconnect between the numbers and the sample names rendered the predictions invalid”
• “Poor documentation allowed errors to go unnoticed until after things had proceeded to clinical trials “
Source: Keith A . Baggerly and Kevin R . Coombes, Handbook of Statistics in Clinical Oncology, Third Edition, Antje Crowley and John Hoering Chapman and Hall/CRC 2012 Pages 605–618 20
WHO: Development Of A Common Quality WHO: Development Of A Common Quality Standard For Biomedical Research A Pressing Standard For Biomedical Research A Pressing Global NeedGlobal Need
• “The world’s population is facing serious health challenges…
• .. there is increased demand for new drugs and new principles for treatment …
• ..it is essential that basic scientific (biomedical) research , …is conducted in a proper fashion… using processes that minimize waste of resources
• ….to reduce the need for costly repetition of work already performed”
Source: Handbook: Quality Practices in Basic Biomedical Research (QPBR), WHO, 2006 21
American Society For Quality (ASQ) : Best Quality American Society For Quality (ASQ) : Best Quality Practices For Biomedical Research In Drug Practices For Biomedical Research In Drug DevelopmentDevelopment
• The objective is to increase the overall productivity of biomedical research by promoting a common quality standard:
• “A common quality guideline for biomedical research in drug development will ensure the validity and credibility of scientific data from different research institutions and facilitate the mutual acceptance of research results”
22
Source: http://asq.org/quality-press/display-item/index.html?item=T919E
A Vision For The FutureA Vision For The Future
End-to-end electronic•Integration•Analysis•Modeling•Simulation
Standardized and validated external research
Standardized and validated internal research
Electronic Submission
23
Summary Of IssuesSummary Of Issues
• Declining productivity of Drug Development
• Poor reproducibility of Biomedical Research
• Lack of a common quality standard in Biomedical Research
24
SolutionSolution
Establish a common quality standard for biomedical research in drug development•WHO and ASQ published quality guidelines for Biomedical Research•Pharmaceutical Industry should promote these guidelines to improve R&D productivity
25
Thank You!
26
Backup slides
27
The Flow Of Research ActivitiesFrom Planning To Publishing
Source: Handbook: Quality Practices in Basic Biomedical Research (QPBR), WHO, 2006
““Best Quality Practices For Biomedical Research Best Quality Practices For Biomedical Research And Drug Development” And Drug Development” - Technical Report Content -- Technical Report Content -
• Management system• Organization• Project Management• Quality Management
System• Documentation• Document control /
Document approval and issue:
• Document changes
• Document storage• Technical Requirements• Test Equipment• Test methods / Method
Validation• Sampling and Chain of Custody• Materials• Legal and Ethical
Considerations• Vendor Selection and
Qualification
