New Walters D3R 2018 - D3R | Welcome... · 2018. 3. 5. · Pat Walters – D3R Workshop February 23, 2018. I’ve Done This a Few Times ... Final Comparisons. 5 The Literature Makes

Howcanwegetbetteratthis?

PatWalters– D3RWorkshopFebruary23,2018

I’veDoneThisaFewTimes

2012 2013 2015 2016 2017 20182014

3

HowISpendMyTimeOnChallenges

Confidential|©2017RelayTherapeutics

Dealingwithpoorlyformattedsubmissions

Validatingevaluations MakingSlides

4

TheEvaluationProcess


PatEvaluate

ConnorandZiedEvaluate

FinalComparisons

5

TheLiteratureMakesItLookLikeActivityPredictionisaSolvedProblem


0.82 0.80

0.66 0.65

Pearsonr

6

ScoringPerformanceFromGC2andGC3


Weneedtoagreeon• Whatconstitutesareasonabledataset• Howdatashouldbereported• Evaluationmetrics• Statisticsforcomparison• Whatconstitutesanullmodel• Formatofsupportingmaterial• Criteriaforreproducibility

7

GuidelinesForReviewing”ScoringFunction”Papers



8



Whenevaluatingaregressionmodel,thedatasetshouldhaveadynamicrangesimilartothoseobservedindrugdiscoveryprojects(typically4-6logs)

9

DatasetsShouldSpanaReasonableDynamicRange


Thisdataset(PDBindv.2016coreset)spans10logsanddoesn’tprovideanappropriaterepresentationofcorrelation

10

CorrelationsCanChangeDramaticallyWithDynamicRange


R2=0.22MAE=0.69

R2=0.76MAE=0.55

Thisisthesamedataset.Ontheleftweconsidertheentireset,whichhasanunrealisticallylarge(~10log)dynamicrange.Ontherightweconsideramorerealisticsubsetwitha3logdynamicrange.Notethechangeincorrelation.

11

GC3CatSDatasetSpansaRealisticDynamicRange



12



13

Don’tCramMultipleDatasetsontotheSamePlot


http://pubs.acs.org/doi/abs/10.1021/acs.jpcb.7b07224 http://pubs.acs.org/doi/abs/10.1021/ja512751q

14

EvenMyFriendsAreGuilty


MillandNeysa(Yesterday)

15

Trellisingprovidesamuchmoreeffectivemeansofcomparingdatasets



16



ReportPearson,SpearmanandKendallcorrelationsFavorR2 overRwhenreportingaPearsoncorrelationcoefficientReportMAEand/orRMSE

17

Alwaysreportcorrelationsappropriately


Ihavenoideawhatthismeans

http://pubs.acs.org/doi/abs/10.1021/acs.jpcb.7b07224

18

MaximumAchievableCorrelation


StartwithexperimentaldataAddGaussianerror

§ Mean=0.0§ Standarddeviation=0.3log

CalculationcorrelationRepeat1000times

Brown,ScottP.,StevenW.Muchmore,andPhilipJ.Hajduk."Healthyskepticism:assessingrealisticmodelperformance.”DrugDiscoveryToday14.7(2009):420-427.

19

MaximumAchievableCorrelation- HPS90D3R1


https://github.com/PatWalters/metk

OpenSourceEvaluationCode(MoretoCome)


21



22

EnsureThatDifferencesinCorrelationAreSignificant


Inparticular,bothMM-PB/SAandMM-GB/SAproducedbetterresultsbyusingarepresentativestructure(R)0.72-0.79)ratherthanaveragingovertheconformationalensembleofeachgivencomplex(R)0.61-0.74

23Confidential|©2017RelayTherapeutics

M1_dynamic M1_static M2_static M3_dynamic M3_static M4_dynamic M4_static

Table L2

abs(

Pea

rson

r)

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Aliteraturecomparisonof7methodsforscoringprotein-ligandinteractions

24

Rememberthatcorrelationshaveconfidenceintervalsandreporttheseintervals


25

It’sAlltheSame!


M1_dynamic M1_static M2_static M3_dynamic M3_static M4_dynamic M4_static

Table L2

ab

s(P

ea

rso

n r

)

0.0

0.2

0.4

0.6

0.8


26



MolecularweightandcalculatedLogParepoornullmodels

GenerateRDKitfingerprintsforligandsTrainonPDBbindrefinedset(n=4057)TestonPDBbindcoreset(n=290)Wallclocktime<5min

28

SimpleQSARasaNullModel


29

WhatConstitutesanAppropriateNullModel


MolecularWeight XLogP SimpleQSAR

30

ANullModelforRMSE


1.SampleNobservedvalues2.CalculateRMS3.Repeat1and2*1000

31

NullModelforGC1HSP90FreeEnergyChallenge


RMSE(kcal/m

ol)

32

ComparingRMSvsNullforGC1HSP90Challenge


Dashedlineindicatesthenullmodel


33



Alwaysprovideamachinereadabletable(e.g.csv)ofpredictedandexperimentalvaluesAtableinapaperisnotsufficient,itisoftenverydifficulttoextracttablesfrompdffilesChemicalstructuresshouldbeincludedasSDFor,whereappropriate,SMILEStofacilitatecomparisonwithothermethodsNeedtoenablereaderstoevaluatecorrelationsanderrors

34

Includeappropriatesupportinginformation



35



36

CanIReproduceYourMethod?


Code!!!AthoroughdescriptionofyourmethodAwebimplementationNoneoftheabove

37

WhatConstitutesReproducibility?



38



39

HowCanYouHelp?


40

DockingChallengesHaveBecomeMoreChallenging


Arewespendingenoughtimeunderstandcompoundsthatdockedpoorly?• Insufficientconformationalsampling• Insufficientposesampling• Inadequatescoring• LigandposeswithlimiteddensityIseveryonemissingthesamecompounds?Cangroupsworktogethertoimprovetheirmethods?

41

QuestionsonDockingChallenges


D3RParticipantsCSARParticipantsTDTParticipantsSAMPLParticipants

RommieAmaroMikeGilson

MillLambertNeysaNevins

ConnorParksZiedGaieb

ShuaiLiu

42

Acknowledgements


https://github.com/PatWalters/metk

OpenSourceEvaluationCode(MoretoCome)

BACKUP

44Confidential|©2017RelayTherapeutics

45

LooksLikeActivityPredictionisaSolvedProblem


0.82 0.80

0.66 0.65

Pearsonr

46




47




48

Evaluatemaximumpossiblecorrelationforadatasetgivenexperimentalerror

Confidential|©2017RelayTherapeutics https://www.sciencedirect.com/science/article/pii/S1359644609000403

StartwithexperimentaldataAddGaussianerror• Mean=0.0• Standarddeviation=0.3logCalculationcorrelationRepeat1000times

49

MaximumAchievableCorrelation


Documents

New Walters D3R 2018 - D3R | Welcome... · 2018. 3. 5. · Pat Walters – D3R Workshop February 23, 2018. I’ve Done This a Few Times ... Final Comparisons. 5 The Literature Makes