Academ

icResearchIntegrity

HMSAR

INovem

ber2

018AcademicResearchIntegrity:

Practicalapplicationsinthelab

MaryC.Walsh,Ph.D.ChiefScientificInvestigatorOfficeforAcademicandResearchIntegrity(ARI)HarvardMedicalSchoolmary_walsh@hms.harvard.edu

DanielH.Wainstock,Ph.D.AssociateDirectorforResearchIntegrity

OfficeforAcademicandResearchIntegrity(ARI)HarvardMedicalSchool

daniel_wainstock@hms.harvard.edu

Academ

icResearchIntegrity

HMSAR

INovem

ber2

018

https://www.wordclouds.com/

Whatdoesthephrase“researchintegrity”meantous?

What are we doing everyday in the lab that helps us “do it right”?

22surveyresponses~86% have participated

in seminars and/or discussions concerning

research integrity

Academ

icResearchIntegrity

HMSAR

INovem

ber2

018

Whathelpsus“doitright”?

Our approach to research and data integrity

Labpractice

Institutionpractice

Stateoftheart

Academ

icResearchIntegrity

HMSAR

INovem

ber2

018

Whathelpsus“doitright”?ResearchandDataIntegrity

Think“local”v Whatdoesmylab(labPI)expectofme?v Systemformaintainingrawandanalyzeddata(localnetworks)v Systemfortrackingexact protocolsandstatisticalanalysis

(notebooks/ELNs)v Availablehardwareandsoftwarev Trackimportantbiologicalmetadata

Think“field”v Federalgrants:understandexpectationsofmaintainingandsharingdatav Publishingdata:whatarethenormsinthefield(s)?

Think“organizational”v Doesmyhomeinstitutionhaveexpectationsarounddatamaintenance?v Whataremyresourcestomeettheseexpectations?v HowaboutmylocalDepartmentorCenter?

Academ

icResearchIntegrity

HMSAR

INovem

ber2

018

FieldResourcesResearchandDataIntegrity

o RequirementsforOpenAccessandDataSharingØ Federalfundingmandates*sharing!(NIH,NSF)

* whileNIHPublicAccessisaglobalrequirementDataSharinghasapplicabilityguidelines

Academ

icResearchIntegrity

HMSAR

INovem

ber2

018

FieldResourcesResearchandDataIntegrity

o NIHRigorandReproducibilityInitiativeØ Consensusofthoughtleadersonscientificdata(FAQs onRigorandTransparency)

Academ

icResearchIntegrity

HMSAR

INovem

ber2

018

FieldResourcesResearchandDataIntegrity

o Reportingguidelinesthatmayberequired/applicable/helpfulTheEQUATOR network(EnhancingtheQUAlityandTransparencyOfhealthResearch)

Ø ARRIVE (AnimalResearch:ReportingofInVivoExperiments)Ø CONSORT (ConsolidatedStandardsofReportingTrials)Ø PRISMA (PreferredReportingItemsforSystematicReviewsandMeta-Analyses)

o Datamanagementplans(DMPs)Ø NISOPrimerØ DMPTool

Academ

icResearchIntegrity

HMSAR

INovem

ber2

018

FieldResourcesResearchandDataIntegrity

o WhatguidancedoAssociations/Societiesprovide?Ø SocietyofNeuroscience(JournalofNeuroscience)

§ ImprovingYourScience:Sample-sizePlanning,Pre-Registration,andReproducibleDataAnalysis

§ Datascienceresources [BigData:Whatyoushouldknow]§ Webinar(membershiprequired):DataScienceApproachesforNeuroscientists

Ø AAAS (Science)

o Whataboutjournals?Ø Nature(TransparencyandReproducibility)

Ø Neuron(CellPress/Elsevier*)§ STAR methods§ Mendeley*

ü Importer andDesktopü MendeleyData

*Elsevierhaspartiallysupportedotherwork independentofARIOutreachinitiatives

Academ

icResearchIntegrity

HMSAR

INovem

ber2

018

OrganizationalResourcesResearchandDataIntegrity

o DMWG (DataManagementWorkingGroup)Ø LinkstofantasticresourcessupportingtheResearchDatalifecycle

§ HMSDataStorageü OMERO

§ Repositories§ ELNs

o HarvardMedicalSchool ResearchComputingØ LinkstoITresourcesforHMS

§ StorageandTraining§ EnterpriseLicensedSoftware§ HMSSoftwareWiki

o CountwayLibraryØ Inadditiontoaccesstooutstandingelectronicjournalsandliterature(Hollis)

§ ResearchServices§ HarvardGuidelinesregardingpublicaccessandtools

o CatalystØ Great researchresources(forbasicandtranslationalwork)andtrainingopportunities

§ Biostatistics andBioinformatics consulting§ ProgramsandClasses available

o edXØ Greatfreeclassesfromexpertsonmanytopics

§ Principles,StatisticalandComputationalToolsforReproducibleScience

o LocalDepartmentResourcesØ e.g.,DepartmentofGeneticshasaComputerFacility intheNRB.Windows,Macintosh,andLinuxoperating

systemsonawidevarietyofhardware.Installation/maintenanceofdesktopcomputersandservers;automatedbackupsofservers;emailfortheGeneticsDepartment;assistancewithwebsitesandprogramming.

Academ

icResearchIntegrity

HMSAR

INovem

ber2

018

Biomedicalmetadata

LabDataStorageStructureandLabNotebook(NotebooksandStorage)o ELNso DirectoryStructureo Filenamingconventions

§ ReadMefiles§ Protocols(standardsand,importantly,adaptation!)–

Checkoutprotocols.io§ StatisticalAnalysis

Reagents*[andAnimals!]o antibodieso celllineso reagentbatching andlot#so animalwork!(ARRIVE,IACUC,localfacility,etc.)o biologicalvariables(e.g.,sexasabiologicalvariable

[SABV])

HardwareandSoftware(Tools)o Rawdataproduction!o understandandtrackspecso HMScorefacilitiesequipmentthatmayhave

networkeddatastorageonhardwareo labequipment(networkedorlocaldatastorage)o pairedsoftware(proprietaryorotherwise)o Code?CheckoutGitHub,Jupyter andCodeOcean

LocalResourcesandPracticeResearchandDataIntegrity

RESULTS/RAW DATA

ReproducibleData/literature

*NIHrequiresareagentvalidation statementinallgrantapplications.TheirResourceChartisahelpfulwaytothinkaboutoveralldataorganization.

Data Integrity

Academ

icResearchIntegrity

HMSAR

INovem

ber2

018

X

LocalResourcesandPractice

original/analyzed/publisheddata(a“synthetic” * examplepartI)

*“L&A”disclaimer:anyresemblancetoactualpersons,livingordead,oractualeventsispurelycoincidental.

“Figure2”Data.NomodificationstooriginalimagesreportedinMaterialsandMethodsorlegends.Whenaskedtomakerawdataavailableforeditorialreview,thecomposite.jpegfigurewasprovided

“Figure2”Data.Allmodificationstooriginalimages(global) reportedinMaterialsandMethodsandLegend.Whenaskedtomakerawdataavailableforeditorialreviewallrawimages(.oib)analyzedfiles(.tif)andthecomposite.jpegfigurewithandwithoutmodificationwereprovided

ü

ORIGINAL dataacquisition-staysthatway!

MCW9OMEROmw100318DymeckiDiscussion/SyntheticDataPartI

.OIBfile(s) “rotate.OIB”OlympusFluoViewFV1000

PUBLISHEDResultsandDiscussionsection(Welshetal2018)

ANALYZED datainfigurepreparationsoftware• trackallmodifications/alterations(.jpeg)• Labpresentationmw100918.Dymecki

discussion.ARI.pptx• DataIntegritymanuscriptinpreparation

COPY oforiginaldataintomw100318DymeckiDiscussion/SyntheticDataPartI/ANALYSIS• .tifseriescreated• namingconventions/identificationconsistent

originalè analyzedè publisheddata

Academ

icResearchIntegrity

HMSAR

INovem

ber2

018

PUBLISHEDMaterialsandMethods section(Welshetal2018)

Notebook6(MCWMBLKOwork:05.25.18- )pp.22-mw082118-“CytokineexpressioninisolatedMBLKOvsWTmicroglia”

LocalResourcesandPractice

protocols/finalprotocols(a“synthetic”*examplepartII)

SnapshotstakenbyMaryC.Walshfrom:DraughtofLivingDeath- HarryPotterandtheHalf-BloodPrinceYouTube·7,000+views·5/31/2013·byAceGoins

X

*“L&A”disclaimer:anyresemblancetoactualpersons,livingordead,oractualeventsispurelycoincidental.

09251835

10.11.18

MicrogliawereisolatedandhomogenizedaccordingtotheprotocolsfoundinMicrogliaMethodsandProtocols(Springer2013,Chapter9)MicroglialActivation:MeasurementofCytokinesbyFlowCytometryhttps://link-springer-com.ezp-prod1.hul.harvard.edu/content/pdf/10.1007%2F978-1-62703-520-0_9.pdf

MicrogliawereisolatedandhomogenizedaccordingtotheprotocolsfoundinMicrogliaMethodsandProtocols(Springer2013,Chapter9)MicroglialActivation:MeasurementofCytokinesbyFlowCytometryhttps://link-springer-com.ezp-prod1.hul.harvard.edu/content/pdf/10.1007%2F978-1-62703-520-0_9.pdf withthefollowingnotableadaptations:Optimalintermittentvortexingofinitialisolationofhomogenateswasdeterminedtobe2xvortex,@15minand@45minfornotlongerthan10seconds/applicationduringthe1hrhomogenateicingperiod.

ü

ORIGINALNotebook6:pp.23-24,mw082518MicrogliaMethodsandProtocols (Springer2013)MicroglialActivation:MeasurementofCytokinesbyFlowCytometry(Chapter9)

ANALYZEDNotebook6:pp.24-35,mw082518-mw092518AdaptationofMicrogliaMethodsandProtocols(e.g.,intermittent vortexing 1hr on ice= 2x vortex, @15 min and @45 min for not longer that 10seconds/application)

originalè analyzedè publisheddata

Academ

icResearchIntegrity

HMSAR

INovem

ber2

018

*“L&A”disclaimer:anyresemblancetoactualpersons,livingordead,oractualeventsispurelycoincidental.

LocalResourcesandPractice

“statistics”forthought(a“synthetic”*examplepartIII)

ORIGINALDymeckiDiscussion/SyntheticDataPartIII/mw082118/DataPlan

“CytokineexpressioninisolatedMBLKOvsWTmicroglia”• Variables• Samplesizes• Statisticaltests

CC0 CC0

ANALYSIS PartIDymeckiDiscussion/SyntheticDataPartIII/mw090518/ANALYSIS• realizesamplesizesmaybetoosmall(not

seeingsignificantdataonearlysamples)• addmoreanimals

ANALYSIS PartIIDymeckiDiscussion/SyntheticDataPartIII/mw092218/ANALYSIS• theoriginalstatisticaltestsproposeddonot

providesignificance(evenwithadditionalanimals)

• realizeappropriatealternativetestavailable,applicationresultsinsignificance

X

PUBLISHEDMaterialsandMethods section(Welshetal2018)

XFormicrogliacytokineexpressionexperiments“wecontinuouslyincreasedthenumberofanimalsuntilstatisticalsignificancewasreachedtosupportourconclusions.”Statisticaltestswereappliedtoassesssignificanceofdifferentialcytokineprofiles(p<0.05).

Formicrogliacytokineexpressionexperimentsweprovideallsamplesize,dataexclusionandreplicationinformationaspartofadetaileddatamanagementplanoutlinedattheinitiationofexperimentationandprovidedaspartofthesubmitted“Lifesciencesstudydesign”.Anydeviationsfromthisplanareclearlydetailedwithaccompanyingrationale.Additionalstatisticalassessmentsondatasetswereincludedtoprovideacompleteprofileofhypothesistestingandresultinganalysisrationale.

ü

originalè analyzedè publisheddata

Academ

icResearchIntegrity

HMSAR

INovem

ber2

018

https://www.theonion.com/national-science-foundation-science-hard-1819566466

Academ

icResearchIntegrity

HMSAR

INovem

ber2

018

"Somewhere, something incredible is waiting to be known."- (Carl Sagan) Sharon Begley

“Genius is 2% (1%) inspiration and 98% (99%) perspiration."-Thomas Edison

Academ

icResearchIntegrity

HMSAR

INovem

ber2

018

Thank You!v ResearchandDataIntegrityandtheresponsibleconductofresearchcan

havemanymeaningsv Thestepswetakeeveryday,bothlargeandsmall,canhaveanexponential

impactv Buildingcarefuldataandmetadatamanagementintoexperimentaldesign

canspeedupanalysis,andeasetrouble-shootingandreplicationv Whenplansaren’tenough,documentchangesv Revisitthediscussionssuchastheonewearehavingtoday(innovate)v Sharethewealth(disseminate)

MaryC.Walsh,Ph.D.ChiefScientificInvestigatorOfficeforAcademicandResearchIntegrity(ARI)HarvardMedicalSchoolmary_walsh@hms.harvard.edu

DanielH.Wainstock,Ph.D.AssociateDirectorforResearchIntegrity

OfficeforAcademicandResearchIntegrity(ARI)HarvardMedicalSchool

daniel_wainstock@hms.harvard.edu

Academ

icResearchIntegrity

HMSAR

INovem

ber2

018

AcknowledgementsHMSOfficeforAcademicandResearchIntegrity(ARI)TeamGretchenBrodnicki,J.D.JenniferRyan,J.D.MortimerLitt,M.D.BlakeTalbot,M.P.H.

Disclosures:MaidstoneConsultingGroup,LLC

ColumbiaUniversityOfficeofResearchComplianceandTrainingTheResearchandDataIntegrity(ReaDI)ProgramMichelleBenson,Ph.D.

HMSDataManagementWorkingGroupJessicaPierce

Academ

icResearchIntegrity

HMSAR

INovem

ber2

018

SURVEYFEEDBACK:Elements(orresources)specifictoresearchintegrity,dataintegrity,transparency,and/orexperimentalrigorandreproducibilitythatyouwantedtolearnmoreabout

o Electroniclabnotebookorganizationandrigor,reproducibility,andnotetaking?Ø Checkoutslide10Ø DMWGmatrixonELN (hyperlinkedtothesiteswhereyoucangetasenseoforganization)andguideson

BiomedicalMetadataØ ColumbiaReaDIprogram(seeslide24,andGoodLaboratoryNotebookPractices)Ø TakealookatOSF (OpenScienceFramework)– searchfor“electroniclaboratorynotebook”somegreat

examples

o Statisticaltests?Ø Checkoutslide10Ø UCLAidre:ChoosingtheCorrectStatisticalTestinSAS,STATA,SPSSandRandthereareotherexamplesout

there…Ø TalktoyourlabmentorandcolleaguesØ Askastatistician– CatalystBiostatisticalConsultingØ Reviewyoursoftwareguides(e.g.,Statahandbookisactuallygreatguideforappliedstatistics)

o GitHubandpostingcode?Ø Checkoutslide10Ø GitHub,Jupyter andCodeOcean

o Reproducibilityofcomputationalresearchandworkingwithlargedata?Ø DepartmentofBiomedicalInformaticsØ HarvardChanBioinformaticsCoreØ BigDatatoKnowledge(BD2K)

Ø BestPracticesforBiomedicalResearchDataManagementØ BigDataSciencewiththeBD2K-LINCS(LibraryofIntegratedNetwork-basedCellularSignature)Data

coordinationandIntegrationCenterØ Fieldspecificfieldexamples(e.g.,checkoutslide8forneuroscienceexample- Datascienceresources [Big

Data:Whatyoushouldknow])

Academ

icResearchIntegrity

HMSAR

INovem

ber2

018

SURVEYFEEDBACK:Elements(orresources)specifictoresearchintegrity,dataintegrity,transparency,and/orexperimentalrigorandreproducibilitythatyouwantedtolearnmoreabout

o Researchmisconductthatwouldbeanaccidentorthroughignorance,notanythingdeliberate?Ø HonesterrorisnotresearchmisconductØ Corrections(andretractions)areanappropriateelementoftypicalscientificdiscourseØ Ifyouwanttolearnmoreaboutthestandardsformisconductdecisions(42CFR§ 93.104)

o Havetherebeencasesofresearchmisconductthatwouldn'tbeobvioustothoseofuswhohavetakencoursesonresearchmisconduct?Ø DependsontheaudienceandthecaseØ CheckoutRetractionWatch togetasenseofexamplesofpossiblecasesandcommunitydiscoursearound

processandoutcome

o Statisticsformisconduct?Ø ORIWebsite– CaseSummaries

o Howcanwesupportorencourageourentirelabtoevaluate(andpossiblyupdate)datamanagementpractices?Ø Discussionwithyourmentor(s)regardinglocalpracticeØ Discussionsatlabmeetingsregardinglocalpractice(inviteARItocomeifhelpful!)

o Atwhatpointisreproducibilityconsideredsignificant?Ø Fromday1.Consider:Ifyouandonlyyoucandoathingonce,howdoesitmovetheknowledgebase

forward?Andkeepinmind:youareyourmostfrequentcollaborator!Sodoyourselfasolidandinvestthesweatequitysoonerratherthanlater.

Academ

icResearchIntegrity

HMSAR

INovem

ber2

018

Resources- slides

NIHPublicAccessPolicyhttps://publicaccess.nih.gov/NIHDataSharingPoliciesandImplementationGuidancehttps://grants.nih.gov/policy/sharing.htmhttps://grants.nih.gov/grants/policy/data_sharing/data_sharing_guidance.htm#appNIHDataSharingPoliciesTablehttps://www.nlm.nih.gov/NIHbmic/nih_data_sharing_policies.htmlNSFPublicAccessRepository(NSF-PAR)https://www.nsf.gov/news/special_reports/public_access/about_repository.jsp

TheEQUATORnetworkhttps://www.equator-network.org/DMPToolhttps://dmptool.org/NationalInformationStandardsOrganization.ResearchDataManagement:APrimerhttps://www.niso.org/publications/primer-research-data-management

NIHRigorandReproducibilityhttps://www.nih.gov/research-training/rigor-reproducibility/principles-guidelines-reporting-preclinical-researchNIHRigorandReproducibilityFAQshttps://grants.nih.gov/reproducibility/faqs.htm

FreeOnlineWordCloudGeneratorhttps://www.wordclouds.com/

Academ

icResearchIntegrity

HMSAR

INovem

ber2

018

Resources- slides

SocietyforNeurosciencehttp://www.sfn.org/SfNProfessionalDevelopment(Neuroscience2017):ImprovingYourScience:Sample-sizePlanning,Pre-Registration,andReproducibleDataAnalysishttps://www.sfn.org/sitecore/content/Home/OMP/Articles/Professional-Development/2018/Improving-Your-Science-Sample-Size-Planning-Pre-Registration-and-Reproducible-Data-AnalysisSfNScientificResearch:DataScienceResourceshttps://www.sfn.org/sitecore/content/Home/OMP/Articles/Scientific-Research/2016/Data-Science-ResourcesAAAShttps://www.aaas.org/Naturehttps://www.nature.com/authors/policies/reporting.pdfCellPressSTAR«METHODShttps://www.cell.com/star-authors-guideMendeleyReferenceToolshttps://www.mendeley.com/reference-management/web-importer#id_1https://www.mendeley.com/download-desktop/MendeleyDatahttps://data.mendeley.com/

TheDataManagementWorkingGrouphttps://datamanagement.hms.harvard.edu/hms-data-management-working-groupHMSResearchComputinghttps://rc.hms.harvard.edu/HarvardMedicalSchoolSoftwareResources(HMSwiki)https://wiki.med.harvard.edu/Software/CountwayLibraryhttps://www.countway.harvard.edu/HMSCatalysthttps://catalyst.harvard.edu/edXhttps://www.edx.org/DepartmentofGeneticshttp://genetics.hms.harvard.edu/

Academ

icResearchIntegrity

HMSAR

INovem

ber2

018

Resources- slides(con’t)DMWGELNMatrixhttps://docs.google.com/spreadsheets/d/1ar8fgwagOh30E31EAPL-Gorwn_g6XNf81g3VDQnQ_I8/edit#gid=0DMWG(BestPracticesMenu)https://datamanagement.hms.harvard.edu/UCLAidre:WhatStatisticalTest?https://stats.idre.ucla.edu/other/mult-pkg/whatstat/Protocols.iohttps://www.protocols.io/AntYbuddY https://www.antybuddy.com/InternationalCellLineAuthenticationCommitteehttp://iclac.org/databases/cross-contaminations/GlobalBiologicalStandardsInstitute https://www.gbsi.org/AnimalResearch:ReportingofInVivoExperiments(ARRIVE)https://www.nc3rs.org.uk/arrive-guidelinesHMSCoreFacilitieshttps://corefacilities.hms.harvard.edu/GitHubhttps://github.com/Jupyterhttp://jupyter.org/CodeOceanhttps://codeocean.com/AuthenticationofKeyBiologicaland/orChemicalResourceshttps://grants.nih.gov/grants/guide/notice-files/NOT-OD-17-068.html

TheBookDesignerhttps://www.thebookdesigner.com/2010/01/6-copyright-page-disclaimers-and-giving-credit/NaturePublishingImageIntegrityandStandards https://www.nature.com/authors/policies/image.htmlMicrogliaMethodsandProtocols(Springer2013)https://experiments-springernature-com.ezp-prod1.hul.harvard.edu/springer-protocols-closure

Academ

icResearchIntegrity

HMSAR

INovem

ber2

018

Resources- slides(con’t)

TheOnion(Science&Technology)2002NationalScienceFoundation:ScienceHardhttps://www.theonion.com/national-science-foundation-science-hard-18195664661977August15,Newsweek,Volume90,SeekingOtherWorlds(ProfileofCarlSagan),StartPage46,QuotePage53,Newsweek,Inc.,NewYork.(Verifiedonmicrofilm)https://quoteinvestigator.com/2013/03/18/incredible/1898April,TheLadies’HomeJournal,TheAnecdotalSideofEdison,Subsection:HisEstimateofGenius,StartPage7,QuotePage8,Column2,CurtisPublishingCompany,Philadelphia.(ProQuestAmericanPeriodicals)https://quoteinvestigator.com/2012/12/14/genius-ratio/#note-5018-3

Mousepichttp://4.bp.blogspot.com/-k9rfuvH0IFI/Tqq6LtQxfJI/AAAAAAAAALQ/6duHJ4osYVQ/s1600/Mus_Musculus_Unibe.jpegStatspichttp://www.aph.gov.au/~/media/05%20About%20Parliament/54%20Parliamentary%20Depts/544%20Parliamentary%20Library/Flagpost/Statistics/125806739.jpg?as=1&w=250

Academ

icResearchIntegrity

HMSAR

INovem

ber2

018

AdditionalresourcesColumbiaReaDI Programhttps://research.columbia.edu/ReaDI-Program• GoodLaboratoryNotebookPractices• BestPracticesforDataManagementwhenUsingInstrumentation• ReproducibilityResourcesandGuidelinesbyTopic• ResourcesbyDiscipline:Neuroscience

InternationalSocietyforPharmaceuticalEngineering:GMPResourceshttps://ispe.org/initiatives/regulatory-resources/gmp

RigorandReproducibilityinNIHApplications:ResourceCharthttps://grants.nih.gov/grants/RigorandReproducibilityChart508.pdfRigorandReproducibilityinNIH:Traininghttps://www.nih.gov/research-training/rigor-reproducibility/training

NIHResearchIntegrityandRCRhttps://grants.nih.gov/policy/research_integrity/index.htm

PenelopeAIhttps://www.penelope.ai/

OSF(OpenScienceFramework)https://osf.io/“TOP”[TransparencyandOpennessPromotion]Guidelineshttps://osf.io/ud578/?_ga=1.211230620.829898984.1435325845