Reproducibility in Cartography

AmyGriffin,RMIT,Australia&ArzuÇöltekin,UZH,Switzerland

ReproducibilityinCartography

Whoarewe(ifyoudidn’talreadyknow)

• Amy (withSidonie,whocouldn’tbeheretoday)isICA’sCogVis co-chair

à https://cogvis.icaci.org/activities.html

• Arzu isISPRS’Geovis,VirtualandAugmentedRealityworkinggroupchair(Sidonie alsoco-chairthere)

à https://geoviz.casa.ucl.ac.uk/

http://science.sciencemag.org/content/346/6210/679

Whydowedothis?Nature’ssurvey

https://w

ww.nature.com

/new

s/1-500-scientists-lift-th

e-lid-on-reprod

ucibility-1.1997

0

(2)B.Carey,Psychology’s fearsconfirmed:Recheckedstudiesdon’tholdup.NewYorkTimes(27August2015), p.A1.

http://science.sciencemag.org/content/351/6277/1037.2.full

Terminology

• Reproducibilityvs.replication?• Experimentalvs.otherformsofempiricalresearch?– Spatialanalysesà sampling(?),datamaintenance&annotation/coding+software/methods

– Experimentalresearchà alsoparticipants,descriptionofcontext/tasks,conditionsinwhichtheexperimentwasconductedfromtherecruitmenttotheactualroom,formsofcommunication(priming),…

Interdisciplinarity

http://science.sciencemag.org/content/349/6251/aac4716.fullhttps://www.nature.com/articles/483531ahttps://dl.acm.org/citation.cfm?id=2812803

Interdisciplinarity(II)

http://science.sciencemag.org/content/354/6317/1240.fullhttps://genomebiology.biomedcentral.com/articles/10.1186/s13059-015-0850-7https://biostats.bepress.com/bioconductor/paper2/https://agupubs.onlinelibrary.wiley.com/doi/full/10.1002/2015EA000136https://www.nsf.gov/sbe/AC_Materials/SBE_Robust_and_Reliable_Research_Report.pdfhttps://www.nature.com/articles/548485a

Ourcorner

…seemslikesofarthere’snothingonexperimentalstudies(inCarto/Vis/GIscience)though

https://link.springer.com/chapter/10.1007/978-3-319-57336-6_13https://onlinelibrary.wiley.com/doi/abs/10.1111/tgis.12195https://peerj.com/preprints/26561/http://journals.sagepub.com/doi/abs/10.1177/0309132515599625http://o2r.info

Interventions

• Systemic(grants,grantagencies...)

• Individual(initiatives,workshops…)

http://www.sciencemag.org/news/2017/07/replication-grants-will-allow-researchers-repeat-nine-influential-studies-still-raisehttp://science.sciencemag.org/content/354/6308/142https://doi.org/10.1186/s13059-017-1205-3http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1001747

Whatnext(today)?

• Wediscuss...• Wewritedownacollectivedocument(Googledocs)– Addopinions,directions,experiences,suggestions...

– Addsomeresources,literature,bestpracticesalsothere

https://fivethirtyeight.com/features/science-isnt-broken/

OutcomesofDiscussions,I

• Importanceofpoliticaleconomy&institutionsandscientificbehaviour

• Whatisestablishedcartographicfact?– Ofwhatthingsarewesure?– Whataretheassumptionsofthecartographiccannon?

• Twodatabaseeffortsthatmightbehelpful– Travis’sCartographicUserStudyDatabase(2000-2017)– Katarzyna Słomska’s studyofmapsusedinuserstudies(2000- onwards)

OutcomesofDiscussions,II

• Multilingualsearchtermsfordatabasetoextendittoincludenon-Englishstudies– 1973ICApublicationofmultilingualtermsasastartingpoint(carto-nyms?)

• Buildingout/extendingTravis’sdatabase– HostingonUUUorCogVis siteontheICAwebsite?– Topopulate/update:Hackathon eventevery2-4years

• thatwillupdatethedatabase• thenuseitasaresearchtooltodometaanalysis orotheranalysisofwhatweknow

• à examiningtheassumptionsofthecartographiccannon

Outcomes,III

• Exploringpotentialofreplicationstudiesand/ormultisitestudies(generalizabilityoffindings)?– MasarykUni hassomeexperiencewiththisonasmallscale• à largelyfoundsimilarfindings(buildssomeconfidencewedonothaveacrisisincartographyresearchJ)

• couldbetriedbyotherswithMScstudents

– Wewillexplorethepotentialformulti-siteefforts• Suggestionsfortopicswelcome!

Youropinions/involvement/commitment?• What’spossible,what’sdesirable• Whatdoweconsider‘established’experimentally?– Litreview?Find,code,…

• Canwereplicatethis?• Yourownwork?Stuffyouknowwell,andwillingtosupport(answerquestionsfromothers)– Multi-siteMScstudies?

• Meta-analyses?