16
1 ANALYSIS OF CYBER METRICS WORKSHOP London, 23 May 2018 INTRODUCTION The Research Institute in Science of Cyber Security (RISCS) is the UK’s first academic Research Institute to focus on understanding the overall security of organisations, including their constituent technology, people and processes. RISCS is focused on giving organisations more evidence, to allow them to make better decisions, leading to the development of cybersecurity as a science. It collects evidence about what degree of risk mitigation can be achieved through a particular method – not just the costs of its introduction, but ongoing costs such as the impact on productivity – so that the total cost of ownership can be balanced against the risk mitigation that has been achieved. RISCS main goal is to move security from common, established practice to an evidence base in much the same way as happened in sectors like medicine. On May 23, 2018, RISCS held a workshop in London that looked at the utility of cyber security metrics. The purpose of the workshop was to develop a deeper understanding of the ways in which cyber security metrics are used in decision-making more generally, and also to raise questions about how data is best presented to the board and the policy community more specifically. We wanted to explore the potential for metrics to help but we also want to take a critical approach to the underlying values that can shape metrics – and consequently, decisions. For further information about this report or about other RISCS initiatives, please contact Emma Bowman at [email protected] .

Metrics workshop analysis · 2018-07-05 · metrics. The purpose of the workshop was to develop a deeper understanding of the ways in which cyber security metrics are used in decision-making

  • Upload
    others

  • View
    1

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Metrics workshop analysis · 2018-07-05 · metrics. The purpose of the workshop was to develop a deeper understanding of the ways in which cyber security metrics are used in decision-making

1

ANALYSISOFCYBERMETRICSWORKSHOPLondon,23May2018

INTRODUCTIONTheResearchInstituteinScienceofCyberSecurity(RISCS)istheUK’sfirstacademicResearchInstitutetofocusonunderstandingtheoverallsecurityoforganisations,includingtheirconstituenttechnology,peopleandprocesses.RISCSisfocusedongivingorganisationsmoreevidence,toallowthemtomakebetterdecisions,leadingtothedevelopmentofcybersecurityasascience.Itcollectsevidenceaboutwhatdegreeofriskmitigationcanbeachievedthroughaparticularmethod–notjustthecostsofitsintroduction,butongoingcostssuchastheimpactonproductivity–sothatthetotalcostofownershipcanbebalancedagainsttheriskmitigationthathasbeenachieved.RISCSmaingoalistomovesecurityfromcommon,establishedpracticetoanevidencebaseinmuchthesamewayashappenedinsectorslikemedicine.

OnMay23,2018,RISCSheldaworkshopinLondonthatlookedattheutilityofcybersecuritymetrics.Thepurposeoftheworkshopwastodevelopadeeperunderstandingofthewaysinwhichcybersecuritymetricsareusedindecision-makingmoregenerally,andalsotoraisequestionsabouthowdataisbestpresentedtotheboardandthepolicycommunitymorespecifically.Wewantedtoexplorethepotentialformetricstohelpbutwealsowanttotakeacriticalapproachtotheunderlyingvaluesthatcanshapemetrics–andconsequently,decisions.

ForfurtherinformationaboutthisreportoraboutotherRISCSinitiatives,[email protected].

Page 2: Metrics workshop analysis · 2018-07-05 · metrics. The purpose of the workshop was to develop a deeper understanding of the ways in which cyber security metrics are used in decision-making

2

METHODOLOGY:Toinvestigatetheutilityofcybersecuritymetricsinthedecisionmakingprocessofindustryandthepolicycommunity,wegatheredagroupof70peoplefromacademia,industry,thepolicycommunityandthetechnicalcommunity.Weaskedthesepeopletoself-identifythemselvesas‘providers’or‘consumers’ofmetricsandtoindividuallyorcollaborativelyrecordtheirresponsestofourquestionsthatweaskedoftheirgroup.Forthecybersecuritymetricsprovidergroup,weaskedthemtopopulatethefollowingtable:

Weaskedthosepeoplewhoidentifiedastheconsumersofcybersecuritymetrics(decisionmakersaboutinvestment,policyetc)torespondtothequestionsonaseparatetable:

AcompletetranscriptoftherecordedresponsesisincludedatAppendixA.Onthefollowingpages,wepresentouranalysisofthefindings.

Page 3: Metrics workshop analysis · 2018-07-05 · metrics. The purpose of the workshop was to develop a deeper understanding of the ways in which cyber security metrics are used in decision-making

3

SUMMARYOFFINDINGSProvidingtailoredcybermetricsisanopportunitytoengagewithleadersandshapetheirperceptionsofinformationrisk.Theoutputsofthisworkshopsuggestthatsuccessdependsonprovidersdeliveringmaterialthat:

• genuinelyreducesuncertainty;• addressesspecificquestions;and,• usesthelanguageofbusiness.

Thetablebelowbringsoutthetrendsfromourworkshopintermsofwhatisandisnotrequiredbydecision-makers.Belowaresomeofthemostinterestingconclusionsthatwehavedrawnfromthedata.

➡ Wesawthatsomerequirementsmaybemetwithreticencefrommetricsproviders,perhapsbecausetheyareincendiaryorembarrassing(e.g.rankings,disclosures,overlyambitiousinformationsharingregimes).Providerstendnottowanttodeliverunwelcomenews,forexampleproofthatpastinvestmentshavedeliveredlittlebenefit.Themostsignificanttensionappearstobebetweentheneedtoinformfinancialdecisionsandthereluctancebysomeproviderstodelivermetricswhichover-promiseonthatfront.Theextenttowhichthissomethingtodowithtrustandliability,oralackofmutualunderstandingbetweenconsumerandproviderwouldrequirefurtherstudy.

➡ Theresponsesalsorevealedasenseofmistrustinmetricsdeliveredbysomecommercialproviders.Whilethesemetricswouldbeincreasinglyusefulasmoreservicesareoutsourced,therewasafeelingamongsomeparticipantsthatcommercialserviceprovidershadn’tnecessarilythevestedinteresttoprovideaccurateortimelymetrics,especiallyifServiceLevelAgreementswerebeingbreached.

➡ Someconsumerrequirements–whilevalid–aredifficulttoachieve,perhapsbecauseoffinancialconstraints,frailtyofcommercialproductsorlackofqualitydata.Forexample,wesawafewcommentswhichsuggestedthatcommonvulnerabilityscanningtoolslackedaccuracy.Otherexamplesofmetricswhichcouldbedifficulttoproduceincluded:protectivemonitoringandalertingcapabilities,assessingthetruecostofanincident,gainingconfidenceincyberinsurancepolicies,understandingtheoverallcostsfromcybercrime,metricizingtheblockerstosuccessfulGDPRcompliance,anddelivering“micro-narratives”todecision-makers.

➡ Thereweresomenotableinstanceswhereprovidersofmetricscouldoffernewideas.Forexample,howmuchsystemsdowntimehadbeenencounteredduetorogueeventsoractivitieswhichwerebeyondthecontrolofnetworkmanagers.Someproviderssawvalueinengaginginadialoguewithleaderstorefineprioritiesandbuildconsensus,forexamplewhereexcessivecosthadbeenincurredduetonotinvestingearlierinsecurity.Therewasalsothesuggestiontoobtainaccurateimpactandvulnerabilityscoringthroughtable-topexercising.Someproviderswereinclinedtodeliverquiterevelatorymetrics–conditionalonfeelingsafetodoso–includingthereal“value”ofextendedsupportcontractsandevidenceofsensitivecompromisesandvulnerabilities.

Page 4: Metrics workshop analysis · 2018-07-05 · metrics. The purpose of the workshop was to develop a deeper understanding of the ways in which cyber security metrics are used in decision-making

4

WHATCONSUMERSWANT WHATCONSUMERSDON’TWANT

THECOMPETITION:• HowamIperformingagainstmycompetitors?• Whathappenedtomyoppositenumbersinother

companiesafterabreach?• Whatwastheimpact/costoftheirmistakes?• Benchmarking:howamIdoingcomparedtomypeer

group?Whatistheprobabilityofabreachcomparedtomycompetitors?

• Made-upnumbers.• Feelingbombardedby

meaninglessornitty-gritty,technicaldata.

• Snapshotswithnotemporalcontext.

• RAG(Red,Amber,Green).• Unqualifiedopinion.• Biasorexcessivesubjectivity,

especiallywrappedupinsomethingsuccinctorscientific-looking.

• Fear-mongering.• Blame.• Salespitchforsnake-oilormagic

bullets.• Unstableorunrepeatablestats.• Styleoversubstance.• Jargon.• Olddata.• Inappropriateorineffective

visualisationtechniques.• Spin.

IT’SALLABOUTTHEMONEY• Relevance:whatismyreturnoninvestment?• Howdomycyberrisksaffectmyabilitytoraise

capital?• HowmuchriskamIunabletotransfer(e.g.through

cyberinsurance)?• AmIcarryingcriminalliabilitythatIcan’ttransfer?• Gettingtorootcause:solvemanyproblemswithone

fix.• Costofrecoveryvscostofcontrol(proactivevs

reactivepostureoncybersecurity,andcomparativecosts).

• What’stherisk/costassociatedwithdoingnothing?

CYBERSECURITYATAGLANCE• SomecommentspointedtowardsabasicSecurity

Operationscapability• Dashboarding,e.g.networkboundaryactivities

THREATS&VULNERABILITIES• Pastbreachesareagoodindicatoroffuture

vulnerabilities.Andmetricsofrealincidentsaremoreinformativethanpotentialones.

• Isthreatintelligenceused?Howusefulisit?Whereelseisitused?Whereisthreatintelrealandvaluable?

Page 5: Metrics workshop analysis · 2018-07-05 · metrics. The purpose of the workshop was to develop a deeper understanding of the ways in which cyber security metrics are used in decision-making

5

• Whataremypriorities,ratherthanjustloomingthreats?Don’tscareme,informme.

• Aretherequickwinsavailable?HowcanImakesomeprogressfast?Helpmeliftopportunitiesoutofthenoise–whatcanIdorightnow?

• Trackingcapabilityoflow-capabilitythreatactors.• Impact,ratherthanquantity,ofincidentsis

important.

THEUSER• Areusersabsorbingtraining?E.g.arethey

forwardingsuspiciousemailsontothesecurityteam?

• Securityvsproductivity:spottingwheresecuritypolicyisfatiguingpeopleandimpedingproductivity.

• Cultureandindicationsofuserwellbeingandbehaviour(andtheriskthatpresents).

THIRDPARTIES• Whereissensitivedatagoing?Whomisitbeing

sharedwith?• Howdoyoutrustthestatsprovidedtoyoubythird

partyserviceproviders?

Page 6: Metrics workshop analysis · 2018-07-05 · metrics. The purpose of the workshop was to develop a deeper understanding of the ways in which cyber security metrics are used in decision-making

6

APPENDIXA:TranscriptionofCyberMetricsworkshopresponses

PartOne:Responsesfromconsumersofcybersecuritymetrics

“Keepitcoming!”(Alreadyreceivethisandfindituseful)

• CommonVulnerabilities&Exposures(CVE)vulnerabilitydatabase• Resultsofcyberdefencematurityassessments• Detailsofpastincidents• Timingseriesanalysisofpastbreaches• RiskassessmentofITchangeprojects• Patchingstatus• CyberEssentials• Consistenttimeseriesmetrics• Organisations’strategy:relevanttohowthisbusinessmakesitsmoney• Incidents,butbydept/functiontofindriskareas:simpletotalnumberisnotthatuseful• RiskassessmentsofITchangeprojects:base,withcontrols,costsandoptions• Returnoninvestment• Barrierstouptakeofbehaviour• Realincidentbreaches,ratherthanpotentialones• Punishmentsreceivedbyotherboards’directors• Overviewofattackattempts,especiallyovertime• Livedashboard,e.g.AVlevels/status• Metricsintermsweunderstand,e.g.businesslanguage• Understandingperceptionsofrisk• Doorganisationsunderstandwhatiscriticalforthem,i.e.whattoprotect?• Malwarenotifications• Qualitativeanalysisofwhat’suseful• Whatismyuninsuredrisk?• Datathatenablesmetodosomething,whatchangescantheorganisationdotorespond?• Whatincidentshaveaffectedorganisationsandwhatisthetrend?• Short,sharpandtothepoint• Howdoesmylevelofuninsuredriskaffectmyabilitytoraisecapital?• Metricsalignedtosolutions• Levelofrealvaluableknowledgesharingwithinasector• Doorganisationsreceiveanyusedthreatintelligence(understand)?

Page 7: Metrics workshop analysis · 2018-07-05 · metrics. The purpose of the workshop was to develop a deeper understanding of the ways in which cyber security metrics are used in decision-making

7

• Datathathasasupportingnarrative• Surgecapacity,usedspace,e.g.whenit’s>80%,itshouldgiveyouanalert• Usagepatterns,responsesinshort,mediumandlongterm• AV/malwarecoverage• No.ofinstallsuptodate• Detectionsalerts:typesandpropagation• Actions:cleanedandquarantined• Networkcapacity:e.g.usedbandwidth/time>x%,ittriggersanalert• Positionmetricsasatooltoenhancethebusiness,nottopresenthurdles• Phishingtests• Governanceofcybersecurity,seeCyberAssessmentFramework(CAF)• Howdoesanexpenditureaffectmyuninsuredrisk?• Bostonconsultancymatrix:Iwanttoknowmoreaboutmy“star”and“cashcow”areasthan

my“dog”ormy“questionmarks”• Areaswherewearenotcompliant• Howdowehavetheconfidencetodiscusscyberonceayear,ratherthanonceamonth?• HowismycybersecuritypostureaffectingmydefencewithrespecttocriminalliabilitythatI

cannotconvert?• Small&Medium-sizedEnterprises(SME)withCyberEssentials(Basic&PLUS),IASMEor

otherlevelofsecuritymanagement

Wishlist(Don’tcurrentlyreceivethisbutwouldliketo)

• ConfigurationstatusofthecorporateIT:noofdevices,softwarerunning,knownvulnerabilitiesmappedfromCVEdatabase,plusseverity

• Sensordata:typesofattack,accessvectors,effects,impact• Effectivenessoftrainingandawarenesspackages,e.g.howmanypeopleclickedthelink

duringaspear-phishingcampaign?• Suitablequantitativemetricsforhumanfactors(notsureifthisispossible)• Physicalsecurity:notseenmuchinthisarea• Riskassessmentandassociatedinvestmentplan• Securebehavioursvstarget/norm,e.g.phishisreported• Supportinganalysis/datathatgivemoredetailswhenneeded• Awareness,engagement:minusdone+howwelldone,e.g.lingertime• Metricsaren’tjustaboutnumbers• Dayslostperyearduetosecurity‘features’• Quantitativeandqualitativedata(holisticview)• Userawarenesslevel

Page 8: Metrics workshop analysis · 2018-07-05 · metrics. The purpose of the workshop was to develop a deeper understanding of the ways in which cyber security metrics are used in decision-making

8

• Robustnessofchangemanagementprocessinorganisation• TimetofixbytheISP/SystemIntegrator• ITprojectswithsecuritydesignedasaproportionofmereexistenceofsecuredesign

practice• Likelihoodofthebreaches(supportedbyrobustmodel,e.g.[couldn’treadit])• Insecurebehaviours:clickingdodgylinks,useofUSBs,useofdropbox,webmail• Doingtrainingtooquickly,ignoringawarenessmaterial• Realincidentdata• Impactofbreaches• Metricsthatshowimpactofincidentsnotjustnumberofincidents• Behavioursthatcouldbeinsecure,whoclickslotsoflinks,whobrowsesalot?• Anythingthatdemonstratesimpactonfinancialaccountingmetrics• Thirdparties• Goodbenchmarks• Threatactoractivity• Relevantbusinessactivity• Puttinginformationinthecorporatecontext,thinkaboutourannualreport• Sector-by-sectormetricswithuniformmethodology• Qualitative,outcomes-baseddata• Networkmapsofhowdifferenttypesofattackaremaybecausedbythesamerootcauses• Aquicksummaryforhighlevelexecutivepack• Identifiedcybersecurityrequirementstobeimplementedbyusers• Quantitativeconsequencesfromcasestudiesofsimilarcompanies• Whenweareinmergersandacquisitions:overviewofsystemintegrationriskaspartofbid

costvsbringingdataontoourexistingsystem• Numberofsilentconnectionstomyphone/device,theirthreatlevelandactionablestepsto

reducerisks• Resilienceofmydevicestodifferenttypesofattack• McGraw/BSIMMdata• Numberofpasswordsre-used/repeatedacrosswebsitesandservices• Metricsmusthavecontext,otherwisethey’rejuststats• Mandatorycriteriatobenchmarkcybersecuritystatusofanorganisation• SecurityactionsIhavedoneright• Notificationofpersonalimpact,notorganisationalortechnicalimpact• Quickwinsandlonger-termsolutions• Realtimedashboard• Returnoninvestment/cost-benefit• Pleaseputsystemrisksinbusinesscontext,productivity,costetc• Productivity,cost-benefitofcontrols,e.g.timetrainingvsvalue• Developsecurecodingcapability• Howourcompetitorsaredoing:notseeingmuch• Whatagoodprocesslookslikeratherthananoutcome• IfwehaveoutsourcedourIT,whatinformationshouldwecontractourprovidertoreport?

Howcanwetrustthem?• Understandingofeffectivenessofasecuritycontrol• Quantifiableriskofdoingnothingdifferently

Page 9: Metrics workshop analysis · 2018-07-05 · metrics. The purpose of the workshop was to develop a deeper understanding of the ways in which cyber security metrics are used in decision-making

9

• Robuststatsaboutbehaviourratherthanthinking/intention/awareness• Riskmanagersreport• Option/negotiation:information/threatalertsuggest/requiresactions/response,butwhat

arethealternatives?Thein-betweenoptionsandconsequences?• Meaningfulconnection:whatdothesenumbers/percentagesmeanintermsofrequiredor

recommendedactions?• IndicationofpossibleemployeeabusethatmightactuallyindicateanHRissue,e.g.stressor

poormanagement• Analysiscouchedintermsofbusinesscontinuity• Securityculture(seeCPNItools)• Truecostofrecoveryvscostofcontrol• Wantmore!80%say….[referencetoMadeline’spresentation]• CostofmeasureimplementedthisFY• Formulatoturnthreatintelligenceintoriskprofile• Capabilitylevelofattackeryoucandefendagainst(STIX/TAXII)• Howmuchproductivitylosttostupidsecuritypolicies?• Subjectiveassessmentofriskofattack,threatvsmeasures• Benchmarkingagainstasimilargroupoforganisations• Capabilitytoachieverecoverytimeandpoint[sic]objectives• Lessstats,moreinfo• Changesofnetworktrafficfollowingnewsecuritypolicyimplementation• Incidentstracedtorootcause• Howmanyotherpeopleforwardphishingemailsforanalysis?• Probabilityofabreachinmyindustryformyapplication• Howservicesareinterrogatingmydata,e.g.howismyemailbeingread?• Onwardtransmissionofmypersonaldata• Largecompaniesaretakinganactiveleadershiproleinsupportingtheirindustrysector• Howdoyoucapture/representtheintentoftheactioninametricform?• Performanceoftrainedusersthreemonthsaftertraining?!• Doyouhaveanincidentmanagementplanandhaveyouexercisedit?

“Stop!”(Currentlyreceivingthisbutdon’tfindituseful)

• Makingupnumbers• Numericalstatsoverload• Singlereports,i.e.notinrelationtopastorfuturemetrics• Numberofhitsonmyfirewall• Numberofemployeesclickingonfakephishingemails

Page 10: Metrics workshop analysis · 2018-07-05 · metrics. The purpose of the workshop was to develop a deeper understanding of the ways in which cyber security metrics are used in decision-making

10

• Privacystatementsandpasswordstrength• RAG(RedAmberGreen)ratings:doesn’tmeanthesamethingtoeveryone• “Expertopinion”• Howmanyorgshaveasecuritypolicy?• Numberofincidentsordetectioneventswithnobaserateofoccurrence• Cherrypickingdata,i.e.biasedanalysis• Bullyingwiththreatsof“badthings”• Machinespatched• Networkmonitoringstats• Blamingusersbytellinguswhatthey’vedonewrong• Userstrained• Magicbulletsolutions• Trafficlights• Anythingthatyoucannotprovetomewillbestableenoughtoinvestinmeasuringovertime• 3Dpiechartsorbubblecharts• Uncontextualizednumbers• Non-contextualstats• Metricsfullofjargonwithoutexplanations• Drowningmeindata• “indexes”thathidecomplex,subjectivemethodologies• SIEM• AVstats• Patchedpercentages:I’monlyinterestedineffects• Nitty-grittydetailofsystempatching• Tick-boxprocessconfirmation• Outdateddata• Anythingqualitative• Historicevents,e.g.pastebin• Detailsofindividuallower-significanceincidents/issues• Progressagainstcompliancerequirement• Irrelevantdatawheretheresultsarenotsignificant• Phishingteststats:hugelyvariableifcomparedonetothenext,butusefultocompare

betweendeptstofindriskareas• Poorpresentationofthegraphics,e.g.poorchoicesofcolour,inappropriatecharttypes• Incidentnumberswithoutcontext,e.g.6,000incidentsacrosssectorXin2015

Page 11: Metrics workshop analysis · 2018-07-05 · metrics. The purpose of the workshop was to develop a deeper understanding of the ways in which cyber security metrics are used in decision-making

11

“Thanksfortheoffer,but….”(Don’treceiveitandwouldn’twanttoifitwereoffered)

• Metricsforafee• Riskmetricswithoutsolutions• Metricsthatpromptmorequestionsthananswers:don’tgivemeproblemswithout

solutions,I’mbusyenoughalready!• Datathathasbeensanitisedbymiddlemanagement• Unsolicitedsalespitch:informationgathering,socialengineering

Page 12: Metrics workshop analysis · 2018-07-05 · metrics. The purpose of the workshop was to develop a deeper understanding of the ways in which cyber security metrics are used in decision-making

12

PartTwo:Responsesfromprovidersofcybersecuritymetrics

“Ican/doprovidethis”(Currentlydeliveringandplantocontinue)

• Totalcybersecurityspendasapercentageofrevenue/profit• Existenceandrehearsalofincidentresponseplan• BGProutingtables• DNStracedataaboverecursiveresolver• Likelihoodoffuturebreach• Numberofphishingattacksinmyorganisation• Timetoresumeservicedeliverypost-incident• Impactofincidentonservicedelivery/BaUprocesses• Timetoincident/compromisedetection• Maxnumberofroguechangedaysonmynetwork• Evidenceofnetworkcompromise• #comment:categorisepreventionmetrics,detectionmetrics,responsemetrics,and

recoverymetrics• Totalnumberofknownvulnerabilitiesonthenetwork• Numberofdetectednetworkintrusions• Numberofbreachesasaresultofuntargetedandunsophisticatedattacks• Riskregisterentries,i.e.likelihoodofabreachofcustomerpersonaldata• Motivationofthreat• SoftwareinventoryviaSoftwareID(SWID)tags• Casestudies/scenarios• Allmetricsareproxiesandsubjecttocalibrationerrors• Numberofemployeesattending/completingtraining,infosec,phishingetc(doesn’tshow

howeffectiveit’sbeen)• Evidenceofcompromise• Numberofstaffwithoutadequatesecuritytraining• Metrics/dataforthesakeofit=comforting• “value”ofextendedsupport• Worstcasescenario,e.g.dayswebsitewouldbedown,systemsthatwouldneedtobere-

built• Uptakeof/barrierstopasswordbehaviour(maybe)• Costofpastbreaches• Totalcosttoinsure

Page 13: Metrics workshop analysis · 2018-07-05 · metrics. The purpose of the workshop was to develop a deeper understanding of the ways in which cyber security metrics are used in decision-making

13

• Quantifiedinformationonpersonaldata,i.e.whatisourexposure,emailandcarddetailsforonemillioncustomers

• Moneyspentrespondingtopreventableincidents,i.e.ifthere’dbeenmoreinvestmentinthefirstplace

• Supportstatusofmyestate• Performanceofsecurebehaviours:reportingincidents,engagingwith

awareness/engagementduration• Cyberbreachessurvey• Numberofpasswordresetrequests• Boardengagementwithcybersecurity(FTSE350survey)• Differentmetricsonthesamesystemfordifferentperspectives• Endusercompliancewithphishingdetection/avoidancerules• Numberofcyberincidentspreventedoraverted• Insiderthreat• AdamJoinson’sobesitymap• Accountinglogs(AAA)• Malwaredetectedandquarantined• Cyberdefencematurityassessments(policy,e.g.NISTCybersecurityframework,CDCat,IA

MaturityModel)• Syslogs:firewalllogs,visualisationthroughgraphs• Patchstatus• Desktopbuild• Phishesblocked• PhishingemaillifejourneyinaSankeydiagram• Capturebysecurityproductvendor• Readteaming:table-topexerciseswithstakeholdersandsystemownerstoagreeimpactand

vulnerabilityscoresforidentifiedattackvectors• Aretheseconvenientbutnotuseful?• Levelofengagementofemployeesingoodpracticesonsecuritybehaviour,bothpassiveand

active• Statisticsonsecurityincidentfromsoftware,hardwareandhumananalysis(serverand

clientsides)• Statisticsonhumansecuritybehaviourestimatedfromsoftware,hardware,surveys,

observations,analysisofdataandreportedincidentsetc

Page 14: Metrics workshop analysis · 2018-07-05 · metrics. The purpose of the workshop was to develop a deeper understanding of the ways in which cyber security metrics are used in decision-making

14

“I’dliketobut…”(Wouldbehappytodeliverbuttherearefeasibilityissues)

• Analyselogsbeforeaproblemoccurs• Completevulnerabilitymanagement• Behaviours/securityculture• Assurancelevels• Numberofunknownvulnerabilitiesinthenetwork• Costofanincident• AmIcompliantwiththetermsofmycyberinsurance?• Asinglemetricthatcanbecomparedacrossallorganisations• Costoffuturebreaches• Accuratecostofcybercrimetoacompany• GDPRconstraints• Micro-narratives• Mismatched/ill-fittingdataprotectionrules• Instrumentsnotavailable(tooexpensive)• Feedbacklooponmetrics–stillrelevant?• Howsecurearewe(withasinglepercentagevalue)?• Qualitativedata:resourceautomate• Informationthatinherentlyrepresentsthetask(conceptualratherthanthedata(computer

science))• Knowingwheninformationisstillrelevantandnotoutofdatetobeofanyuse–create

informeddecision

“IcoulddobutI’drathernot”(It’sfeasibletodeliverbutmakesmeuncomfortable)

• Accuracyofvulnerabilityassessmenttools• Estimatedcostsofsecurityincidentsandcrimeincyberspace,intermsofmonetaryvalue• RAG(RedAmberGreen)ratings• Boardengagementbysector

Page 15: Metrics workshop analysis · 2018-07-05 · metrics. The purpose of the workshop was to develop a deeper understanding of the ways in which cyber security metrics are used in decision-making

15

• Rankingsofcompaniesbycybersecuritycapacity• Fixtimesbysupplier/contractor:incendiaryormisleadingbecauseofServiceLevel

Agreements• CyberEssentialsuptake• Confidentialityconstraintstodisclosingdata• Numbersofscansoneachportonawebserver• Timetofixwebsitevulnerabilitiesinapublicleaguetableoforganisations• Degree/reportofvulnerabilitiesfixed/recommendationsaddressedpost-penetrationtest• Resultsoftestingemployeesecurity/awareness• Daysexposedtodisclosedvulnerabilities• Costofproviding/managingtechnicalcontrols• Numberoftimes(threat)intelhasbeensharedviaCISPetc• Employees’digitalfootprint/corporateinformationexposedviainternet/socialmedia• Numberor%ofemployeespassingformaleducation/training/certificationetc• Benchmarksagainstpeersinsameindustry/sector• Improvement-relatedstatisticsonsecurityincidentsandbehavioursintermsofnumbers• Thesecanbeusedtogameotherindicators• Numberofhigherprivilegeaccesses• Toohardtodisentanglefromvalueproposition• Netflow/IPFixatorganisation/internetboundary• DNSTracedatabelowrecursiveresolver• Madeupdataormisleadingdata• Penetrationtestresults• Provenance• Ensurecoverage/samplesize• Isametricwhichofthese?Evidence,data,measurements,mathematicalsenseofdistance

betweentwoitems(Ithinkwemeanevidence)?• Haverawdatabutdifficulttoaggregateorvisualise• Numberofourbackdoors

Page 16: Metrics workshop analysis · 2018-07-05 · metrics. The purpose of the workshop was to develop a deeper understanding of the ways in which cyber security metrics are used in decision-making

16

“Weneedtotalk”(Thiscannotbedelivered.Evenifitcouldbe,Iwouldn’twantto)

• Anestimateofthenumberofcyberbreachesprevented• Anythingthatshowsmeinabadlight• AgreedmetricsfromUKGovernment(NCSC)forscoringimpactandvulnerability• Anythingthatclaimstodemonstrateimpactonfinancialaccountingmetrics(ProfitandLoss,

balancesheetsetc)• Awidersharing/collaborativenetworkofexpertspreparedtoshareinformationandwork

together• Contextualisedquantitativedata• Numberofcompetitors’backdoors