Applying Data Science to Your Business Problem

1 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

World®’16

ApplyingDataSciencetoYourBusinessProblem

PaulDulany - VPDataScience- CATechnologies

SCX31S

SECURITY


©2016CA.Allrightsreserved.Alltrademarksreferencedhereinbelongtotheirrespectivecompanies.

Thecontentprovidedinthis CAWorld2016presentationisintendedforinformationalpurposesonlyanddoesnotformanytypeofwarranty. The informationprovidedbyaCApartnerand/orCAcustomerhasnotbeenreviewedforaccuracybyCA.

ForInformationalPurposesOnlyTermsofthisPresentation


Abstract

Forawhilenow,anumberofindustrieshavebeeninterestedindatascienceandadvancedanalytics.Butitisn’talwaysclearhowbesttousethesewithinthebusinesscontext.Inthissession,we’lldiscusshowtoturnabusinessproblemintoadata-scienceproblem,andthenback.We’llusecard-not-presentpaymentfraudandloginattemptsasexamplesofhowtoidentifytheproblem,determineifdatascienceandadvancedanalyticscanhelp(andifthesituationwarrantsthem),andthenfollowthroughondevelopingasolutiontotheproblem.

PaulDulany,PhDCATechnologiesVPDataScience


Agenda

WHATISDATASCIENCE?

DETERMININGAPROBLEMOFINTEREST

UNDERSTANDTHEPRODUCTIONENVIRONMENTANDDEMANDS

MODELCREATIONANDEVALUATION

Q&A

1

2

3

4

5


KeyPointsforApplyingDataScience

§ Identifyahigh-valueBusinessProblemwithHighQualityData

§ Determinetheclassoftheproblemtosolve

§ Utilizebusiness-domainknowledge– Understandthe"ecosystem"– Defineappropriatemetrics– Understandthedatainfull

§ Developfeaturesandmodels/Evaluate/Iterate

§ Alwayskeepthebusinessprobleminmind!


WhatisDataScience?

§ Theapplicationofanalyticaltechniquestolargeand“big”data– Awidefieldencompassingmanydifferentaspectsofanalytics,

statistics,anddatamining– Fundamentallydatadriven– Baseduponthescientificmethod– Thegoalistousedataandanalyticaltechniquestosolveproblems

§ Requiresknowledgeinmultipledomains– Analytics– Scientificcomputations

– Dataformats– Businessdomain

– Statistics


StepsinApplyingDataScience

§ Identifyahigh-valuebusinessproblem– Thebusinesscaseiscritical

§ IntelligentMainframeOperations– Needearlydetectionofissues

§ Bestistopredictandavoidissues– Currently,falsepositives(falsealarms)aretooprevalent– Expert-maintainedsystemsofthresholdsarehardtomaintain

BusinessProblem



§ PaymentSecurity:– FraudineCommerceisasignificantproblem

§ 3-DSecurewasdevelopedtocombatthis– Issuersincurthemostpainfromthecurrentstate

§ Fraudlosses§ Lossofincomefrominterestandinterchangefees§ Customerexperienceandannoyance§ Costofinboundcalls

– Merchantsfeelpaintoo…

BusinessProblem


BusinessImpactforaClient:3YearImpact

Credit DebitFraudsavings(abovecurrentvendor) £13,441,843 £15,174,365

Interchangefees(abovecurrent) £466,462 £3,441,882

Interest income(abovecurrent) £3,674,803 N/A

Operationalsavings(notcalculated) - -

Total £17,583,108 £18,616,247



§ Identifydatarelatedtothebusinessproblem– Themoredata,thebetter!– Isitcategorical,ordinal,ornumerical?Whatcardinality?– Isthereauniqueadvantageoverthecompetition?

§ PaymentSecurity– 3DSdata:PAReq message,deviceinformation,…– Widemixoftypesofdata– Timeseriesisimportant– SaaSDeploymentallowsqualitydatatobegathered

IdentifyData(ResultsProportionaltoQuality!)



§ IntelligentMainframeOperations– Multiplepossibledatasources

§ VSAM,DB2,IMSDB,IDMS,DATACOM,SMF,Syslogs,Vtape,CICS,…– UtilizeCASYSVIEW’sexcellence– Embedanalyticstodetectabnormalpatterns

IdentifyData(ResultsProportionaltoQuality!)



§ Determinethegeneralclassofproblem– Classification,regression,anomalydetection,etc.– “Supervised”(teachingyourchildrentoread,teachingthemmanners)– “Unsupervised”(university)– “Semi-supervised”(schoollunchroom)



§ PaymentSecurity– Supervisedclassification– Fraudinformationforlossesislikelyingoodshape– Complexitieshappenonceyouhaveasysteminplace

§ Censoredproblem,bothinmarkingandinchangingfraudsterbehavior

§ IntelligentMainframeOperations– Unsupervisedtobegin– Needtodevelopbaselinesofnormalbehavior

§ Butmustprovideresultsfromday0– Possibilityofsemi-supervisedinthefuture

Determinethegeneralclassofproblem



§ Understandtheecosystem– Whatactionscanbetaken?– Isgettingtheresulttimesensitive?

§ IntelligentMainframeOperations– Predictiveanalyticsneeded– Multiplepossibleactions,keyistoinformtheoperator’sactions– Differenttime-scalesforproblems– Reaction-timeiscritical– real-time


StepsinapplyingDataScience

§ PaymentSecurity– Predictiveandprescriptiveanalyticsneeded– Multiplepossibleactions,atthetransactionandthecardlevel– Timingiscritical– real-time,i.e.,<50msforvastmajority

§ Wemustbeabletotakeactionnow

UnderstandtheEcosystem



§ Determinetheappropriatemetrics– Howdowemeasuresuccess?

§ Welldefinedmeasuresarecritical

§ IntelligentMainframeOperations– HighAvailability– Problemavoidance– ReducedMTTR– ReduceSMEdependenceforissuedetection



§ PaymentSecurity– Anumberofpossibilities,baseduponcustomer’sobjectives– Considerthemall

§ Detectionrates§ “Outsort”rates§ False-positiveratios§ False-positiverates§ Value-based/transactionbased/cardbased

Metrics

TOR 𝑆 = ∑ 𝐹 𝑠* + 𝑁 𝑠*�./0.

∑ 𝐹 𝑠* + 𝑁 𝑠*�122.*

TDR 𝑆 = ∑ 𝐹 𝑠*�./0.

∑ 𝐹 𝑠*�122.*


What’sNext?

§ Nowwehaveawelldefinedproblem

§ Sowespendalotoftimewiththedata!– “Browse”thedata– Runsomedescriptivestatistics– Seeifyoucansurpriseyourself– Ifsupervised,viewthetaggingdataandthe

productiondataseparately,andthentogether.


ReviewtheTaggedData

§ Browsethedataagain

§ Runmanystatisticaltest– Bewareof“TargetLeaks”!!– Begingettingafeelforthevariations,correlations,idiosyncrasies

§ Youneverwantperfectlycleandata– Youwantdatathatsimulatesproduction!

§ Bewareofanychangestothedata,especiallynon-causalchanges§ Modeltrainingisanumericalsimulationofproduction


DataPartitioning:BewareWhatYouPartitionandHow!

§ Partitioning– Oftenneedtousestratified

sampling– Whenusingmultipleentities

fortrackingbehavior,interactionsaretricky!§ Lookforirreducibility§ Gotoout-of-timeifneeded

HistoricalData

Training

Fraud

Non-Fraud

Validate

Fraud

Non-Fraud

Holdout

Fraud

Non-Fraud


FeatureDevelopment

§ “Features”pullthedistinguishingcharacteristicsfromthedata– Timeseriesanalysistechniques– DigitalSignalProcessingtechniques– Statisticalmeasuresofdifferences– Bayesianapproaches– Lineardiscriminants– Non-lineartransformations– …


FeatureDevelopment:SimpleExample

§ Determineapeergroupfromthehistoricaldata– Alltransactionswheretherewerefourprevioustransactionsinthelast

week,atleasttwoofwhichwereonthesamedevice,butindifferentcountries

§ Determinethedistributionsofclassesforacontinuousvariable– Let’ssay,theamount– Useadiscriminantcalculationto

determinelikelihoodofbelongingtoeitherclass.


FeatureDevelopment:MoreComplexity

§ Attheotherendofthescaleareonline-learningmodelstodeterminebehaviors– Autocorrelationmodels,exponentialweighting,KDE,etc.– Manytechniques–

§ butmustkeepinmindtheCPUconstraints,I/Oconstraints,etc.

§ Conversionofhigh-cardinalitycategoricaldataintonumericalinputs

𝑥"(𝑡𝑛 , 𝑡𝑛−1, 𝑡0) = 𝛼(𝑡𝑛 , 𝑡0)𝑥𝑛 + 𝛽(𝑡𝑛 , 𝑡𝑛−1, 𝑡0)𝑥"𝑛−1

𝛼(𝑡𝑛 , 𝑡0) =1 − 𝛾

1 − 𝛾(𝑡𝑛−𝑡0)

𝛽(𝑡𝑛 , 𝑡𝑛−1, 𝑡0) =𝛾(𝑡𝑛−𝑡𝑛−1)11 − 𝛾(𝑡𝑛−1−𝑡0)2

1 − 𝛾(𝑡𝑛−𝑡0)


Example:UnsupervisedAnomalyDetection

§ UtilizeHistoricaldatatodefinebandsofdifferentprobabilities– Maprealtimemetricstreamsagainstsystemdefinednormal– Multi-pointalertsgeneratedusingindustry-standardWestern-Electric

rules– Makestaticthresholdsoptional!

Unlikely

MostLikely

Metric

Time

LessLikely


Example:SupervisedModel

§ AlwaysrememberOccam'sRazor!– Amongcompetinghypotheses,theonewiththefewestassumptions

shouldbeselected.– Avoidneedlesscomplexity

§ Startwithsimplemodels,andgrowmorecomplexasneeded– Linearregression– Logisticregression– Decisiontrees– NeuralNetworks– SVM…


Example:SupervisedModelNeuralNetwork


Example:SupervisedModel

§ Therearemanyaspectsoftraininganeuralnetwork– differentactivationanderrorfunctions– differenttrainingalgorithms,– variationsofseeds,learningrate,momentum,– self-regulation,– numberofhiddenlayers,– numberofnodes,– boosting/bagging,– preventingoverfitting,– etc.

TraintheModel(s)


Review

§ Reviewtheresultsofyourtraining,andstartalloveragain!– Considersegmentation– Tryleavingoutthevariableswiththehighestsensitivity– Subdividethedatatoseeifthereareregionsofinstability– Iterateasneeded

§ Finally,selectyourmodel(s)!

§ Butwe’renotdone…– Nowworryaboutcalibration,upgrade/downgrade,primingtime,

packaging,integration,modelreport,…


ModelPerformanceChart


IntelligentMainframeOperations

TypicalVolatility

Anomaly

Tasksreadytobe

disp

atched


KeyPointsforApplyingDataScience

§ Identifyahigh-valueBusinessProblemwithHighQualityData

§ Determinetheclassoftheproblemtosolve

§ Utilizebusiness-domainknowledge– Understandthe"ecosystem"– Defineappropriatemetrics– Understandthedatainfull

§ Developfeaturesandmodels/Evaluate/Iterate

§ Alwayskeepthebusinessprobleminmind!


RecommendedSessions

SESSION# TITLE DATE/TIME

SCX50S ConvenienceandSecurityforbankingcustomerswithCAAdvancedAuthentication

11/17/2016at3:00pm

SCX34S SecuringMobilePayments:ApplyingLessonsLearnedintheRealWorld

11/17/2016at3:45pm

SCT05T ThreatAnalyticsforPrivilegedAccessManagement 11/17/2016at4:30pm


Don’tMissOurINTERACTIVESecurityDemoExperience!

SNEAKPEEK!



WeWanttoHearFromYou!

§ ITCentralisaleadingtechnologyreviewsite.CAhasthemtohelpgenerateproductreviewsforourSecurityproducts.

§ ITCSstaffmaybeatthissessionnow!(lookfortheirshirts).Ifyouwouldliketoofferaproductreview,pleaseaskthemaftertheclass,orgobytheirbooth.

Note:§ Onlytakes5-7mins§ Youhavetotalcontroloverthereview§ Itcanbeanonymous,ifrequired


Questions?


Stayconnectedatcommunities.ca.com

Thankyou.


Security

FormoreinformationonSecurity,pleasevisit:http://cainc.to/EtfYyw

Technology

Applying Data Science to Your Business Problem