Building the Enterprise Data Lake with Cloudera & Cisco · cloudera introduces the enterprise data hub and cloudera enterprise 5 2015 cloudera includes kafka, kudu and record service

1©Cloudera,Inc.Allrightsreserved.

BuildingtheEnterpriseDataLakewithCloudera&CiscoPreparedby:MarilynTan,CountryManagerSingaporeXueDaming,SeniorSystemsEngineer


DigitalTransformationwithData


DATAisTransformingtheWorld!

DIGITALDEMOCRACY&SECURITY

- DigitalSharingEconomy:OpenData&Algorithms

- EnterprisereadyOpenSource(e.g.Apache)

- Digital(distributed)Trust(esp.Blockchain)

RISEOFOPENSOURCE

SMARTAPPs

- ConnectedExperience

- Ubiquitouscomputing:Everywhere&oneverydevice(Voice,VR,AR,mobile,Wearables)

THENEWUX

CONNECTEDWORLD

INTERNETOFTHINGSINDUSTRY4.0

- SmartThings/Devices

- NewUseCases

- NewArchitectures

- By2020:20.8bdevices

- IoT:$1.7trillionin2020

MOREDATA

- NewAnalytics

- MachineLearning&AI

- Newdatasources

- DataVirtualization

- DataScience

DATAAS4th PRODUCTIONFACTOR


The9yearClouderajourney…

2008CLOUDERAFOUNDEDBYMIKEOLSON,AMRAWADALLAH&JEFFHAMMERBACHER,CHRISTOPHEBISCIGLIAJOINEDBYDOUGCUTTING(2009)

2013CLOUDERAEXPANDSBEYONDMRANDHBASE,INTRODUCINGIMPALA,SOLRANDSPARK

2011CLOUDERAREACHES100PRODUCTIONCUSTOMERS

2012CLOUDERAENTERPRISE4THESTANDARDFORHADOOPINTHEENTERPRISE

2014CLOUDERAFOCUSSESONSECURITY,ANDGOVERNANCEWITHNAVIGATOR2ANDCLOUDWITHDIRECTOR

2009CDH:FIRSTCOMMERICAL

APACHEHADOOPDISTRIBUTION&

CLOUDERAMANAGER

2011CLOUDERAUNIVERSITY

EXPANDSTO140COUNTRIESSUPPORTIMPLEMENTS

FOLLOWTHESUNMODEL

2012CLOUDERACONNECT

REACHES300PARTNERSACROSSSI,HARDEWARE,

ANDSOFTWAREPARTNERS

2014CLOUDERAINTRODUCES

THEENTERPRISEDATAHUBANDCLOUDERAENTERPRISE5

2015CLOUDERAINCLUDES

KAFKA,KUDUANDRECORDSERVICEWITHINCLOUDERAENTERPRISE

CDH / CM ENTERPRISEDATAHUB

CLOUDERAENTERPRISE

4

2016NAVIGATOROPTIMIZERGENERALAVAILABILITY,IMROVEDCLOUDCOVERAGEWITHAWS,AZUREANDGCP

∀Clouds

2017…CLOUDERAACQUIREDFAST

FORWARDLABS,ANNOUNCEDPaaSALTUS,DATASCIENCEWORKBENCH,

SHAREDDATAEXPERIENCE(SDX)ANDMORETOCOME!

Altus

CDSW


WhatHappenNext:ADecadeofHadoop

18Projects

andbeyond


GartnerAnalyticsAscendancyModel

Information

Optimization

DescriptiveAnalytics

Whathappened?

DiagnosticAnalytics

Whydidithappened?

PredictiveAnalytics

Whatwillhappened?

Hindsig

ht

Insight

Fo

resight

Value

Difficulty

PrescriptiveAnalytics

Whatwillmakeithappened?


Cloudera&CiscoEnterpriseDataLakeInnovation


BigDataStoreUCSC240/C3160

CiscoUCSIntegratedInfrastructurewithClouderaforIoT

Fog KafkaCiscoUCSC240

DataInject(CoAP/MQTT.XMPP) DataProcessing

DATAAggregator

CiscoUCSC240

C800/UCSMini/UCSC240

Real-TimeDataStoreUCSC220/C240

Batch

Real-Time

SpeedLayer BatchLayerISR8x9with4GLTEandDual802.11n

a/g/n(WiFi)RadiosManagedbyCiscoFogDirector

ServingLayer

DataAnalytics

CiscoUCSatalllayers,fullyvalidatedarchitectureswithallmajorplayers


FabricCentricDesign

HighPerformance40GB/sEthernet;320GB/sperChassis

UnifiedFabricSingleCableforNetwork,Storage,andManagementTraffic

EasytoScaleSinglePointofManagement:AddCablesforBandwidthvs.FabricType

UCSManager

Management

Ethernet

Storage


ManagementSimplicity

CentralizedManagement

ServiceProfilesforServers

•Manageallserverscentrally

ApplicationProfilesforNetwork

•Manageallnetworkcentrally

BigData:ManagementConsistency

HundredsofServers

Thousandsofmanagementpoints

SimplifiedScalability

EasilyScaleyourinfrastructurefromfewserverstothousandsofserverswithafullyIntegratedInfrastructure UCSServiceProfile CiscoACIApplicationProfile


TheenterpriseplatformformachinelearningPATTERNRECOGNITION

DETECTIONPREDICTION500+

CUSTOMERSRUNON

DRIVECUSTOMERINSIGHTSMarketsegmentationCustomer360NextbestofferChurnanalysis&prevention

PROTECTBUSINESS

CybersecurityFraudAnti-moneylaunderingRiskmodeling&assessmentSPAMdetection

CONNECTPRODUCTS&SERVICES(IoT)

PredictivemaintenanceGenomics&personalizedmedicinePredictingandpreventingdisease

Naturallanguage


Machinelearningrequiresacomplete stack.

Prepare AnalyzeData Deploy

• Loadexternaldata• Processstructureddata• Processunstructureddata• Processstreamingdata• Cleansedata• Vectorize data

• Diagnose/treatdataissues• Designexperiments• Partitiondata• Engineerfeatures• Trainandvalidatemodels• Evaluateandassessmodels

• PublishtoBI/Viz• Deploytobatchscoring• Deploytoreal-timescoring• DeploytoscoringAPI• Managemodels• Monitormodelperformance

Administration,GovernanceandSecurity

BusinessIntegration

● BatchProcessing● StreamProcessing● InteractiveSQL● SearchTools● Text/ImageProcessing

● AnalyticLanguages● MLLibraries

● BI/Viz● InteractiveSQL● BatchProcessing● StreamProcessing● OperationalDB


Acomplete,integratedenterpriseplatform

ClouderaEnterpriseDataHub

ClouderaDistributionforHadoop


• Supportsdatascienceend-to-end• Fullaccesstodata• Secureself-serviceprovisioning• Containerizedenvironments• SupportsPython,R,andScala• Automates:

WorkflowVersioncontrolCollaborationSharing

Cloudera Data Science Workbench


DataScientists•Webbrowser,nodesktopfootprint• UseR,Python,orScala• Installanylibraryorframework• Isolatedprojectenvironments• Directaccesstodatainsecureclusters• Shareinsightswithteam• Reproducible,collaborativeresearch• Automateandmonitordatapipelines• Built-injobscheduling

IT

• Supportself-servicedatascience• Fullplatformsecurity• Kerberosauthentication• Runon-premisesorinthecloud

CDSWBenefits


DeeplearninginClouderawithApacheSpark

• Twopackages:• CaffeOnSpark• TensorFlowOnSpark

• DevelopedbyYahoo• PythonandScalaAPIs• AllDLarchitectures• Integratedpipeline• Runonexistingclusters• Trainingandinference

• OpensourceDLlibrary• DevelopedbySkymind• BuiltonJVMs• SupportsCPUsandGPUs• Java,Scala,PythonAPIs• Trainingandinference• Importsmodelsfrom:

• TensorFlow• Caffe• Torch• Theano

• Runsonexistingclusters

• Deeplearningframework• DevelopedbyIntel• SupportsCPUsonly• LeveragesIntelMKL• Scala,PythonAPIs• Importsmodelsfrom:

• TensorFlow• Caffe• Torch

• Runsonexistingclusters

SparkPackages DL4J BigDL


“OurdatascientistswantGPUs,butwecan’tfindawaytodelivermulti-tenancy.Iftheygotothecloudontheirown,it’sexpensiveandwelosegovernance.”

● ExtendexistingCDSWbenefitstoGPU-optimizeddeeplearningtools

● Schedule&shareGPUresources

● TrainonGPUs,deployonCPUs

● Workson-premises orcloud

New!Accelerateddeeplearningon-demandwithGPUs

DataScienceWorkbench

GPUCPU

CDH

CPU

CDH

CPU

single-nodetraining distributedtraining,scoring

Multi-tenantGPUsupporton-premisesorcloud


EnterpriseDataLakeArchitecture


CanonicalIngestion&SparkStreamingAnalyticswithCiscoBigDataAnalyticsPlatform

• IntegratewithApacheSparkStreamingforreal-timeanalysisofdata• WritebacktoKafkaforfurtherprocessingortosendtoanapplicationlayer


ProposedArchitectureforEnterpriseDataPlatform

DataWarehouse

DataWarehouse

DataSourcesMeteorologicalData

SensorsData

Unstructuredfiles

Social

Video/Image

Historian/SCADA

Geospatial

BWHANABPC

anySAPNW

DataVisualizationAdvancedAnalyticsAIPlatform

DataWarehouseEnterpriseDataWarehouse


BigDataBlueprints:CiscoValidatedDesignsDesignsBigData

Cisco® ValidatedDesignswithCloudera

Solutiondesigned,tested,anddocumentedtofacilitatefaster,morereliable,andmorepredictablecustomerdeployments.

Whatyouget

Industry-leadingpartnerships

Testedandvalidatedreferencearchitecturestomeetperformance,capacity,andscale

Jointengineeringlab

Extensiveoptionsfordatamanagement(Hadoop,MPP,andNoSQL)tomeetyourbusinessneedsSolutionbundlesoptimizedforcostofownershipandeaseofordering


OurCustomers’SuccessStories


UsingPredictiveMaintenancetoImprovePerformanceandReduceFleetDowntime• OnCommandConnectioniscollectingtelematics andgeolocation dataacrossthefleet

• Reducedmaintenancecoststo$.03permilefrom$.12-$.15permile

• Centralizingdatafrom13systemswithvaryingfrequencyandsemanticdefinitions

• Real-time visibilityof250,000+ trucksinordertoimproveuptimeandvehicleperformance

TRANSPORTATION»PREDICTIVEMAINTENANCE»IMPROVEDSERVICE»DATADRIVENPRODUCTS

DATA-DRIVENPRODUCTS

CASESTUDY


UsingSensors&IoTtoImprovePassengerSafetyandAirportEfficiency

Challenge:• Improvetravelersatisfactionandsafety,byreducingdowntimeforcriticaloperationalmachinery

Solution:• ClouderaonAzuretocapture,secure,andcorrelatesensor(IoT)datacollectedfromescalators,elevators,andbaggagecarousels

• Providenecessaryfixestopreventunplanneddowntime

SmartBuildings- PreventativeMaintenance

DATA-DRIVENPROCESS

CASESTUDY

DATA-DRIVENPRODUCTS

TRAVEL &TRANSPORTATION»SMARTBUILDINGS»PREDICTIVEMAINTENANCE»ADVANCEDANALYTICS


EnablingtheStateofKentuckymanagesnowandiceeventsinrealtime

Challenge:• Neededmoreefficientapproachtoinclementweatherroadmanagement

Solution:• Real-timeweatherresponsesystemthatincorporatesreal-timedatafromWaze,HERE,ESRI’sGeoEvent processor,andAutomaticVehicleLocations(providingsensordatafromsalttrucks).

• KYTCaggregates15-20millionrecordseverydayandprocessmorethanamillionrecordspersecond.

SmartCities

2016DataImpactAwardWinnerStateofKentuckyDepartmentofTransportation

CASESTUDY


DatainapplicationsilosLimitedInsightsSummarized

BasicSecurityControlsAuthorizationAuthentication

ComprehensiveAuditing

DataSecurity&GovernanceLineageVisibility

MetadataDiscoveryEncryption&KeyManagement

FullyComplianceReadyAudit-Ready&Protected

AuditReadyFor:PCIPII

Fullencryption,keymanagement,transparency,andenforcementforalldata-at-restanddata-in-motion

SecurityCompliance&RiskMitigation

0– DatainIsolation

1– BehaviorandTransactionFusion

2– ExpandedDataSurfaceArea

3– EDH:SecureDataVault

DataProtection&Governance


WhyCloudera

ThePlatformforNext-GenerationAnalyticsClouderaEnterprisedeliversthecapabilitiesrequiredbythelargestenterprises,spanninganalytics,security,governance,andmanagement.WemakeHadoopfast,easy,andsecure.

TheExperience toHelpYouSucceedNooneknowsHadooplikeCloudera.AsthefirstHadoopcompany,Clouderais theworld’sleading contributorto andproviderofenterpriseHadoop,withexperienceyoucanrelyontohelpyousucceed.

OpenInnovationOuruniquehybridopensourcestrategyenablesustoleadtheenterpriseexpansionoftheHadoopecosystem,drivinginnovativenewcapabilitiesandopenstandardsinthecommunity.


[email protected] |[email protected] |+6593682316

Documents

Building the Enterprise Data Lake with Cloudera & Cisco · cloudera introduces the enterprise data hub and cloudera enterprise 5 2015 cloudera includes kafka, kudu and record service