10
How machine learning is shaping the future of data warehousing 3 BENEFITS OF A SELF-ADAPTING DATA WAREHOUSE

3 BENEFITS OF A SELF-ADAPTING DATA WAREHOUSE...exciting new era: the age of the self-adapting data warehouse. In the pages that follow, we’ll explain how data warehousing evolved,

  • Upload
    others

  • View
    2

  • Download
    0

Embed Size (px)

Citation preview

Page 1: 3 BENEFITS OF A SELF-ADAPTING DATA WAREHOUSE...exciting new era: the age of the self-adapting data warehouse. In the pages that follow, we’ll explain how data warehousing evolved,

How machine learning is shaping the future of data warehousing

3 BENEFITS OF A SELF-ADAPTING DATA WAREHOUSE

Page 2: 3 BENEFITS OF A SELF-ADAPTING DATA WAREHOUSE...exciting new era: the age of the self-adapting data warehouse. In the pages that follow, we’ll explain how data warehousing evolved,

2 How do we use all this data?

3 The increasing complexity of data warehousing

4 Towardtheself-adaptingdatawarehouse

5 Benefit#1:Enjoygreaterspeedandagility

6 Benefit#2:Anticipatepatternsfasterandoptimizequeries

7 Benefit#3:Automateddataorganization&optimizedworkloads

8 Conclusion

9 AboutSnowflake

Page 3: 3 BENEFITS OF A SELF-ADAPTING DATA WAREHOUSE...exciting new era: the age of the self-adapting data warehouse. In the pages that follow, we’ll explain how data warehousing evolved,

HOW DO WE USE ALL THIS DATA?

As businesses become more data-driven, data warehouses play an increasingly important role, helping businesses store, process, harness, and manage their data to make informed business decisions. The underlying goal for any business leveraging a data warehouse is to reduce the time to value for that company’s data. In other words, the goal is to shorten the amount of time it takes for a business to derive value from its data.

While that goal has remained steadfast over time,achievingithasbecomemoredifficult.Datawarehousinghasbecomemorecomplicatedduetotheriseofnewdatastreamsandtheexponentialgrowthofthedatawithinthosestreams.Atpresent,only1%-2%ofacompany’sdataisconsideredactionable,andsoisstoredlong-term,accordingtoIDC.1Clearly,thechallengeofloading,storing,andmanagingdataisonlygoingtogrowlarger.

Asdatasetshavegrownincomplexity,sohasthetechnologyrequiredtosupportafullyintegrateddatawarehouse.Butastechnologyadvancestomeetnewdatademands,italsocreatesnewareasofopportunityforbusinessgrowthandoperationalefficiency.Theabilitytoaccessmoredataandprocessitmorequicklywillenablebusinessestogeneratedeeperinsights,fasterthaneverbefore.

Today,forexample,sometechnologiesalreadyenableautomaticqueryoptimization,whilemachinelearning algorithms help automate a variety of once-manualfunctions.Advancesintechnologyareevenstartingtoletdatawarehousestunethemselves.Thiscapabilityisacceleratingthespeedatwhichdatawarehousesdelivervaluetobusinesses.AtSnowflake,Inc.,weseethesebenefitsfirsthand,asourcustomersbringmoredataofvaryingtypesintoSnowflake,andtheygetincreasingvaluefromthatdataatafasterpace.

With the advance of machine learning and availabilityofnear-infinitestorageandcomputingpowerinthecloud,we’reheadedtowardan excitingnewera:the age of the self-adapting data warehouse.

Inthepagesthatfollow,we’llexplainhowdatawarehousingevolved,thebusinessdriversfuelingthemovetoaself-adaptingdatawarehouse,andthebenefitsthatawaitdatawarehousearchitectsandbusinessleaderswhomaketheshiftfromlegacyplatforms.

2

1IDC’sFirstGlobalStorageSphereForecastSeesInstalledBaseofStorageCapacityMoreThanDoublingby2023:https://bit.ly/2USrXeA

Page 4: 3 BENEFITS OF A SELF-ADAPTING DATA WAREHOUSE...exciting new era: the age of the self-adapting data warehouse. In the pages that follow, we’ll explain how data warehousing evolved,

3

THE INCREASING COMPLEXITY OF DATA WAREHOUSING

As companies track data of varying types from an increasing number of sources and channels, it’s no surprise that managing and getting value from the data has become more complex.

Legacydatawarehouseplatformshavebecomemorecomplicatedbynecessity.Theamountofdataabusinessmustprocesstodaywouldseemunfathomabletoadatabaseadministratoronlyadecadeago.Whereweoncetalkedaboutmegabytes,gigabytes,andterabytesofdata,moreoftentodaywetalkabouthundredsofterabytes,ifnotpetabytes,ofdata.Inthepastdecadealone,newandconstantlyevolvingdatastreams,rangingfrommobileandclickstreamdatatomachinelogsandothersystem-generateddata,haveoverwhelmedtheselegacysystems.This,inturn,hasrequireddatawarehousearchitectsanddatabaseadministratorstodevelophighlyspecialized,platform-specificskillstocreateworkarounds,customtools,andnewcapabilitiestohandletheincreasedload.

Despitetheincreasedcomplexity,leadingbusinessesrecognizetheimportanceofdatainmakingdecisions,andtheyareinvestingmoreintime,tools,andpeopletomanagetheirdata.Technology investments are expected to increase

atmorethanhalf(57%)oforganizationsworldwideinthenext12months,withcloudcomputingandbusinessprocessmanagementastwoofthetopthreeareastargetedforgreaterinvestmentbytechCIOs.2RecentIDCresearchhighlightstherapidaccelerationofdata-drivenbusinesspriorities,predictingthat:

• Worldwide revenue for big data and business analytics solutions will reach $260 billion in 2022, representing an 11.9% compound annual growth rate (CAGR) from 2017-20223

• The installed base of storage capacity worldwide will more than double over the 2018-2023 forecast period, growing to 11.7 zettabytes (ZB) in 20234

1IDG.CIOTechPoll:TechPriorities2019.https://bit.ly/2GrxN3R2RevenuesforBigDataandBusinessAnalyticsSolutionsForecasttoReach$260Billionin2022,LedbytheBankingandManufacturingIndustries,accordingtoIDC.https://bit.ly/2Mlkt3z3IDC’sFirstGlobalStorageSphereForecastSeesInstalledBaseofStorageCapacityMoreThanDoublingby2023.https://bit.ly/2USrXeA

CH

AM

PIO

N G

UID

ES

Page 5: 3 BENEFITS OF A SELF-ADAPTING DATA WAREHOUSE...exciting new era: the age of the self-adapting data warehouse. In the pages that follow, we’ll explain how data warehousing evolved,

TOWARD THE SELF-ADAPTING DATA WAREHOUSE

4

Fortunately, computing power and memory have advanced to the point that machines can process much larger and more complicated data sets. Data warehousing can greatly reduce the time to value for a company’s data and make data management processes more efficient.

Asaresult,manybusinessesareshiftingtocloud-baseddatawarehousingasaservice,inwhichbusinessescanaccessandquerytheirdatainnearrealtime,withoutincurringthecostsofhousing,managing,andmaintaininganon-premisessolutioninadatacenter.Thisshiftfromfixedcapacity,on-premises,serverstoelasticcloud-baseddatawarehousingasaservicealsotransformsworkloadtuningactivities,oneofthecriticalpartsofasuccessfuldatawarehouse,fromamanualin-houseactivitytopartoftheserviceoffering.

Thesecondmajordevelopmentistheemergenceofself-adaptingdatawarehouses.Withself-adaptingdatawarehouses,tuningisnotjustpartoftheserviceoffering,butcanbeincreasinglyautomatedwithmachinelearning.Thescaleandvolumeofdataisn’talimitingfactor.Instead,it’sanadvantage,helpingdiscernpatternsmorequicklyanddeliveringtherightdatatotherightperson,attherighttime.

Inthisway,datawarehousingwillresembleothersmartapplications,whichincorporatedata-driven,actionableinsightsintotheuserexperienceitself.AsSnowflakeInc.,VPofEngineeringSameetAgarwalsaid,“Idon’twanttorunmyownsoftware.It’smuchbettertotreatmydatawarehouselikeaniPhone.WithmyiPhone,IletAppleupdatemysoftwareonthebackend.Igotobedandchargemyphone,and‘magically’it’supdated.That’safarbetteruserexperience.”Likewise,datawarehousingasaservicehasadvancedto the point that upgrades and patches to the system arecompletelyinvisibletotheendusers.

Morebroadly,themovetoaself-adaptingdatawarehousecomplementsanindustry-wideshifttowardagilemethodologies,inwhichbusinessesextract value faster through regular deployment of incrementalimprovements.Intheworldofsoftware,agilemethodshelpdevelopersgettomarketwithaminimumviableproduct(MVP)faster.Intheworldofdatawarehousing,self-adaptingsystemshelpbusinessesgetinsightsfaster,withlessmanualintervention.Asthedatavolumegrows,andmorequeriesareexecuted,theself-adaptingdatawarehousewilldeliverexponentiallybetterresultsbasedontheadditionalinputsithastoanalyze.

Asweentertheageoftheself-adaptingdatawarehouse,threekeybenefitsawaitthosebusinessesthatmaketheshift.

Page 6: 3 BENEFITS OF A SELF-ADAPTING DATA WAREHOUSE...exciting new era: the age of the self-adapting data warehouse. In the pages that follow, we’ll explain how data warehousing evolved,

5

The self-adapting data warehouse will load, process, and present data in the best possible way for each user without human intervention, delivering more business value faster.

Aself-adaptingdatawarehousewillanalyzelargerdata sets—data sets that were previously out ofreach—morequickly,withoutthemonthsofpreparationandplanningrequiredforlegacyon-premisesplatforms.Forexample,amajorpaymentsprovider loading one trillion rows of semi-structured dataperdaywantedtoanalyze15yearsofweblogdata.Inthepast,thiswouldhavebeenimpossible.Withaself-adaptingdatawarehouse,it’spossibletoloadthedataandbeginanalyzingitimmediately,withouttheneedtospendtimetuning.

BENEFIT #1: ENJOY GREATER SPEED AND AGILITY C

HA

MP

ION

GU

IDE

S

Page 7: 3 BENEFITS OF A SELF-ADAPTING DATA WAREHOUSE...exciting new era: the age of the self-adapting data warehouse. In the pages that follow, we’ll explain how data warehousing evolved,

6

The more data a machine algorithm is exposed to, the faster, and more accurately, it will recognize patterns.

Thatsaid,ittakesalotofdatatoreachastatisticallysignificant,accurateconclusion.Datawarehouseshaveanadvantageinthisarena,sincetheyseedatafromacrossdozensofsourcesandpotentiallymillionsofqueries.Withtheseinputs,aself-adaptingdatawarehousewillanticipatefutureuserqueries(muchinthesamewayauto-fillcompletescommonsearches),orevenrecommendqueryrewritesbasedonpastquerypatterns.Theself-adaptingdatawarehousewillalsoprovideintelligentqueryrewritesautomatically,withtheenginedeterminingwhichview of the data will deliver the fastest result to each userforeachquery.

BENEFIT #2: ANTICIPATE PATTERNS FASTER AND OPTIMIZE QUERIES C

HA

MP

ION

GU

IDE

S

Page 8: 3 BENEFITS OF A SELF-ADAPTING DATA WAREHOUSE...exciting new era: the age of the self-adapting data warehouse. In the pages that follow, we’ll explain how data warehousing evolved,

7

As any database administrator can attest, determining how to index and organize the data for greatest congruence, performance, and accessibility is a challenge in legacy platforms. Getting performance and value out of a data warehouse historically took a lot of time and manual effort for planning and maintenance.

Self-adaptingdatawarehouseswillautomatetheorganizationofthedata,eliminatingthemanualworkrequiredtodesigntheoptimalorganizationofthedata.Eliminatingthesecomplex,errorprone,andmanualtaskscreatesanopportunityfordatabaseadministratorsandarchitectstofocusonbroaderstrategicdatamanagementinitiatives.

Aself-adaptingdatawarehousewillalsooptimizeworkloads.Inthepast,whenmultipleusersaccessedaverylargedatasetsimultaneously,

legacydatawarehousescouldnoteasilyoptimizeallqueriesandresponsetimes.Butaself-adaptingdatawarehouse with an underlying machine learning algorithmcandeterminehowbesttoorganizecomputeresourcesforthevariousworkloads.Thisincludesautomaticallyscalingcomputetosupport increased concurrency when there is an unexpectedsurgeinusage,maintainingaconsistentlevelofservice.Aself-adaptingdatawarehousewillorganizedataandthecomputepowerautomaticallysothatitprovidesthebestperformanceforeachuserandmaximizesthenumber,andtype,ofworkloadsthatcanexecuteconcurrently.

BENEFIT #3: AUTOMATED DATA ORGANIZATION & OPTIMIZED WORKLOADS C

HA

MP

ION

GU

IDE

S

Page 9: 3 BENEFITS OF A SELF-ADAPTING DATA WAREHOUSE...exciting new era: the age of the self-adapting data warehouse. In the pages that follow, we’ll explain how data warehousing evolved,

8

CONCLUSION

The rapidly growing scale and variety of enterprise data, once a concern among data warehouse experts, is quickly becoming a key asset for reducing time to value. By leveraging large-scale data sets and query history, the self-adapting data warehouse will recognize patterns faster, anticipating user needs and optimizing queries, views, and workloads.

Themovetoaself-adaptingdatawarehouserepresentsthenextphaseintheevolutionofdatawarehousing.Butatitsfoundation,it’sareturntotherealvalueofanydatawarehouse–helpingbusinessesderivemorevaluefromtheirdata,faster.

CH

AM

PIO

N G

UID

ES

Page 10: 3 BENEFITS OF A SELF-ADAPTING DATA WAREHOUSE...exciting new era: the age of the self-adapting data warehouse. In the pages that follow, we’ll explain how data warehousing evolved,

ABOUT SNOWFLAKE

©2019Snowflake.Allrightsreserved.

Snowflakeistheonlydatawarehousebuiltforthecloud,enablingthedata-drivenenterprisewithinstantelasticity,securedatasharing,andper-secondpricing,acrossmultipleclouds.Snowflakecombinesthe

powerofdatawarehousing,theflexibilityofbigdataplatforms,andtheelasticityofthecloudatafractionofthecostoftraditionalsolutions.Snowflake:Yourdata,nolimits.Findoutmoreatsnowflake.com.

ABOUT THE AUTHOR KentGrazianoistheChiefTechnicalEvangelistforSnowflakeInc.Heisanaward-winningauthor,speaker,andtrainerintheareasofdatamodeling,dataarchitecture,anddatawarehousing.HeisanOracleACEDirector-Alumni,memberoftheOakTableNetwork,acertifiedDataVaultMasterandDataVault2.0Practitioner(CDVP2),expertdatamodelerandsolutionarchitectwithmorethan30yearsofexperience.