WSO2Con USA 2015: Keynote - The Future of Real-Time Analytics and IoT

The Future Of Analytics And IoT It Can’t Happen Without You

Mike Gualtieri, Principal Analyst

November 3, 2015 San Francisco

Twitter: @mgualtieri

#Priority

© 2015 Forrester Research, Inc. Reproduction Prohibited 3

52%

53%

53%

54%

58%

64%

64%

65%

66%

73%

75%

0% 10% 20% 30% 40% 50% 60% 70% 80%

Better leverage big data and analytics in business decision-making

Create a comprehensive strategy for addressing digital technologies like mobile, social & smart products

Create a comprehensive digital marketing strategy

Better comply with regulations and requirements

Improve differentiation in the market

Increase influence and brand reach in the market

Address rising customer expectations

Improve our ability to innovate

Reduce costs

Improve our products /services

Improve the experience of our customers

Customer experience is a top priority for business leaders

›  Base: 3,005 global data and analytics decision-makers ›  Source: Global Business Technographics Data And Analytics Online Survey, 2015

For you For all For segments For you

Demographic Relationships

Hyper-Personal, Real-Time

Relationships

Personal Relationships

Mass Relationships

Cus

tom

er E

xper

ienc

e

1800 1900 1950 2000 2015

#Celebrity

Customers want and increasingly expect to be treated like celebrities.

•  Learn individual customer characteristics and behaviors

•  Detect customer needs and desires in real-time

•  Adapt applications to serve an individual customer

Celebrity experiences must:

#Analytics


Learn Model Detect Adapt

True BI means having four kinds of analytics

Predictive Analytics

Streaming Analytics

Descriptive Analytics

(Advanced Analytics)

Prescriptive Analytics

Batch Real-time

Most firms invest here They must invest here too


Source: Forrester Research

Advanced analytics is rightly surging “What is your firm's/business unit's current use of the following technologies?”

Source: Forrester's Global Business Technographics Data And Analytics Survey, 2015 and 2014 Base: 1805 (2015), 1063 (2014)

19%

19%

24%

31%

34%

22%

22%

35%

31%

43%

53%

54%

50%

50%

69%

39%

42%

42%

42%

42%

43%

43%

46%

48%

52%

54%

55%

56%

57%

69%

Non modeled data exploration and discovery

Search/interactive discovery

Streaming analytics

Metadata generated analytics

OLAP

Advanced visualization

Text analytics

Location analytics

Predictive analytics

Process analytics

Embedded analytics

Web analytics

Dashboards

Performance analytics

Reporting

2015

2014

Sweet! Most of your competitors still haven’t

started!

#Data

Some say that data is the new oil.

It’s more like the Sun – virtually limitless.


Every industry is graced with more data ›  Richer transactional data from portfolio of hundreds of

business applications

›  Usage and behavior data from web and mobile apps

›  Social media data

›  Log data

›  IoT device sensor and event data

›  Data economy – firms buying and selling data

Using your best estimate, what is the size of all data stored within your company?

Source: Forrester Research, September 2015 Base: 100 US Managers and above currently using Hadoop for processing and analyzing data.

Enterprises have plenty of data from both internal and external sources

10-49 Terabytes

5% 50-99 Terabytes

12%

100-500 Terabytes

54%

Greater than 500

Terabytes 29%

Internal business

data 49%

External source data

51%

What % of the data available is from internal business applications (ERP and business

applications) versus external sources (social, IoT)?

All data is born fast!

1100

1001

1011

001

0100

1001

1011

001

0100

1100

1101

101

0100

1001

1011

001

Cus

tom

er D

ata

Tran

sact

ions

Dat

a W

areh

osue

IoT

But, analytics is usually done much later.

#WhyWait

Perishable insights can have exponentially more value than after-the-fact traditional historical

analytics.

#Perishable

How can you prevent this dude from fleecing you right now?

What are movers and shakers saying about equities that we cover right now?

How can you know if your baby is sleeping soundly or if something is wrong right now?

How can you warn other drivers that the road is slippery to avoid a crash right now?

What offers should you make to your customer if they are within proximity of your store right now?

What music should you play if your customer is jogging right now?

#


Most firms struggle make insights actionable

Insights for lines of business:

Source: October 16, 2014, “The Customer-Activated Enterprise” Forrester report

Applications are blind – IoT can make them see.



Build analytics into your applications


Streaming Analytics


(Advanced Analytics)


Batch Real-time

Most firms invest here They must invest here too

#Predictive


Top 3 most read Forrester research reports in Q3 2015.

#1 Does Customer Experience Really Drive Business Success?

#2 Forrester Wave: Big Data Predictive Analytics, Q2 2015

#3 The US Customer Experience Index, Q12015

ANALYTICS

PREDICTIVE Techniques, tools, and technologies that use

data to find models – models that can anticipate outcomes with a significant

probability of accuracy.


Ways to create predictive models ›  Handcraft business rules or program code to

create a predictive model based on human experts.

›  Use big data predictive analytics tools to build models that analyze data with machine learning algorithms to build a predictive model.

›  Sometimes you need both.


Predictive models can be very powerful and profitable, but understand that:

›  Predictive models are about probabilities, not absolutes •  E.g. 78% chance you will like Better Call Saul

›  Accurate predictive models may not exist for every question

•  E.g. Economists have a very poor record

›  Prediction models are not necessarily and probably not causative

Correlation does not imply causation.

Data scientists use a combination of statistical

and machine learning algorithms to find

patterns and predictive models.


Are you a data scientist?

38

K-means clustering Association rules Boosting trees CHAID Cluster analysis Feature selection Independent components analysis Kohonen Networks (SOFM) Neural networks Social network analysis (SNA)

Random forests Mars regression splines Linear and logistic regression Naïve Bayesian classifiers Optimal binning Partial least squares Response Optimization Root cause analysis Support vector machines Natural language processing


Big data is the fuel and machine learning is the engine ›  Classifiers

•  Predict a specific event, characteristic, or behavior

›  Recommenders •  Make a recommendation

›  Clusters •  Find groups that share common characteristics

What customers are likely to plan a vacation in the next 6 weeks?

#Prescriptive

Adapt and act in real-time.

ANALYTICS

PRESCRIPTIVE Tools, techniques, and technologies that are used to determine the next best decision or

action using a combination of methods including business rules, descriptive/predictive/streaming

analytics, and optimization.

How can you influence swing voters to vote for you?

Image source: iStockphoto

45 © 2015 Forrester Research, Inc. Reproduction Prohibited

You must support both analytical and intuitive decision makers.

Analytical Intuitive

Automated Decision can be modeled Information is available Algorithm can be defined

Expert Decision is not modeled Unknown or fuzzy information Experience and judgment

Image sources: Cryteria, Eurogamer.net, Business Insider, CBS Entertainment, Cinema Blend

46 © 2015 Forrester Research, Inc. Reproduction Prohibited

Prescriptive analytics relies on a combination of approaches •  Business rules and/or program code based on

human expertise and experience. •  Use previously built predictive models. •  Use previously solved mathematical

optimizations. You will likely need a combination of all of these to make the best possible decision.

#Streaming

Streaming analytics can detect and act on perishable insights.

DEFINITION

FORRESTER Streaming analytics filter, aggregate, enrich, and analyze a high throughput of data from

disparate live data sources to identify patterns, detect urgent situations, and automate

immediate actions in real-time.


Real-time means business-time ›  A customer walks into a shopping mall

›  A shopper clicks on an online add

›  A temperature sensor spikes

›  A stock price rises

›  A customer uses a credit card

›  A customer wakes up


Thinking in streams is different…

›  Ingest

›  Filter

›  Transform

›  Normalize

›  Link

›  Enrich

›  Correlate

›  Location/motion (geofencing)

›  Time windows

›  Temporal pattern detection

›  Business logic/rules execution

›  Action interfaces

Continuous ETL Continuous Analytics

How can an online retailer sell more motorcycle helmets and optimize

profits?

›  Temporal pattern detection

›  Time windows

›  Business logic/rules execution

›  Action interfaces


Streaming analytics enables an entirely new real-time selling model

› Analytic: When has this user viewed at least three motorcycle safety products including at least one helmet?

› Action: Display most profitable motorcycle helmets.

› Analytic: What is the real-time daily total sales of motorcycle helmets?

› Action: If sales trending lower than usual, then dynamically lower price.

Temporal Pattern Detection Time Window


Try doing that with plain-old, conventional SQL

SELECT SUM(CustomerViews.view) WHERE CustomerViews.productCategory = ‘MCSafety’ AND …

#

Hadoop is designed for volume.

Spark is designed for speed.

Spark on Hadoop minimizes the latency of loading HDFS data into memory, and DevOps can use

YARN to tune performance for simultaneous jobs.


Spark and Hadoop can coexist in the same cluster.

Hadooponomics makes batch analytics at scale feasible

SELECT * FROM Hadoop;

Pure SQL for Hadoop

Boosted SQL for Hadoop

•  Apache Hive •  Apache Drill •  Apache Phoenix

(for HBase) •  Cloudera Impala •  Presto •  Spark SQL

(through Spark)

•  Actian Vortex •  HP Vertica SQL

on Hadoop •  IBM Big SQL •  JethroData •  Pivotal HAWQ

•  Microsoft Polybase

•  Oracle Big Data SQL

•  Teradata QueryGrid

Database+ SQL for Hadoop

Spark’s directed acyclic graph (DAG) engine maximizes parallelization for batch jobs.


Spark also includes a growing number of specialized tools

#Lambda



Only the analytical enterprise can compete and win in the age of celebrity


Streaming Analytics


(Real-time)


(Continuous Batch)

þ þ þ þ Invest here Invest here Invest here


Combine application middleware and analytics to create industrial strength “lambda” applications

#

1100

1001

1011

001

0100

1001

1011

001

0100

1100

1101

101

0100

1001

1011

001

His

toric

al

Tran

sact

ions

Cus

tom

er d

ata

Sec

urity

Silos must disappear or appear to disappear.

Performance of analytics and services must be blazing fast.

Scale to handle any amount of data.

Fault-tolerance is non-negotiable.

Confidential information must be secure.

Solutions and/or platforms must fit seamlessly into existing architectures.

Embed and act on real-time analytics in existing and new applications.

Diverse, community-based rapid innovation.

#You

Your customers are ready to be

treated like celebrities!

What kind of celebrity customer experiences can you create with a

great open source platform?


Think like a venture capitalist to find opportunities in your business.

1. Walk through critical or challenging business processes - At each step of the business process ask how analytics could improve the process

2. Walk through customer experience to improve customer experience

-  At each step of the customer journey, ask how analytics could help create celebrity customer experiences

Build real-time, predictive apps to make it happen.

forrester.com

Thank you

Mike Gualtieri [email protected] Twitter: @mgualtieri

Technology

WSO2Con USA 2015: Keynote - The Future of Real-Time Analytics and IoT