When Decisions Can’t Wait: From Analysis to Action …...OPERATIONALIZING ANALYTICS Move analysis to event source, Analyze before data is stored, Keep what is relevant. High-speed

#analyticsx

C o p y r ig ht © 201 6, SAS In st i tute In c. A l l r ig hts r ese rve d.

13433 - When Decisions Can’t Wait: From Analysis to Action – in Real Time

Dan Soceanu, Senior Solutions ArchitectSAS


#analyticsx


SESSION AGENDA

4Use Cases

1

2

3

The Need for Streaming Analytics

The Streaming Analytics Lifecycle

SAS® Event Stream Processing

#analyticsx


BIG DATA NEW LANDSCAPE – NEW NEEDS

Volume

Velocity

Variety

Immediate low latency answers

Reduced time to decision action

Continuously evaluate opportunities and risks

More agile, more responsive

Better equipped to address big data

#analyticsx


STREAMING ANALYTICS DEFINED

… it is about applying analytics while the data is in motion, before

it is stored – and keeping what it is relevant

#analyticsx


Event = a message indicating that something has

happened

Event Stream = an ordered sequence of events, of the same type

ESPEvent Streaming Processing = matching and transforming source events into result events;Data-in-Motion analyses data before storage

DEFINITIONS WHAT IS AN EVENT, EVENT STREAM AND ESP?

#analyticsx


Batch-Loaded Data

WarehouseMicro-Batch

Data Warehouse

Trickle-feed DW with CDC

Complex Event Processing

Event Stream Processing

Days Hours Minutes Seconds Milliseconds Microseconds

NEW ERA OF INFORMATION PROCESSING

THE EVOLUTION OF OPERATIONALIZING ANALYTICS

Move analysis to

event source,

Analyze before

data is stored,

Keep what is relevant.

High-speed querying of

data in streams, and

applying algorithms to

the event data.

#analyticsx


NEW ERA OF INFORMATION PROCESSING

USERS NEED IMMEDIATE DECISIONS

Processing streaming data is about getting immediate answers to reduce time to decision

Time to decisionMicro-seconds Days

Batch

Streaming

Streaming

Analytics

Micro-Batch

#analyticsx


STREAMING ANALYTICS

CROSS-INDUSTRY APPLICABILITY AND VALUE

Cyber Security

IT Operations

Real Time Marketing

Supply Chain

Fraud Detection

Capital Markets

Manufacturing

Industry, Energy

Enterprise Decisions

Telecommunications

#analyticsx


WHEN TO USE SAS ESP?

Digital Security

Connected Devices (IOT) & Sensor Networks

On-line Behavior

When milliseconds matter Where latency devalues events (e.g., operational data) When volumes (i.e. throughput rate) overwhelm existing

analytics When velocity leads to unacceptable latency

SAS EVENT STREAM PROCESSING

#analyticsx


BIG DATA TRADITIONAL ANALYTICS LIFECYCLE

DeployETL

Data Data Storage

f

Access – Store - Analyze

Alerts / Reports

#analyticsx


SENSE – UNDERSTAND - ACT STREAMING ANALYTICS LIFECYCLE

DeployETL

Data Data Storage

Alerts / Reports/ Decisioning

De

plo

y

f

Streaming Data Intelligent Filter / Transform

Streaming Model Execution

#analyticsx


DISTRIBUTED ANALYTICS STREAMING ANALYTICS VALUE STREAM

DeployETL

Data Data Storage

Alerts / Reports/ Decisioning

De

plo

y

f

IoT Data Intelligent Filter / Transform

Streaming Model

Execution

#analyticsx


STREAMING ANALYTICS CONTINUUM

Cloud Streaming Edge


#analyticsx


SAS®

Event Stream ProcessingOverview

#analyticsx


PURPOSE-BUILT PROVIDES REAL-TIME ANSWERS

SAS Event Stream Processing

Processing high throughput, low latency streaming events requires moving from

(reactive) real-time to (proactive) real-time

Reactive real-time Proactive real-time

High

throughput

Low latency

Medium

throughput

and latency

Continuously analyze to define relevant

action

Real-time action occurs as the result of pattern

detection

SAS Real-Time Solutions

Listen and react to incoming requests

Real-time action occurs as the result of a triggering event

#analyticsx


Known Patterns

•Rule: Set up rules for filter fraudulent transactions

•Examples: two transactions in different time zones within short period of time

Unknown Patterns

•Anomaly Detection: Detect individual and aggregate abnormal patterns

•Examples: Mean , standard deviation, percentiles, univariate and multivariate regression, clustering, sequence analysis, peer group analysis

Complex Patterns

•Advanced Analytics: Perform knowledge discovery, data mining, predictive assessment

•Examples: Neural networks, decision trees, generalized linear models, econometric models, gradient boosting

Associative Link Patterns

•Social Network Analysis: Perform knowledge discovery through associative linkage analysis

•Examples: Social network + linkage analysis + community detection + advanced analytics

PATTERN DETECTIONPURPOSE-BUILT

#analyticsx


PURPOSE-BUILT IDENTIFIES PATTERNS OF INTEREST

One or more:

• Pre-built data quality routines

• Business rules and policy definitions

• Advanced analytics:

• Scoring events (models

developed on data at rest)

• Machine Learning clusters

(models defined in-stream)

• Extract entities, classify and

identify sentiment (NLP methods)

• Filter, aggregate and correlate events

• Reference historic data (store in-

memory)

• Continuous queries or periodic queries

• Pattern detection at event stream source

• Offline, data at rest identifies emerging trends

• Feed new insights back into event streams

• Dynamically update queries into live stream

#analyticsx


PURPOSE-BUILT FAST AND ADAPTIVE ACTION

SAS-generated Insights

Event Actions

SAS In-Memory

SAS®

Event Stream Processing Model

Continuous Query

Pu

bli

sh

Su

bs

cri

be

Streaming Events

Enrichment Data

Analytic Models

Business Rules

Pattern detection at event stream source

Offline, data at rest identifies emerging trends

Feed new insights back into event streams

Dynamically update queries into live stream

#analyticsx


PURPOSE-BUILT A SINGLE LEARNING ENVIRONMENT

Dataflow centric modeling

Drag & drop visual

modeler

Visual, XML or C modeling

Dynamic model update

Publish & Subscribe API

(Java, C, Python)

Model definition and maintenance, simplified with visual

modeling interfaceo Create and maintain streaming models easily for fast and flexible

adaptive actiono Full set of components to build any type of processo Incremental model testing

Easy deployment of streaming analytic models

o Deployment of existing analytic models using embedded SAS® DS2, SAS® Datastep or Python code

o Deploy ESP models as XML fileso Dynamic model updates

#analyticsx


PURPOSE-BUILT ENGINEERED FOR SPEED

Throughput - how many events per second can be ingested

Latency - the time it takes for an event to be processed through the defined workflow

• Millions of events per second throughput

• Millisecond-microsecond response latency

• On standard commodity hardware

Event Streams are high throughput, low latency data flows

SAS Event Stream Processing provides:

Continuous in-memory

processing

OS native application

Threaded pool - clustering

Linear scalability

Fastest ESP in the market

#analyticsx


PURPOSE-BUILT ENGINEERED FOR AGILITY

Lightweight embedding

technology

Cloud ready

OS native application

Clustering

Dynamic model update

Low footprint OS native application

From lightweight embedded technology to cloud

distributed architecture

Fulfill new architecture needs

Edge Small Large Cluster Cloud

#analyticsx


STREAMING ANALYTICS COMPLEMENTS SAS ANALYTICS

Real Time Alerts

VISUALIZATION, ALERTS

Continuous,Incremental Analytics

ADVANCED ANALYTICS

HIGH PERFORMANCE

ANALYTIC SOLUTIONS

Risk Analytics

Fraud Analytics

Asset Performance Analytics

Customer Intelligence

Decision Management

Visual Analytics

Etc …

STREAMING DATA ANALYSIS

EVENT STREAM

PROCESSING

ENGINE

Continuous processing of

events, in high-volume streams,

to detect actionable information

Rules

Correlation

BIG DATA STREAMS

BATCH DATA ANALYSIS

ACCESS

ENGINES

#analyticsx


IN-STREAM ANALYTICS PROCEDURAL WINDOW

Build analytical models using SAS® DS2

• Decision Tree

• Neural Network

• Regression

• Rule Induction

• Scoring

• And more

Or SAS DATAStep, C/C++, Python

• Coming soon: R

#analyticsx


ECOSYSTEM INTEGRATION 300+ ENDPOINTS

OPEN SOURCE

SYSTEMS & APPLICATIONS

PUBLISH & SUBSCRIBE API

CONNECT TO ANY SYSTEM WITH JAVA, C, PYTHON

FULLY DOCUMENTED AND EASY TO USE

RendezVous

STANDARDS

FILE/SOCKET

XML / JSON

ODBC

JMS

MQTT

SYSLOG

DB LOG SNIFFERS

HTTP RESTFUL

SMTP

NETWORK SNIFFERS

WEB SERVICES

* *

*

*


#analyticsx


SAS®

Event Stream Processing Use Cases

#analyticsx


REAL-TIME CONTEXTUAL DATA MINING

BUSINESS ISSUE

• Needed a solution for real-time campaign management

(anytime, anywhere market challenges)

• Current process was extremely manual and resulted in high-

latency reporting

• Lacked advanced analytical capabilities and was limited in

terms of real-time capabilities

RESULTS

• Contextual real-time analysis of the streaming call data

records (CDRs) - 20,000+ requests per second

• The right offer at the right time with the right channel

• Enhanced the accuracy of predictions and decisions

COMMUNICATIONS

#analyticsx


PATIENT MANAGEMENT

BUSINESS ISSUE

• Detect relevant patterns in patient real-time data to alert critical

care teams

• Address Alert Fatigue٭

• Patient vital statistics from various sensors across different

equipment

• Incoming lab results joined with real time sensor data

RESULTS

• Monitor data to trigger actions based upon detected patterns

• Send messages across email and SMS

• Alert immediately appropriate critical care teams

• Send immediate recommendation to remote patient

HEALTH CARE

#analyticsx


ONLINE FRAUD DETECTION BANKING

BUSINESS CHALLENGE

• Under attack from fraudsters

• Bank wanted to drastically quicken its reporting capabilities and be

able to move from an overnight to intraday reporting speed

• Lack of key elements in detection logic (e.g., beneficiary profiles,

PC session logs)

RESULTS

As a result of this delivery, the bank's market risk managers are now

able to:

• Access 10 times the volume of data on a given day

• Receive risk data four hours earlier than before

• Analyze large data sets of up to seven terabytes of granular risk

data on demand

• Access data through a central feature-rich, web-based portal

#analyticsx


SAS EVENT STREAM PROCESSING

KEY TAKEAWAYS

Detect and monitor

events continuously, taking real-time relevant

action for greatest impact

REAL-TIME

RELEVANT

ACTION

Retain only what’s appropriate, and filter

and cleanse before big data is stored

FOCUS ON

RELEVANT

DATA

One managed, easy to use

environment to examine, assess, action and improve

streaming analytics

GOVERNED

MULTI-PHASE

ANALYTICS

#analyticsx


Questions?Thank you for your time today!

[email protected]


#AnalyticsX

Documents

When Decisions Can’t Wait: From Analysis to Action …...OPERATIONALIZING ANALYTICS Move analysis to event source, Analyze before data is stored, Keep what is relevant. High-speed