© 2013 SpringOne 2GX. All rights reserved. Do not distribute without permission.

Real Time Event Processing and Decision Making

By Lyndon Adams, Sr. Architect Pivotal Real Time

Introduction

What does ‘Real Time’ mean? Applications are getting ‘realer’ and ‘timelier’

Where do we see real time use cases? Banking, Telco, Sensors, Smart Grids, Intelligence and others

How can I make real time decisions in response to events? Route events to servers based on data affinity

What is Real Time? re·al Adjective /ˈrē(əә)l/ Actually existing as a thing or occurring in fact; not imagined or supposed

Julius Caesar was a real person Used to emphasize the significance or seriousness of a situation or circumstance

there is a real danger of civil war the competitive threat from overseas is very real

Relating to something as it is, not merely as it may be described or distinguished (of a substance or thing) Not imitation or artificial; genuine

the earring was presumably real gold True or actual

his real name is James this isn't my real reason for coming

(of a person or thing) Rightly so called; proper he's my idea of a real man Jamie is my only real friend

Complete; utter (used for emphasis) the tour turned out to be a real disaster

Adjusted for changes in the value of money; assessed by purchasing power real incomes had fallen by 30 percent an increase in real terms of 11.6 percent

Of fixed property (i.e., land and buildings), as distinct from personal property he lost nearly all of his real holdings

(of a number or quantity) Having no imaginary part (of an image) Of a kind in which the light that forms it actually passes through it; not virtual

All related information needs to be brought together in one place

“How would you know the difference between the dream world and the real world?”

Morpheus, The Matrix

Applications with consequence need accurate and up-to-date information

Copies of data, caches, eventual consistency and batch windows can all lead to errors

What is Real Time? time Noun /tīm/ The indefinite continued progress of existence and events in the past, present, and future regarded as a whole

one of the greatest wits of all time The progress of this as affecting people and things

things were getting better as time passed Time or an amount of time as reckoned by a conventional standard

it's eight o'clock Eastern Standard Time A point of time as measured in hours and minutes past midnight or noon

the time is 9:30 A moment or definite portion of time allotted, used, or suitable for a purpose

the scheduled departure time should we set a time for the meeting?

An indefinite period traveling always distorts one's feelings for a time

A more or less definite portion of time in history or characterized by particular events or circumstances

Victorian times the park is beautiful at this time of year

The conditions of life during a particular period times have changed

The appropriate or expected time for something, in particular childbirth or death he seemed old before his time

…

“Time is an illusion. Lunchtime, doubly so” Ford Prefect, The Hitchhikers Guide to the Galaxy

Appropriateness is situational, defined by the user

Each use may have its own definition of relevant time

Past, current and predictions of the future may all be relevant

Concurrent events and circumstances can affect the relevant ‘portion’ of time

Pivotal Real Time Intelligence

Pivotal RTI is designed to provide low latency, high throughput, elastic stream processing for applications with a very strong dependence on highly volatile reference data. Pivotal RTI includes fault tolerant and distributed data management for the volatile data it uses.

Thro

ughp

ut

Kafka

Pivotal RTI Rabbit MQ

Storm

Map-Reduce Traditional Client-Server

SpringXD

Rules/CEP Engines

CPU Optimized Data Optimized

ETL

Use Cases

Real Time Pricing and Risk

Market Data

Trades Position

Price Bids

Offers

Threshold Detected

+ Market Volatility isn’t a problem … it’s an

opportunity.

Monte Carlo Simulation

Correction

43.7B records

Historical Data Big Active Data Fast

Customer Behavior

Data Fabric Fast Big Data Archive

MicroCube

‘Customer Habits’

‘Customer Options’

Location Event Targeted Marketing

Periodic Refresh

Location Record

Data Analytics Big

Sensor Data Processing

Big Data

Fast Data 000’s data points/sec

Emergency Shutdown Tuning & Optimization,

Monitor & Control

Continuous Real-time Analysis

Maintenance Billing

Smart Grids, Smart Cites

Phase Variance

Weather Data

Conductor Temp

Insulator Temp

Intrusion Detection1

Conductor Tension

Footing Torque

Shaft Torque

Bearing Temp Windings Temp

Input Pressure

Exhaust Pressure

kVA Output Output Freq.

Coolant Temp

Supply kVA

Demand kVA

Stadium/Convention Schedule

Weather Data Tower Occupancy

Public Transport Demand

Holidays/Events

Emergency Operations

Big Data

Fast Data

Local Sunshine Residential

Consumption Vehicle Charging

Electricity must be consumed in the instant that it is

produced.

Asset Management and Intelligence

Big Data

Fast Data

Events Details

Analysis

Complex Decisions Require More & Real-time Data

Ring My Current Phone

Send to Voicemail

Incoming Call What time is it (locally)?

Am I travelling?

Am I in a meeting?

Is it family calling?

Is call marked ‘urgent’?

Have I interacted w/ phone recently? Am I driving? Am I on

vacation? Do I know the

caller? Who am I talking to?

What phone am I using?

There isn’t time to go to disk for this information.

This information is individual and changing all the time.

Solution

Data Optimized Applications

Customers A-D

Customers E-J

Customers K-O

App A

App A

App B

App C

Customers P-Z

App A

App B

App C

App A

App B

App C

App A

App B

App C

App A

App B

App C

Database w/ all data

Stateless Client App Scale Stateless App w/ more instances Additional Stateless Apps

‘Pivot’ the Architecture

Partition the Data into memory Deploy Apps to all nodes

Events jump nodes only when necessary

Database and network contention limit scale/performance

Caches provide limited value if: •  The data sets are big •  The data changes

Pivotal RTI with Pivotal One

+ +

Widgets

PaaS

Big Data

Fast Data

App Dev

Spring Integration Adapters REST & RSS

JMS & AMQP Mail

Twitter

TCP & UDP

Web Services

File

FTP & SFTP

Byte Streams Tuple, POJO, Message

Real Time Processing

Acknowledgement

Raw Stream Event

Validate Enrich Archive

Widget

Eviction

Sliding Window

Related Streams

Threshold Detection

Merge Business Event

Filter

Widgets

Set Arithmetic

Geo-Spatial

Threshold

Accumulators

Analytic

Real Time Analysis Linear Regression Example

Big Data Analyitics

Ingestion Complex Events Operations

parameters & noise

event data

periodic update

Predict y Is this normal? Is this fraud?

take action

Geo-Spatial Widgets

Geo Heat Maps

Enter/Leave Zone

Distance to POI

Friends in Area

Debouncing

Composition of Applications

/Network Business Unit

/ Root

/myCompany Business Unit

/CDR Ingestion SpringXD App

/Marketing Business Unit

/DroppedCalls

/CustomerRetention Spring Integration App

/DroppedCallThreshold Spring Integration App

/ExcessiveDroppedCalls

Network Events

Shared Channel

/Sales Business Unit

/ContractLIfecycle Spring Integration App

/ExpringContracts

Shared Container

Shared Container

Issue Rebate

Reference: /myCompany/Network/CDRIngestion/DroppedCalls

Demo

Real time dash board •  Streaming UI for call monitoring.

•  Stack •  Spring Integration use case

flows. •  GeoSpatial service and event

set processing. •  OpenLayers, HighCharts and

jqwidgets components. •  Tomcat and GemFire

embedded client. •  Websockets.

Dropped Calls •  Geographic dropped call locations. •  Uses streaming dropped calls feed. •  OpenLayers functionality utilized for

proximity dropped call count. –  Rules used for color coding. –  Radius set to 35 miles.

•  System uses EPSG:4326 encoding standard then we project to map coordinate system.

Client contract alerts •  Alerts generated for a contract coming up for renewal that

has experienced dropped calls in real time. •  Intersect of client Id from contracts and dropped calls sets. •  Uses declarative predicates to filter data.

•  DroppedCall à dropped ? contractId : null; •  Contract à underreview and expired ? Id : null;

•  Grid displays all dropped calls for filtered client. •  Grid provided grouping function etc.

Aggregated dropped calls by region •  Call records are tagged with region location using a geospatial location

service. •  UK regions are specified upfront via a set of uk region polygons. •  Geospatial processes occurs for each call to determine geographic region. •  Geo region count managed by cluster.

High level architecture

Client

Contracts

Calls droppedCalls

reviewClients

GF PR

GF PR (colo)

Web

geoZonedDroppedCalls

filter(contract)

filter(dropped calls)

Set Widget

Geo Widget

Feed Storage Processing Results

Conclusion

Pivotal Real Time Intelligence

Some stream processing cases are CPU intensive.

Other cases benefit from colocation with volatile data.

Spring Integration and SpringXD provide support for elastic Staged Event Driven Architectures for processing high volume event streams.

runs Spring process flows inside of a managed data environment with new cluster-aware widgets

? ?

? ?

Learn More. Stay Connected.

•  Email ladams@gopivotal.com

•  Talk to us on Twitter: @springcentral •  Find Session replays on YouTube: spring.io/video