Real-time Analytics = Better Outcomes Powered by DataTorrent

Nathan Trueblood

VP Product Management

Why Fast Data Matters

Fast Data Analyticsleads to

Better Decision Makingleads to

Competitive Advantage

Real-time = You Win

Real-Time Isn’t Just for Wall Street & Amazon

Survey conducted by LightBend in June 2017 and represents a wide range of industries and company sizes. It is weighted toward the perspective of developers and architects.

Businesses Need To Create Order From Chaos

http://www.visualcapitalist.com/order-from-chaos-how-big-data-will-change-the-world/

Responsiveness Is Not the Only Factor

What is the cost to be competitive?

How can my business remain flexible?

Will I need an army of specialized devops ninjas?

Responsive

Cost-effective

Agile

Operable

Scalable

A Little Bit of History...

Innovation in IT is cyclical

Each cycle aims to balance the same challenges

1969

1989

2017

Centralized Distributed

Before Big Data

Responsive

Cost-effective

Agile

Operable

Scalable

LAMP Stack:

Linux, Apache, MySQL, PHP

Big Data At Rest

Responsive

Cost-effective

Agile

Operable

Scalable

Big Data Stack:

Linux, Apache, Hadoop

Fast Data … Done Right

Responsive

Cost-effective

Agile

Operable

Scalable

Fast Data “KASH” Stack:

Kafka, Apex, Spark, HDFS

Operability is Key to Successful Evolution

Software Operability: the properties of a system that make it work well in production

Each evolutionary step seeks:• More Scale• Lower Cost to Serve• Greater Agility

DEMO

Demo Overview

Omni-channel Fraud Prevention

Account Take-OverFraud Prevention

Independent EvolutionShared Outcome

Better ResultsIn Real-time

Outcom

es

• Two Independent Real-Time Applications• Account Take-Over Fraud Prevention• Payment Card Fraud Prevention

• Loose Integration• Backplane with operability

• Shared Outcomes• Real-time insight sharing leads to

more effective outcomes• Operability, agility and scalability are

preserved

Omni-Channel Payment Card Fraud Prevention

#123-239-292CCN: 1234 5678 90Florian Schmidt$100.02017/11/16Amazon US

data generatoror

real data sourcesdtFraud/transactions

Fraud Analysis

dtFraud/facts

dtFraud/trends

Online Account Take-Over Fraud Prevention

Account: 1234 5678 90Florian SchmidtActivity: Log On2017/11/16Dev IP Addr: 1.2.3.4Comcast US

data generatoror

real data sourcesdtFraud/logevents

Fraud Analysis

dtFraud/facts

dtFraud/trends

Outcome Sharing

data generatoror

real data sources

account activity events

Activity Analysis

Trends

Facts & Actions

data generatoror

real data sources

payment card transactions

Transaction Analysis

Trends

Facts & Actions

Overall FraudFacts & Actions

Overall FraudTrends

Step 1: Add Output Module to ATO Fraud App

Output connector module added to Account Take-Over (ATO) to share outcomes, schema via application backplane

Step 2: Add Input Module to Payment Fraud App

Input connector module is added to Payment Fraud App to subscribe to outcomes, schema via application backplane

Step 3: Example Payment Transaction

Normal transaction generated to show that purchase is approved

Step 4: ATO Rule in Payment Fraud App

CEP rule in Payment Fraud App flags transaction when suspicious account activity detected

Step 5: Confirm That Transaction is OK

Transaction processed as expected.

Step 6: Example Suspicious Account Activity

Suspicious account activity event is generated: Country of access is not the user’s registered country.

Step 7: Generate New Payment Transaction

New sample payment transaction is generated, after suspicious account activity.

Step 8: Confirm Payment Transaction Flagged

Payment transaction is flagged as suspicious, based on account activity.

SUMMARY

Real-Time ApplicationA successful real-time application combines multiple micro-data services to solve a business problem

● Combine real-time events and historical data sources

● Provide business and operational dashboards for production operations

● Integrate with existing, enterprise services (security, governance, etc)

Loose Integration

Analyze & Act

Train & Prepare

Ingest & Enrich

Archive & Persist

Successful Real-time Big Data ArchitectureClosing the loop between insight and action

Events

Historical Big Data at Rest

Legacy Enterprise Datastores

Big Data Storage

Responsive

Cost-effective

Agile

Operable

Scalable

Conclusions...

• Real-time is REAL• Key factors to consider• Innovation is cyclical• Operability is essential

• Loosely coupled data services with operability are the way to go

• To win, some of those services MUST be real-time

Resources:

• DataTorrent Big Data AppFactory - https://datatorrent.com/appfactory

• Download - https://datatorrent.com/download/

• Subscribe to forum - https://groups.google.com/forum/#!forum/dt-users

• Twitter @DataTorrent: Follow - https://twitter.com/datatorrent

• Meetups - http://meetup.com/topics/apache-apex

• Webinars - https://datatorrent.com/webinars/

• Videos - https://youtube.com/user/DataTorrent

• Slides - http://slideshare.net/DataTorrent/presentations

Q&A

Thank You!Please share your feedback.