Lightbend Fast Data Platform

Lightbend Fast Data Platform

A Technical Overview

A Bit of History:IT Services vs. Big Data

Big DataServices

Some Overlap: Concerns, Architecture

The Recent Past…

Microservices & Fast Data

Much More Overlap

The Future?

Drilling Down:Monoliths to Microservices

lightbend.com/reactive-microservices-architecture

http://lightbend.com/reactive-microservices-architecture

• Tangled responsibilities, lead to infrequent, “big-bang” deployments

• App lifetimes months to forever!

Monoliths

• Each does one thing, so must be message driven & asynchronous

• Updates easier, deployments frequent • App lifetimes: minutes! to forever

Microservices

• But be careful: • Message overhead is much better than

function calls!

Microservices

Reactive Systems

reactivemanifesto.org

http://reactivemanifesto.org

OpenSourceCoreForDevelopmentinScala/Java

Produc8onSuiteForOpera8ons/DevOps

Lightbend Reactive Platform

Drilling Down:Big Data

YARN

HDFS

MRjob#1

MRjob#2

Flume Sqoop

DBs

SlaveNode

DiskDiskDiskDiskDisk

NodeMgr

DataNode

Master

ResourceManager

NameNode

Hadoop

YARN

HDFS

MRjob#1

MRjob#2

Flume Sqoop

DBs

SlaveNode


NodeMgr

DataNode

Master

ResourceManager

NameNode

Hadoop 2013: Embrace Spark

Sparkjob#1

Sparkjob#2

YARN

HDFS

MRjob#1

MRjob#2

Flume Sqoop

DBs

SlaveNode


NodeMgr

DataNode

Master

ResourceManager

NameNode

Go beyond batch?

Sparkjob#1

Sparkjob#2

The Emergence of Fast Data:(Time Is Money)

Respond to change

IoT

20

Logs

Sockets

Storage

HDFS,S3,…

SQL/NoSQL ES

KafkaStreams

Streaming

Streams

SQL

Batch

…

Streaming

Flink

Fast Data and Microservices; Are they Converging?

• Each [stream app or μservice]: • does one responsibility • ingests unending [data or messages]

Synergies

22

Logs

Sockets

Storage

HDFS,S3,…

SQL/NoSQL ES

KafkaStreams

Streaming

Streams

SQL

Batch

…

Streaming

Flink

• Each [stream app or μservice] must: • operate asynchronously • offer never-ending service

Synergies

23

Logs

Sockets

Storage

HDFS,S3,…

SQL/NoSQL ES

KafkaStreams

Streaming

Streams

SQL

Batch

…

Streaming

Flink

• These architectures are converging: 1.Similar design problems 2.Data becomes dominant problem

Thesis

24

Logs

Sockets

Storage

HDFS,S3,…

SQL/NoSQL ES

KafkaStreams

Streaming

Streams

SQL

Batch

…

Streaming

Flink

Lightbend Fast Data Platform

1. An accelerated on ramp for building streaming data systems, data applications, and other microservices.

Value Three Ways

2. Best practices guidance for solving specific design problems:

- Sample apps - Documentation - Enablement services

Value Three Ways

3. Machine learning-based monitoring and management:

-Keep your systems resilient, scalable, and responsive with minimal user intervention.

Value Three Ways

Lightbend Fast Data Platform V1.0

Internet

Logs

Sockets

DC/OS:OnPremiseorCloud

FDP

KafkaConnect

Storage

HDFS,S3,…

SQL/NoSQL ES

KafkaStreams

CloudHosted

FDP.ai

ClusterAnalysis

MachineLearningMicroservices

ProducDonSuite

MachineLearning

ML

Streaming

Streams

SQL

Batch

…

Streaming

Flink

Intelligent

Man

agem

ent

…Co

nsoles


Internet

Logs

Sockets


FDP

KafkaConnect

Storage

HDFS,S3,…

SQL/NoSQL ES

KafkaStreams

CloudHosted

FDP.ai

ClusterAnalysis


ProducDonSuite

MachineLearning

ML

Streaming

Streams

SQL

Batch

…

Streaming

Flink

Intelligent

Man

agem

ent

…Co

nsoles


Internet

Logs

Sockets


FDP

KafkaConnect

Storage

HDFS,S3,…

SQL/NoSQL ES

KafkaStreams

CloudHosted

FDP.ai

ClusterAnalysis


ProducDonSuite

MachineLearning

ML

Streaming

Streams

SQL

Batch

…

Streaming

Flink

Intelligent

Man

agem

ent

…Co

nsoles


Internet

Logs

Sockets


FDP

KafkaConnect

Storage

HDFS,S3,…

SQL/NoSQL ES

KafkaStreams

CloudHosted

FDP.ai

ClusterAnalysis


ProducDonSuite

MachineLearning

ML

Streaming

Streams

SQL

Batch

…

Streaming

Flink

Intelligent

Man

agem

ent

…Co

nsoles

Service 1

Log & Other Files

Internet

Services

Service 2

Service 3

Services

Services

N * M links ConsumersProducers

Service 1

Log & Other Files

Internet

Services

Service 2

Service 3

Services

Services

N + M links ConsumersProducers


Internet

Logs

Sockets


FDP

KafkaConnect

Storage

HDFS,S3,…

SQL/NoSQL ES

KafkaStreams

CloudHosted

FDP.ai

ClusterAnalysis


ProducDonSuite

MachineLearning

ML

Streaming

Streams

SQL

Batch

…

Streaming

Flink

Intelligent

Man

agem

ent

…Co

nsoles


Internet

Logs

Sockets


FDP

KafkaConnect

Storage

HDFS,S3,…

SQL/NoSQL ES

KafkaStreams

CloudHosted

FDP.ai

ClusterAnalysis


ProducDonSuite

MachineLearning

ML

Streaming

Streams

SQL

Batch

…

Streaming

Flink

Intelligent

Man

agem

ent

…Co

nsoles


Internet

Logs

Sockets


FDP

KafkaConnect

Storage

HDFS,S3,…

SQL/NoSQL ES

KafkaStreams

CloudHosted

FDP.ai

ClusterAnalysis


ProducDonSuite

MachineLearning

ML

Streaming

Streams

SQL

Batch

…

Streaming

Flink

Intelligent

Man

agem

ent

…Co

nsoles

• Low latency? How low? • High volume? How high?

Streaming Tradeoffs (1/3)

38

• Which kinds of data processing & analytics are required?

• How will this processing be done? • Individual processing of events? • Bulk processing of records?


39

• Which tools and data sources/sinks must interoperate with your streaming tool?


40

41

42

• Low latency • Low volume • Complex flows • Complex Event

Processing

43

• Med. latency • High volume • Data flows, SQL • En masse

processing

44

• Low latency • High volume • Data flows,

correctness • En masse

processing

45

• Low latency • Med. volume

• ETL, “tables” • Data flow or

per event


Internet

Logs

Sockets


FDP

KafkaConnect

Storage

HDFS,S3,…

SQL/NoSQL ES

KafkaStreams

CloudHosted

FDP.ai

ClusterAnalysis


ProducDonSuite

MachineLearning

ML

Streaming

Streams

SQL

Batch

…

Streaming

Flink

Intelligent

Man

agem

ent

…Co

nsoles


Internet

Logs

Sockets


FDP

KafkaConnect

Storage

HDFS,S3,…

SQL/NoSQL ES

KafkaStreams

CloudHosted

FDP.ai

ClusterAnalysis


ProducDonSuite

MachineLearning

ML

Streaming

Streams

SQL

Batch

…

Streaming

Flink

Intelligent

Man

agem

ent

…Co

nsoles

Kafka & Spark Metrics

Correlated Troubleshooting


Internet

Logs

Sockets


FDP

KafkaConnect

Storage

HDFS,S3,…

SQL/NoSQL ES

KafkaStreams

CloudHosted

FDP.ai

ClusterAnalysis


ProducDonSuite

MachineLearning

ML

Streaming

Streams

SQL

Batch

…

Streaming

Flink

Intelligent

Man

agem

ent

…Co

nsoles

• Only metrics are uploaded, not sensitive domain data.

• One less service for you to manage. • …

Why Only Hosted Services?

• … • We can rapidly evolve this service.

without impacting your environment. • You benefit from aggregated knowledge

from all FDP clusters.

Why Only Hosted Services?


Internet

Logs

Sockets


FDP

KafkaConnect

Storage

HDFS,S3,…

SQL/NoSQL ES

KafkaStreams

CloudHosted

FDP.ai

ClusterAnalysis


ProducDonSuite

MachineLearning

ML

Streaming

Streams

SQL

Batch

…

Streaming

Flink

Intelligent

Man

agem

ent

…Co

nsoles

• Alpha: Jan 9 • Beta 1: April • Beta 2: July • V1.0: September

Timeline

54


Internet

Logs

Sockets

DC/OS,MarathonOnPremiseorCloud

FDP

KafkaConnect

Storage

HDFS,S3,…

SQL/NoSQL ES

KafkaStreams

CloudHosted

GravitonClusterAnalysis


ProducConSuite

MachineLearning

ML

Streaming

Streams

SQL

Batch

…

Streaming

Flink

Intelligent

Man

agem

ent

…Co

nsoles

For More Information:

lightbend.com/fast-data-platform

http://lightbend.com/fast-data-platform

For More Information

lightbend.com/ reactive-microservices-architecture

bit.ly/lightbend-fast-data

For More Information

• FDP: lightbend.com/fast-data-platform • Book: bit.ly/lightbend-fast-data

http://lightbend.com/fast-data-platform

http://bit.ly/lightbend-fast-data

Technology

Lightbend Fast Data Platform