Gyula Fóra

gyfora@apache.org

Flink committer

Swedish ICT

Real-time data processing

with Apache Flink

2

▪ Data stream: Infinite sequence of data arriving in a continuous

fashion.

▪ Stream processing: Analyzing and acting on real-time streaming

data, using continuous queries

Stream processing

3 Parts of a Streaming Infrastructure

3

Gathering Broker Analysis

Sensors

Transaction

logs …

Server Logs

4

Apache Storm

• True streaming over distributed dataflow

• Low level API (Bolts, Spouts) + Trident

Spark Streaming

• Stream processing emulated on top of batch system (non-native)

• Functional API (DStreams), restricted by batch runtime

Apache Samza

• True streaming built on top of Apache Kafka, state is first class citizen

• Slightly different stream notion, low level API

Apache Flink

• True streaming over stateful distributed dataflow

• Rich functional API exploiting streaming runtime; e.g. rich windowing semantics

Streaming landscape

What is Flink

5

Gelly

Table

ML

SA

MO

A

DataSet (Java/Scala/Python) DataStream (Java/Scala)

Hadoop

M/R

Local Remote Yarn Tez Embedded

Data

flow

Data

flow

MR

QL

Table

Ca

sca

din

g (

WiP

)Streaming dataflow runtime

Program compilation

6

case class Path (from: Long, to:Long)val tc = edges.iterate(10) {

paths: DataSet[Path] =>val next = paths

.join(edges)

.where("to")

.equalTo("from") {(path, edge) =>

Path(path.from, edge.to)}.union(paths).distinct()

next}

Optimizer

Type extraction

stack

Task

scheduling

Dataflow

metadata

Pre-flight (Client)

MasterWorkers

DataSourc

eorders.tbl

Filter

MapDataSourc

elineitem.tbl

JoinHybrid Hash

build

HTprobe

hash-part [0] hash-part [0]

GroupRed

sort

forward

Program

Dataflow

Graph

deploy

operators

track

intermediate

results

Flink Streaming

7

What is Flink Streaming

8

Native stream processor (low-latency)

Expressive functional API

Flexible operator state, stream windows

Exactly-once processing guarantees

Native vs non-native streaming

9

Stream

discretizer

Job Job Job Jobwhile (true) {// get next few records// issue batch computation

}

while (true) {// process next record

}

Long-standing operators

Non-native streaming

Native streaming

Pipelined stream processor

10

StreamingShuffle!

Defining windows in Flink

Trigger policy• When to trigger the computation on current window

Eviction policy• When data points should leave the window

• Defines window width/size

E.g., count-based policy• evict when #elements > n

• start a new window every n-th element

Built-in: Count, Time, Delta policies

11

Expressive APIs

12

case class Word (word: String, frequency: Int)

val lines: DataStream[String] = env.fromSocketStream(...)

lines.flatMap {line => line.split(" ")

.map(word => Word(word,1))}

.window(Time.of(5,SECONDS)).every(Time.of(1,SECONDS))

.groupBy("word").sum("frequency")

.print()

val lines: DataSet[String] = env.readTextFile(...)

lines.flatMap {line => line.split(" ")

.map(word => Word(word,1))}

.groupBy("word").sum("frequency")

.print()

DataSet API (batch):

DataStream API (streaming):

DataStream API

13

Overview of the API

Data stream sources• File system

• Message queue connectors

• Arbitrary source functionality

Stream transformations• Basic transformations: Map, Reduce, Filter, Aggregations…

• Binary stream transformations: CoMap, CoReduce…

• Windowing semantics: Policy based flexible windowing (Time, Count,

Delta…)

• Temporal binary stream operators: Joins, Crosses…

• Native support for iterations

Data stream outputs

For the details please refer to the programming guide:• http://flink.apache.org/docs/latest/streaming_guide.html

14

Reduce

Merge

Filter

Sum

Map

Src

Sink

Src

Use-case: Financial analytics

15

Reading from multiple inputs

• Merge stock data from various sources

Window aggregations

• Compute simple statistics over windows

Data driven windows

• Define arbitrary windowing semantics

Combine with sentiment analysis

• Enrich your analytics with social media feeds (Twitter)

Streaming joins

• Join multiple data streams

Detailed explanation and source code on our blog

• http://flink.apache.org/news/2015/02/09/streaming-example.html

Reading from multiple inputs

case class StockPrice(symbol : String, price : Double)

val env = StreamExecutionEnvironment.getExecutionEnvironment

val socketStockStream = env.socketTextStream("localhost", 9999)

.map(x => { val split = x.split(",")

StockPrice(split(0), split(1).toDouble) })

val SPX_Stream = env.addSource(generateStock("SPX")(10) _)

val FTSE_Stream = env.addSource(generateStock("FTSE")(20) _)

val stockStream = socketStockStream.merge(SPX_Stream, FTSE_STREAM)

16

(1)

(2)

(4)

(3)

(1)

(2)

(3)

(4)

"HDP, 23.8""HDP, 26.6"

StockPrice(SPX, 2113.9)

StockPrice(FTSE, 6931.7)StockPrice(SPX, 2113.9)StockPrice(FTSE, 6931.7)StockPrice(HDP, 23.8)StockPrice(HDP, 26.6)

Window aggregations

val windowedStream = stockStream

.window(Time.of(10, SECONDS)).every(Time.of(5, SECONDS))

val lowest = windowedStream.minBy("price")

val maxByStock = windowedStream.groupBy("symbol").maxBy("price")

val rollingMean = windowedStream.groupBy("symbol").mapWindow(mean _)

17

(1)

(2)

(4)

(3)

(1)

(2)

(4)

(3)

StockPrice(SPX, 2113.9)StockPrice(FTSE, 6931.7)StockPrice(HDP, 23.8)StockPrice(HDP, 26.6)

StockPrice(HDP, 23.8)

StockPrice(SPX, 2113.9)StockPrice(FTSE, 6931.7)StockPrice(HDP, 26.6)

StockPrice(SPX, 2113.9)StockPrice(FTSE, 6931.7)StockPrice(HDP, 25.2)

Data-driven windows

case class Count(symbol : String, count : Int)

val priceWarnings = stockStream.groupBy("symbol")

.window(Delta.of(0.05, priceChange, defaultPrice))

.mapWindow(sendWarning _)

val warningsPerStock = priceWarnings.map(Count(_, 1)) .groupBy("symbol")

.window(Time.of(30, SECONDS))

.sum("count")

18

(1)(2) (4)

(3)

(1)

(2)

(4)

(3)

StockPrice(SPX, 2113.9)StockPrice(FTSE, 6931.7)StockPrice(HDP, 23.8)StockPrice(HDP, 26.6)

Count(HDP, 1)StockPrice(HDP, 23.8)StockPrice(HDP, 26.6)

Combining with a Twitter stream

val tweetStream = env.addSource(generateTweets _)

val mentionedSymbols = tweetStream.flatMap(tweet => tweet.split(" "))

.map(_.toUpperCase())

.filter(symbols.contains(_))

val tweetsPerStock = mentionedSymbols.map(Count(_, 1)).groupBy("symbol")

.window(Time.of(30, SECONDS))

.sum("count")

19

"hdp is on the rise!""I wish I bought more YHOO and HDP stocks"

Count(HDP, 2)Count(YHOO, 1)(1)

(2)(4)

(3)

(1)

(2)

(4)

(3)

Streaming joins

val tweetsAndWarning = warningsPerStock.join(tweetsPerStock)

.onWindow(30, SECONDS)

.where("symbol")

.equalTo("symbol"){ (c1, c2) => (c1.count, c2.count) }

val rollingCorrelation = tweetsAndWarning

.window(Time.of(30, SECONDS))

.mapWindow(computeCorrelation _)

20

Count(HDP, 2)Count(YHOO, 1)

Count(HDP, 1)

(1,2)

(1) (2)

(1)

(2)

0.5

21

▪ Performance optimizations• Effective serialization due to strongly typed topologies

• Operator chaining (thread sharing/no serialization)

• Different automatic query optimizations

▪ Competitive performance

• ~ 1.5m events / sec / core

• As a comparison Storm promises ~ 1m tuples / sec /

node

Performance

Fault tolerance

22

Overview

23

▪ Fault tolerance in other systems

• Message tracking/acks (Apache Storm)

• RDD lineage tracking/recomputation

Fault tolerance in Apache Flink

• Based on consistent global snapshots

• Algorithm inspired by Chandy-Lamport

• Low runtime overhead, stateful exactly-

once semantics

Checkpointing / Recovery

24

Asynchronous Barrier Snapshotting for globally consistent checkpoints

Pushes checkpoint barriersthrough the data flow

Operator checkpointstarting

Checkpoint done

Data Stream

barrier

Before barrier =part of the snapshot

After barrier =Not in snapshot

Checkpoint done

checkpoint in progress

(backup till next snapshot)

State management

25

State declared in the operators is managed and

checkpointed by Flink

Pluggable backends for storing persistent

snapshots

• Currently: JobManager, FileSystem (HDFS, Tachyon)

State partitioning and flexible scaling in the

future

Closing

26

Streaming roadmap for 2015

State management

• New backends for state snapshotting

• Support for state partitioning and incremental

snapshots

• Master Failover

Improved monitoring

Integration with other Apache projects

• SAMOA, Zeppelin, Ignite

Streaming machine learning and other new

libraries27

flink.apache.org

@ApacheFlink