Cloud Native Streaming and Event-Driven Microservices

Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /

Cloud Native Event-Driven and Streaming MicroservicesBy Marius Bogoevici, Staff Engineer, Pivotal

@mariusbogoevici


Safe Harbor Statement

• The following is intended to outline the general direction of Pivotal's offerings. It is intended for information purposes only and may not be incorporated into any contract. Any information regarding pre-release of Pivotal offerings, future updates or other planned modifications is subject to ongoing evaluation by Pivotal and is subject to change. This information is provided without warranty or any kind, express or implied, and is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions regarding Pivotal's offerings. These purchasing decisions should only be based on features currently available. The development, release, and timing of any features or functionality described for Pivotal's offerings in this presentation remain at the sole discretion of Pivotal. Pivotal has no obligation to update forward looking information in this presentation.

2


CLOUD-NATIVE STREAMING AND EVENT-DRIVEN MICROSERVICES

• Processing large quantities of data in real-time • Use cases:

• predictive maintenance, fraud detection, QoS, IoT • enterprise integration meets ‘Big Data’

• Specific requirements • High throughput, low latency • Grouping, ordering, windowing

• Transition in messaging middleware design: • Traditional (JMS-based, RabbitMQ) • To log-style: Kafka, Amazon Kinesis, Google Pub/Sub, Azure Event Hubs

3



• Resource management • memory, CPU, instance count

• Scaling up/down • Health monitoring and failover • Routing and load balancing

• Applications and deployment strategies designed to target of all above

4

ApacheYARNApacheMesos Kubernetes



• Typically known benefits and drawbacks of microservices:

• bounded context

• development agility

• added complexity

• Failure isolation

• Process-specific resource tuning:

• memory

• CPU

• instance count

• network

5



• Emphasize messaging as communication over HTTP

• Decoupling

• Discoverability

• Availability

• Eventual consistency across heterogenous resources

• alternative to distributed transactions

• Strong similarities between:

• data streaming - ‘big data’: ingestion, analytics

• event streaming - async interaction, event sourcing, etc

6


Spring Cloud Stream: The Origin Story

7

Spring Cloud Stream 2015

Spring XD Spring Cloud Data Flow

Spring Cloud Task


Spring Cloud Stream

• Event-driven microservice framework

• Built on battle-tested components

• Spring Boot: full-stack standalone applications

• Spring Integration: Messaging, EIP patterns, connectors

• Opinionated primitives

• Pluggable middleware abstractions

8


Spring Cloud Stream in a nutshell

9

Application Core

Spring Boot Spring Integration Spring Messaging Reactive APIs

inputs

outputs

Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 / 10



The Binder Abstraction

12

Application

Messaging Middleware Binder channel

can be input or output

adapts channel to target middleware

created by framework

configured via Spring Boot properties


Programming model

13

@EnableBinding + Binder Implementation

Apache Kafka

JMS Google PubSub

Production-ready:

Experimental


Programming model

14

@SpringBootApplication@EnableBinding(Processor.class) public class UppercaseProcessor { @StreamListener(Processor.INPUT) @SendTo(Processor.OUTPUT) public String process(String s) { return s.toUpperCase(); }}


Pipes and filters model

15

http averages topN

ingest data

average value over last 5s

hottest sensors over last 10 seconds


Spring Cloud Stream: Durable Publish Subscribe

16

http

raw-sensor-data

averages

averages

top-n Calculator


Spring Cloud Stream: Durable Publish Subscribe

17

http

raw-sensor-data

averages

averages

top-n Calculator

Failure detector

in Spring Cloud Stream, all destinations are pub-sub


Data and event streaming - conceptually similar

18


Spring Cloud Stream: Consumer groups

19

raw-sensor-data

averages

HDFShttp

pubsub is great for interconnecting logical apps

what if we add more instances ?



20

raw-sensor-data

averages

HDFShttp

we want these two to compete

averages

HDFSand these two !




22

Averages

HDDSHDFSHDFS

raw-sensor-datagroups are in a pub-sub relationship to each other

AveragesAverages

consumers are competing within a group



Spring Cloud Stream: Partitioning

24

Average calculator

Average calculator

http

raw-sensor-data



25

Average calculator

Average calculator

http

partition 1

partition 2

raw-sensor-data



26

Average calculator

Average calculator

http

partition 1

partition 2

raw-sensor-data


Simple topologies : (relatively) easy to deploy …

27

http

HDFS

spring.cloud.stream.bindings.output.destination=httphdfs.1

spring.cloud.stream.bindings.input.destination=httphdfs.1 spring.cloud.stream.bindings.input.group=httphdfs

httphdfs.1


… but how about complex topologies ?

28

http

raw-sensor-data

averages

top-n Calculator

Failure detector

averagesaverages

HDFSHDFSHDFS


Spring Cloud Data Flow

• Orchestration:

• DSL for Stream topologies

• REST API

• Shell

• UI

• Portable Deployment SPI

• OOTB apps for common integration use-cases

29


In a single sentence …

30

dataflow:> stream create demo --definition “http | file”


Spring Cloud Data Flow - Stream DSL

31

Stream definition

Boot Apps built on top of Spring Ecosystem

httpfile = http | file

|


Spring Cloud Data Flow - deployment SPI

• SPI for deploying applications

• Local (for testing)

• Cloud Foundry

• YARN

• Kubernetes

• Mesos + Marathon

• Different resources supported

• Spring Boot Uberjars

• Docker Images

• Different locations

• File system, HDFS, HTTP, Maven, Docker Hub

32


Spring Cloud Data Flow deployment

33

Pla$orm


Spring Cloud Stream 1.1

34


Spring Cloud Stream 1.1

• Under development, currently on master

• 1.1.0.M1 release August 2016

• 1.1.0.RELEASE - early Q4 2016

• Based on Spring Boot 1.4

• Release train model

• binders in separate repositories

• New features

• Kafka 0.9 support (also intending 0.10)

• Reactive programming support

• Schema Registry support

35


Programming model - imperative/procedural

36

@SpringBootApplication@EnableBinding(Processor.class) public class UppercaseProcessor { @StreamListener(“input”) @SendTo(“output”) public String process(String s) { return s.toUpperCase(); }}


Reactive/functional Programming via Reactor

37

@SpringBootApplication@EnableBinding(Processor.class) public class UppercaseProcessor { @StreamListener @Output(“output”) public Flux<String> process(@Input(“input”) Flux<String> s) { return f.map(s -> s.toUppercase()); }}



Handling multiple returns (or none at all)

39

@SpringBootApplication@EnableBinding(Processor.class) public class UppercaseProcessor { @StreamListener @Output(“output”) public Flux<String> process(@Input(“input”) Flux<String> s) { return f.filter(s->startsWith(“a”)).map(s -> s.toUppercase()); }}


Stateful operators: grouping, windowing

40

SpringBootApplication@EnableBinding(Processor.class) public class WordCountApplication { @StreamListener @Output("output") public Flux<?> countWords(@Input("input") Flux<String> words) { return words.window(ofSeconds(5), ofSeconds(1)) .flatMap(window -> window.groupBy(word -> word) .flatMap(group -> group.reduce(0, (counter, word) -> counter + 1) .map(count -> new WordCount(group.key(), count)))); } }


Content-type transformations (since 1.0)

• Users can specify the intended content type of an output value, e.g. “application/json”

• Outgoing messages are converted to that content type

• A message header value carries the content type

• The content type can be used by inputs to convert to expected types (e.g. `@StreamListener` method arguments

• Extensible model, just requires the registration of AbstractMessageConverter beans

41


Schema management and Avro support

• Newly introduced Avro MessageConverters

• Static schema-based resources (classpath, filesystem, etc)

• Schema Registry Client-based

• Schema registry server

• Not Spring Cloud Stream specific

• Allows storing different formats and versions

• Schema evolution support via Schema Registry Client converter

42


Heterogenous data sources/consumers

43

sensor v1

sensor v2

average-temperature v1

sensor data

average-pressure v2


Schema evolution in Spring Cloud Stream 1.1

44


Schema evolution in Spring Cloud Stream 1.1

45


Schema Registry Server - embeddable in apps

46

@SpringBootApplication @EnableSchemaRegistryServer public class SchemaRegistryServerApplication { public static void main(String[] args) { SpringApplication .run(SchemaRegistryServerApplication.class,args); } }


Pluggable schema registry client

47

@EnableBinding(Sink.class) @EnableAutoConfiguration@EnableSchemaRegistryClientpublic static class VoteCounterApplication { @StreamListener(Sink.INPUT) public void listen(Vote vote) { voteService.count(vote); } }


Future plans

48


Future plans

• Additional binders

• JMS

• Google Pub Sub

• Kinesis

• Fully reactive pipelines

• KStream support

• First class support for stateful processing

49


Fully reactive pipelines (planned)

• Use natively reactive middleware clients for reactive support

• e.g. Reactive Kafka client

50

public interface ReactiveProcessor {

@Input(Processor.INPUT) Flux<?> output(); @Output(Processor.OUTPUT) Flux<?> output(); }

@EnableBinding(ReactiveProcessor.class)


Kafka Stream Support (planned)

51

@StreamListener@Output(“output”)public KStream<Integer,String> addByKey(@Input(“input”) KStream<Integer,String> input) { return input.map((i,s) -> new KeyValue<>(i, s.toUpperCase())) .reduceByKey((String value1, String value2) -> value1 + value2, TimeWindows.of("windowName", 1000)) .toStream() }


Related talks

52

Reactive Kafka

Rajini Sivaram

Thursday, August 4, 9:00 AM - 10:05 AM

Orchestrate All the Things! with Spring Cloud Data Flow

Eric Bottard, Ilayaperumal Gopinathan

Thursday, August 4 11:10 AM - 12:15 PM


Learn More. Stay Connected.

Tell your friends!

<Related Session>

@springcentral spring.io/blog

@pivotal pivotal.io/blog

@pivotalcf http://engineering.pivotal.io

Technology

Cloud Native Streaming and Event-Driven Microservices