Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /

Building Resilient and Evolutionary Data Microservices

By Vinicius Carvalho@vccarvalhoviniciusccarvalho

Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /

Agenda• Data microservices• Why you should care about schema evolution• It’s all about the format• Introducing Schema evolution for Spring Cloud Stream• Demo

2

Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /

Data microservices

3

Self contained

Loosecoupled

EventDriven

Dataaware

Independent

Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /

Data microservices

4

Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /

Data pipelines

5

Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /

Data microservices in action: Context maps

6

Sam Newman’s Building Microservices

Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /

Data microservices in action: CQRS

7

MSDN CQRS MSDN CQRS Journey ebook

Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /

Learn more

8

Cloud Native Streaming and Event-Driven MicroservicesWed 8/3 - 4:20 - 5:30 PM Starvine 2

Orchestrate All the Things! with Spring Cloud Data FlowThu 8/4 - 11:10 - 12:15 Starvine 2

Architecting for Cloud Native Data: Data Microservices Done Right Using Spring CloudWed 8/3 2:00 - 3:10 PM Starvine 2

Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /

Evolutionary?

9

What happens when data evolve?

Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /

Schema Evolution

10

“The problem of evolving a data schema to adapt it to a change in modeled reality”

Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /

Backward compatibility

11

V1

V2

• A reader using a newer schema version can read from writers of an older version

Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /

Forward compatibility

12

V1

V2

• A reader using an older schema version can read from writers of a newer version

Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /

Request / Response paradigm

13

It’s all about a handshake

client<V1>

RecommendationService

GET /v1/…

<V2>Recommendation

Serviceclient

GET /v2/…

Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /

Messaging paradigm

14

<V1>Recommendation

Service

Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /

Messaging paradigm

15

<V1>Recommendation

Service

<V2>Recommendation

Service

if msg.headers[version] == “v1” {

}else if …

Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /

Data Microservices

16

But I thought microservices were all about smart endpointsand dumb pipes

Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /

Smart endpoints• Our goal is to make our endpoints

capable of adapting to changes on the model without requiring changes on the system.

17

Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /

How can we achieve that?

18

We have the tools We just need to evolve on how to use it

Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /

It all depends on the right format

19

Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /

Choosing the right format : Features

20

✓Compact✓Strongly typed✓Adaptable✓Versioned✓Centralized

Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /

Choosing a format

21

Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /

Payload size

22

0

350

700

1050

1400

Payload Size (bytes)

1355

763

502

285243216

AvroProto3KryoCSVJSONXML

Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /

Performance

23

Average marshaling time in nanos

Read Write

3,4833,433

1,3001,333

AvroJSON

Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /

Benchmark : because we love one

24

Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /

Bottom line

25

Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /

Why Avro• Binary and compact format• Mature and with active community. Heavily used on hadoop ecosystem• Schemas are a first class citizen on it• Out of the box support for Schema evolution• Support for GenericRecords : Easy integration with SpEL• Cross language implementations for clients

26

Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /

Schema evolution support on SCSt

27

API Level 16

API Level 21 Registry•Stores schemas•Tests compatibility•Tracks utilization

1. Producers registers schemas2. SCSt broadcast location in header

Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /

Schema evolution support on SCSt

28

API Level 16

API Level 21 Registry

1. Consumers fetch message2. Query registry for schema3. If version is different apply resolution

Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /

Starting the Server

29

@SpringBootApplication @EnableSchemaRegistryServer public class SchemaRegistryServerApplication { public static void main(String[] args) { SpringApplication .run(SchemaRegistryServerApplication.class,args); } }

Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /

Schema Registry Server

30

• POST / : Register a new schema, returning it’s version and a location header

• GET /{subject}/{format}/{version} : Returns a schema definition

• GET /{subject}/{format} : Returns all versions of a schema for a given format

• POST /{subject}/{format}/test : Compatibility test against all registered versions

• GET /{subject}/{format}/usage : List apps using all versions of a schema

Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /

Enabling Schema registration

31

@SpringBootApplication @EnableSchemaRegistryClient public class ProducerApplication { public static void main(String[] args) { SpringApplication .run(ProducerApplication.class,args); } }

Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /

Enabling Schema registration - Producer

32

spring: cloud: stream: schemaRegistryClient: endpoint: http://localhost:8990 bindings: output: destination: sensor-topic contentType: "application/*+avro"

Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /

Enabling Schema registration - Consumer

33

spring: cloud: stream: schemaRegistryClient: endpoint: http://localhost:8990 bindings: input: destination: sensor-topic

Content type is inferred from headers

Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /

Schema evolution example

34

#v1 { "namespace" : "io.igx.android", "type" : "record", "name" : "Sensor", "fields" : [ {"name":"id","type":"string"}, {"name":"temperature", "type":"float", "default":0.0}, {"name":"acceleration", "type":"float","default":0.0}, {"name":"velocity","type":"float","default":0.0}, {"name":"accelerometer","type":[ "null",{ "type":"array", "items":"float" } ]}, {"name":"magneticField","type":[ "null",{ "type":"array", "items":"float" } ]}, {"name":"orientation","type":[ "null",{ "type":"array", "items":"float" } ]}

] }

#v2 { "namespace" : "io.igx.android", "type" : "record", "name" : "Sensor", "fields" : [ {"name":"id","type":"string"}, {"name":"internalTemperature", "type":"float", "default":0.0, "aliases":["temperature"]}, {"name":"externalTemperature", "type":"float", "default":0.0}, {"name":"acceleration", "type":"float","default":0.0}, {"name":"velocity","type":"float","default":0.0}, {"name":"accelerometer","type":[ "null",{ "type":"array", "items":"float" } ]}, {"name":"magneticField","type":[ "null",{ "type":"array", "items":"float" } ]}, {"name":"orientation","type":[ "null",{ "type":"array", "items":"float" } ]} ]

}

Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /

Schema Support on Spring Cloud Stream

35

- DEMO -

Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /

References

36

• Martin Kleppmann Schema Evolution in avro, thrift and protobufers: https://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html

• http://dataintensive.net/ - Martin Kleppmann

• The CQRS Journey: https://msdn.microsoft.com/en-us/library/jj554200.aspx

• Oracle Datastore schema evolution : https://docs.oracle.com/cd/NOSQL/html/GettingStartedGuide/schemaevolution.html

• Building Microservices by Sam Newman: http://samnewman.io/books/building_microservices/

• Apache Avro: https://avro.apache.org/docs/1.7.7/gettingstartedjava.html

• https://github.com/viniciusccarvalho/schema-evolution-samples