Deploying & operating microservices

DEPLOYING AND OPERATING MICROSERVICESSam Newman ThoughtWorks London 2015

1

@samnewman#thoughtworks

Sam Newman

Building MicroservicesDESIGNING FINE-GRAINED SYSTEMS


Accounts

Returns

Invoicing

Shipping

Inventory

Customer Service


Accounts

Returns

Invoicing

Shipping

Inventory

Customer Service

Small Autonomous services that work together, modelled

around a business domain


Independently Releasable


Building


Building

Deploying


Building

Deploying

Managing


Building


Hands up if you’re doing Continuous Integration!


Hands up if you’re doing Continuous Integration!

Now keep them up if…


You check in to mainline at least once per day


You have a suite of tests to validate your changes


When the build is broken, it is your team’s #1 priority to fix it


Single Repo

ONE GIANT BUILD

Version Control

/Service A

/Service B

/Service C


Single Repo

ONE GIANT BUILD

Version Control

/Service A

/Service B

/Service C

CI Tool

Monolithic Build


Single Repo

ONE GIANT BUILD

Version Control

/Service A

/Service B

/Service C

CI Tool

Monolithic Build

Service B v123

Service A v123

Service C v123


Single Repo

ONE GIANT BUILD

Version Control

/Service A

/Service B

/Service C

CI Tool

Monolithic Build

Service B v123

Service A v123

Service C v123

One checkin = lots of services built


Single Repo

ONE GIANT BUILD

Version Control

/Service A

/Service B

/Service C

CI Tool

Monolithic Build

Service B v123

Service A v123

Service C v123

One checkin = lots of services built

Can’t independently release in a safe manner


Single Repo

ONE BUILD PER SERVICE, SINGLE REPO

Version Control CI Tool

/Service A

/Service B

/Service C


Single Repo



/Service A

/Service B

/Service C

Service A


Single Repo



/Service A

/Service B

/Service C

Service AService A

v123


Single Repo



/Service A

/Service B

/Service C

Service AService A

v123

Service B


Single Repo



/Service A

/Service B

/Service C

Service AService A

v123

Service B v456

Service B


Single Repo



/Service A

/Service B

/Service C

Service AService A

v123

Service B v456

Service B

Service C


Single Repo



/Service A

/Service B

/Service C

Service AService A

v123

Service B v456

Service C v789

Service B

Service C


Single Repo



/Service A

/Service B

/Service C

Service AService A

v123

Service B v456

Service C v789

Can build & release independently

Service B

Service C


Single Repo



/Service A

/Service B

/Service C

Service AService A

v123

Service B v456

Service C v789


Can encourage making cross-repo changes

Service B

Service C


ONE BUILD & REPO PER SERVICE


Service A

Service B v123

Service A v123

Service C v123

Service B

Service C

/Service A

/Service B

/Service C




Service A

Service B v123

Service A v123

Service C v123


Service B

Service C

/Service A

/Service B

/Service C




Service A

Service B v123

Service A v123

Service C v123


Firmer separation, but developer workflow can suffer

Service B

Service C

/Service A

/Service B

/Service C


Customer Service


Customer Service

S/M TestsBuild Large Tests


Customer Service



Customer Service



S/M TestsBuild Large Tests UAT ProdLarge Tests

Artifacts should be built once, and moved through environments









More Production Like




Faster Feedback

More Production Like



SO, ARTIFACTS….


SO, ARTIFACTS….

▫︎Bundles-o-stuff


SO, ARTIFACTS….


▫︎ Tech-stack specific


SO, ARTIFACTS….



▫︎OS-specific


SO, ARTIFACTS….



▫︎OS-specific

▫︎Custom Images


SO, ARTIFACTS….



▫︎OS-specific

▫︎Custom Images

▫︎Containers


Deploying

@samnewman#thoughtworks 18

Host

Service

Which is best?


Host

Service

Host

Service Service

Service Service

VS

Which is best?


Application Containers

OSGI

Actor-based Systems


Host

Service Service

Service Service

Application Containers

OSGI

Actor-based Systems


Host

Service

Linux Containers/Docker

Custom Images



Host

Service


Host

Service

Host

Service Service

Service Service

VS


Host

Service

Host

Service Service

Service Service

Much Easier To Reason About

VS


Host

Service

Host

Service Service

Service Service


Easier To Provision

VS


Host

Service

Host

Service Service

Service Service


Easier To Provision

Fewer Side-effects

VS


Host

Service

Host

Service Service

Service Service


Easier To Provision

Fewer Side-effectsLower host-management overhead

VS


Host

Service

Host

Service Service

Service Service


Easier To Provision

Fewer Side-effectsCheaper!

Lower host-management overhead

VS


Host

Service

Host

Service Service

Service Service


Easier To Provision

Fewer Side-effectsCheaper!

Lower host-management overhead

VS

Enforces Interdependence


Reducing Host Management Overhead?


API-driven Host Provisioning


VMWare

Virtualisation Platforms


VMWareAWS

Digital Ocean

Virtualisation Platforms


Step 0: Provision Host



Step 1: Configure Host



Step 2: Install Service




Step 2: Install Service




Ansible

Puppet

Chef


Ansible

Puppet

Chef


Ansible

Puppet

Chef

AWS


Ansible

Puppet

Chef

AWS

Digital Ocean


Ansible

Puppet

Chef

AWS

Digital Ocean

OpenStack


Ansible

Puppet

Chef

AWS

Digital Ocean

OpenStack

VMWare


Ansible

Puppet

Chef

AWS

Digital Ocean

OpenStack

VMWare

Vagrant


Ansible

Puppet

Chef

AWS

Digital Ocean

OpenStack

VMWare

Vagrant

Immutable Servers


Ansible

Puppet

Chef

AWS

Digital Ocean

OpenStack

VMWare

Vagrant

Immutable Servers

Fast Spin-up


Ansible

Puppet

Chef

AWS

Digital Ocean

OpenStack

VMWare

Vagrant

Immutable Servers

Fast Spin-up

Provider Agnostic


Ansible

Puppet

Chef

AWS

Digital Ocean

OpenStack

VMWare

Vagrant

Immutable Servers

Fast Spin-up

Provider Agnostic

Feedback Can Suffer


Ansible

Puppet

Chef

AWS

Digital Ocean

OpenStack

VMWare

Vagrant

Immutable Servers

Fast Spin-up

Provider Agnostic

Feedback Can Suffer

Cycle Time


Immutable Servers


Cost?


Machine

Base OS

Hypervisor

VM

OS

Apps

VM

OS

Apps

VM

OS

Apps


Machine

Base OS

Hypervisor

VM

OS

Apps

VM

OS

Apps

VM

OS

Apps

Expensive!




DOCKER


DOCKER

Machine


DOCKER

Machine

Base OS


DOCKER

Machine

Base OS

Docker


DOCKER

Machine

Base OS

Docker

Apps


DOCKER

Machine

Base OS

Docker

Apps Apps


DOCKER

Machine

Base OS

Docker

Apps Apps Apps


DOCKER

Machine

Base OS

Docker

Apps Apps Apps

Docker Image Registry


DOCKER

Machine

Base OS

Docker

Apps Apps Apps



DOCKER

Machine

Base OS

Docker

Apps Apps Apps





Semi-permanent Servers



Image-based Deployments



Image-based Deployments

Docker


S/M TestsBuild Large Tests UAT Prod



DB

Machine CI Node

Large Tests Environment

Large Tests



DB

MachineMachine



Master DB

Machine

Production Environment

Machine Machine Machine

Slave DB


ENVIRONMENTS


ENVIRONMENTS

▫︎Artifact remains the same…


ENVIRONMENTS


▫︎But the environment differs


ENVIRONMENTS


▫︎But the environment differs

▫︎How can we handle this?


Environment Definition


http://www.fig.sh/


https://www.terraform.io/


Operating


Shop

How Do We Find Things?

??? Inventory


HARDCODE IP!

Shop Inventory

inventory: 10.2.4.52


DNS

Shop Inventory

inventory: inventory.mycompany.com

DNS Service

http://inventory.mycompany.com


DNS

Shop Inventory


DNS Service



DNS

Shop Inventory


DNS Service

Configuring DNS?



DNS

Shop Inventory


DNS Service

Configuring DNS?

DNS propagation time?



SERVICE DISCOVERY TOOL

Shop

Service Discovery



Shop Inventory

Service Discovery



Shop Inventory

Service Discovery

I’m here!



Shop Inventory

Service Discovery

inventory: 10.1.1.10 I’m here!



Shop Inventory

Service Discovery




Shop

Service Discovery




Shop

Service Discovery

inventory: 10.1.1.10

Inventory

I’m here!



Shop

Service Discovery

Inventory







DNS SRV


DNS SRV REST API



Service Discovery


Service Discovery Configuration



Watches



Consul-templateWatches


Beware of writing your own service discovery system




Horizontal Duplication



Data Partitioning



Data Partitioning

Functional Decomposition


Functional decomposition = splitting microservices!


HORIZONTAL DUPLICATION - LOAD BALANCING





Client



Load Balancing

Client



Load Balancing

Client



Load Balancing

ClientServer or client load balancing



Load Balancing


Distribution algorithms can vary



Load Balancing


Distribution algorithms can vary

Software or hardware based


HORIZONTAL DUPLICATION - COMPETING CONSUMERS



Customer Servicecreate




customer created




customer created




customer created




customer created

Requires shared state




customer created

Requires shared state

Lots of variations on this ‘scatter gather’ approach


DATA PARTITIONING - SHARDING BY KEY

Customer Service Nodes




A-E




A-E F-M




A-E F-M N-R




A-E F-M N-R S-Z




A-E F-M N-R S-Z

Load Balancing

Client




A-E F-M N-R S-Z

Load Balancing

Client




A-E F-M N-R S-Z

Getting shard keys right is key - rebalancing is painful!

Load Balancing

Client




A-E F-M N-R S-Z

Getting shard keys right is key - rebalancing is painful!

Can combine with other scaling (e.g. load balancing)

Load Balancing

Client






AVOID THE DISTRIBUTED SINGLE POINT OF FAILURE!


Strangler App


Strangler App

Legacy App

Legacy AppLegacy App


Strangler App

Legacy App

Legacy App

Requests

Legacy App


Strangler App

Legacy App

Legacy App

Requests

Legacy App


Strangler App

Legacy App

Legacy App

Requests

Legacy App


Strangler App

Legacy App

Legacy App

Requests

Legacy App

30 - 60 Concurrent Requests


Strangler App

Legacy App

Legacy App

Requests

Legacy App


> 800 Concurrent Requests


Strangler App

Legacy App

Legacy App

Requests

Legacy App


> 800 Concurrent Requests


Strangler App

Legacy App Legacy App Legacy App

Thread Pool


Strangler App


Thread Pool

Failing…slowly!


Strangler App


Thread Pool

Failing…slowly!


Strangler App


Thread Pool

Failing…slowly!

Thread-pool exhausted


Strangler App


Thread Pool

Failing…slowly!


No requests to other downstream apps


Strangler App


Thread Pool

Failing…slowly!




Strangler App


Thread Pool

Failing…slowly!


Requests Building Up



Strangler App


Thread Pool


Strangler App


Fix Timeouts

Thread Pool


Strangler App


Fix Timeouts

Thread Pool


Strangler App


Fix Timeouts

Thread Pool Thread PoolThread Pool


Strangler App


Fix Timeouts

Thread Pool Thread PoolThread PoolBulkhead

Downstream Connections


Strangler App


Fix Timeouts




Strangler App


Fix Timeouts



Circuit Breakers



http://www.flickr.com/photos/kalexanderson/5421517469/








???


You have to get *much* better at monitoring






You are not a badass if you use an SSH Multiplexer




AGGREGATE METRICS


Response TimeResponse Time

Response Time

AGGREGATE METRICS



Response Time

AGGREGATE METRICS



Response Time

AGGREGATE METRICS


HEALTH CHECK PAGES


HEALTH CHECK PAGES


SYNTHETIC TRANSACTIONS





@samnewman#thoughtworkshttps://www.flickr.com/photos/yercombe/9033732392/


CORRELATION IDS


CORRELATION IDS


CORRELATION IDS


ID: 123CORRELATION IDS


ID: 123

ID: 123

ID: 123

CORRELATION IDS


Building


Building

Deploying


Building

Deploying

Managing


Sam Newman

Building MicroservicesDESIGNING FINE-GRAINED SYSTEMS

http://buildingmicroservices.com/

AUTHD

http://buildingmicroservices.com/


http://lanyrd.com/profile/samnewman/

http://lanyrd.com/profile/samnewman/

Sam Newman @samnewman

THANKS!

Technology

Deploying & operating microservices