Embed Size (px)
DESCRIPTION
What do deployment pipelines look like when your system consists of 10s of different types of services? How do you know what to test before deployment? Should you release a service at a time, or bunch them up? This talk goes into the nitty gritty of managing build,test and release of micro services and also covers the often ignored tradeoff between testing before deployment, and testing afterwards. Presented at GeeCon 2014
Citation preview
Testing & Deploying Microservices
Sam Newman
@samnewman#geecon2
@samnewman#geecon4
Sam Newman
Building MicroservicesDESIGNING FINE-GRAINED SYSTEMS
@samnewman#geecon5
@samnewman#geecon5
Web Shop
Shopping Cart
Catalog
Registration
Customer
Finance
@samnewman#geecon5
Web Shop
Shopping Cart
Catalog
Registration
Customer
Finance
1000 lines of code or less
@samnewman#geecon6
@samnewman#geecon6
@samnewman#geecon7
V1
@samnewman#geecon7
V2
@samnewman#geecon8
Go
Java
NodeJS
Ruby
@samnewman#geecon8
Go
NodeJS
Ruby
Clojure!
@samnewman#geecon9
@samnewman#geecon9
@samnewman#geecon10
Accounts
Returns
Invoicing
Shipping
Inventory
Customer Service
@samnewman#geecon10
Accounts
Returns
Invoicing
Shipping
Inventory
Customer Service
@samnewman#geecon11
@samnewman#geecon11
@samnewman#geecon11
DB
@samnewman#geecon12
Unit
Service
UI
MIKE COHN’S TEST PYRAMID
@samnewman#geecon12
Unit
Service
UI
MIKE COHN’S TEST PYRAMID
Increasing Scope
More Confidence
@samnewman#geecon12
Unit
Service
UI
MIKE COHN’S TEST PYRAMID
Increasing Scope
More Confidence
Faster!
Better Isolation
@samnewman#geecon12
Unit
Service
UI
MIKE COHN’S TEST PYRAMID
???
???
Increasing Scope
More Confidence
Faster!
Better Isolation
???
@samnewman#geecon13
Small
Medium
Large
GOOGLE’S TEST PYRAMID
Increasing Scope
More Confidence
Faster!
Better Isolation
@samnewman#geecon14
DB
Small
Large
Medium
@samnewman#geecon14
DB
Small
Large
Medium
@samnewman#geecon15
DB
Small
Large
Medium
@samnewman#geecon15
DB
Small
Large
Medium
@samnewman#geecon16
DB
Small
Large
Medium
@samnewman#geecon16
DB
Small
Large
Medium
@samnewman#geecon16
DB
Small
Large
Medium
@samnewman#geecon1717
Small
Medium
Large
TEST SNOWCONE
@samnewman#geecon18
Small
Medium
Large
@samnewman#geecon19
SmallMedium
LargeBuild
TestsTests
Source Control
…
BUILD PIPELINE
@samnewman#geecon19
SmallMedium
LargeBuild
TestsTests
Source Control
Faster Feedback
…
BUILD PIPELINE
@samnewman#geecon19
SmallMedium
LargeBuild
TestsTests
Source Control
Faster Feedback
Binary Artifact(s)
…
BUILD PIPELINE
@samnewman#geecon19
SmallMedium
LargeBuild
TestsTests
Source Control
Faster Feedback
Binary Artifact(s)
…
BUILD PIPELINE
@samnewman#geecon20
DB
@samnewman#geecon20
DB
@samnewman#geecon21
S/M TestsBuild Large Tests UAT Prod
@samnewman#geecon21
S/M TestsBuild Large Tests UAT Prod
DB
Machine CI Node
Large Tests Environment
Large Tests
@samnewman#geecon22
S/M TestsBuild Large Tests UAT Prod
DB
Machine
UAT Environment
Machine
@samnewman#geecon23
S/M TestsBuild Large Tests UAT Prod
Master DB
Machine
Production Environment
Machine Machine Machine
Slave DB
@samnewman#geecon24
S/M TestsBuild Large Tests UAT Prod
More Production Like
@samnewman#geecon24
S/M TestsBuild Large Tests UAT ProdLarge Tests
More Production Like
@samnewman#geecon24
S/M TestsBuild Large Tests UAT ProdLarge Tests
Faster Feedback
More Production Like
@samnewman#geecon25
@samnewman#geecon26
Customer Service
@samnewman#geecon26
Customer Service
S/M TestsBuild Large Tests
@samnewman#geecon26
Customer Service
S/M TestsBuild Large Tests
@samnewman#geecon26
Customer Service
S/M TestsBuild Large Tests
@samnewman#geecon27
Customer Service
Web Shop
Fulfilment Service
@samnewman#geecon27
Customer Service
Web Shop
Fulfilment Service
Large
Medium
Small
@samnewman#geecon28
Customer Service
Fulfilment Service
Large
Medium
Small
@samnewman#geecon28
Customer Service
Fulfilment Service
Large
Medium
Small
@samnewman#geecon28
Customer Service
Large
Medium
Small
Fulfilment Service
Stub
@samnewman#geecon30
mountebank
:2525
@samnewman#geecon30
mountebank
:2525
@samnewman#geecon30
mountebank
:2525
:5555
@samnewman#geecon30
mountebank
:2525
:5555
Customer Service
@samnewman#geecon31
S/M TestsBuild Large Tests
@samnewman#geecon31
S/M TestsBuild Large Tests
Customer Service
Fulfilment Service
Stub
@samnewman#geecon32
Customer Service
V1
Web Shop
Fulfilment Service
@samnewman#geecon32
Web Shop
Customer Service
v2
Fulfilment Service
@samnewman#geecon32
Web Shop
Customer Service
v2
Fulfilment Service
@samnewman#geecon32
Web Shop
Customer Service
v2
Fulfilment Service
@samnewman#geecon33
S/M TestsBuild Large TestsCustomer
Service
Customer Service
v1
Web Shop v1
Production
@samnewman#geecon33
S/M TestsBuild Large Tests Integration TestCustomer
Service
Customer Service
v1
Web Shop v1
Production
@samnewman#geecon33
S/M TestsBuild Large Tests Integration TestCustomer
Service
Customer Service
v1
Web Shop v1
Production
Customer Service
v2
Web Shop v1
Integration Test
@samnewman#geecon34
Customer Service
v1
Web Shop v1
Production
@samnewman#geecon34
Customer Service
v1
Web Shop v1
Production
S/M TestsBuild Large Tests Integration TestCustomer
Service
Customer Service
v2
@samnewman#geecon34
S/M TestsBuild Large TestsWeb Shop
Customer Service
v1
Web Shop v1
Production
S/M TestsBuild Large Tests Integration TestCustomer
Service
Customer Service
v2
@samnewman#geecon34
S/M TestsBuild Large TestsWeb Shop
Customer Service
v1
Web Shop v1
Production
S/M TestsBuild Large Tests Integration TestCustomer
Service
Customer Service
v2
Web Shop v2
@samnewman#geecon34
S/M TestsBuild Large TestsWeb Shop
Customer Service
v1
Web Shop v1
Production
S/M TestsBuild Large Tests Integration TestCustomer
Service
Customer Service
v2
Web Shop v2
???
@samnewman#geecon35
S/M TestsBuild Large TestsWeb Shop
S/M TestsBuild Large TestsCustomer
Service
@samnewman#geecon35
S/M TestsBuild Large TestsWeb Shop
S/M TestsBuild Large TestsCustomer
Service
Integration Test
@samnewman#geecon35
S/M TestsBuild Large TestsWeb Shop
S/M TestsBuild Large TestsCustomer
Service
Integration Test
S/M TestsBuild Large TestsInvoice Service
@samnewman#geecon35
S/M TestsBuild Large TestsWeb Shop
S/M TestsBuild Large TestsCustomer
Service
Integration Test
S/M TestsBuild Large TestsInvoice Service
S/M TestsBuild Large TestsBasket
@samnewman#geecon35
S/M TestsBuild Large TestsWeb Shop
S/M TestsBuild Large TestsCustomer
Service
Integration Test
S/M TestsBuild Large TestsInvoice Service
S/M TestsBuild Large TestsBasket
S/M TestsBuild Large TestsFulfilment
@samnewman#geecon36
@samnewman#geecon36
Browsers
@samnewman#geecon36
Timing
Browsers
@samnewman#geecon36
Provisioning of Environments
Timing
Browsers
@samnewman#geecon36
Provisioning of Environments
Networks
Timing
Browsers
@samnewman#geecon36
Deployment
Provisioning of Environments
Networks
Timing
Browsers
@samnewman#geecon36
Deployment
Provisioning of Environments
Networks
Timing
Browsers Diagnosis
@samnewman#geecon37
@samnewman#geecon37
Integration Test
@samnewman#geecon37
Integration Test Prod…
@samnewman#geecon37
Integration Test Prod…
@samnewman#geecon37
Integration Test Prod…
@samnewman#geecon37
Integration Test Prod…
@samnewman#geecon38
@samnewman#geecon38
@samnewman#geecon39
John Allspaw: “Ops Metametrics” http://slidesha.re/dsSZIr
@samnewman#geecon39
John Allspaw: “Ops Metametrics” http://slidesha.re/dsSZIr
@samnewman#geecon40
Integration Test Prod…
v1v2
v6v4 = v10
@samnewman#geecon40
Integration Test Prod…
v1v2
v6v4 = v10
@samnewman#geecon41
Danger Will Robinson!
@samnewman#geecon42
Show Tangle
@samnewman#geecon43
Golden Rule: Get good at releasing services independently
@samnewman#geecon44
SO NO INTEGRATION TESTS?
@samnewman#geecon45
@samnewman#geecon45
@samnewman#geecon45
SEMANTIC MONITORING
@samnewman#geecon46
Customer ServiceWeb Shop
@samnewman#geecon46
Customer ServiceWeb Shop
Small
Medium
Large
@samnewman#geecon46
Customer ServiceWeb Shop
Small
Medium
Large
@samnewman#geecon46
Customer ServiceWeb Shop
Small
Medium
LargeConsumer Driven Contracts
@samnewman#geecon47
Customer ServiceWeb Shop
@samnewman#geecon47
Customer ServiceWeb Shop
Expectations
@samnewman#geecon47
Customer ServiceWeb Shop
Expectations
@samnewman#geecon47
Customer ServiceWeb Shop
Expectations
Prod
@samnewman#geecon47
Customer ServiceWeb Shop
Expectations
Prod
@samnewman#geecon48
@samnewman#geecon48
https://github.com/realestate-com-au/pact
@samnewman#geecon49
Prod
Prod
Prod
Prod
@samnewman#geecon49
Prod
Prod
Prod
Prod
@samnewman#geecon49
Prod
Prod
Prod
Prod
QA
@samnewman#geecon49
Prod
Prod
Prod
Prod
QA
Good Monitoring
@samnewman#geecon49
Prod
Prod
Prod
Prod
QA
Good Monitoring
Fast Remediation
@samnewman#geecon49
Prod
Prod
Prod
Prod
QA
Good Monitoring
Fast Remediation
@samnewman#geecon50
S/M TestsBuild Large Tests UAT ProdLarge Tests
Faster Feedback
More Production Like
@samnewman#geecon50
S/M TestsBuild Large Tests UAT ProdLarge Tests
Faster Feedback
More Production Like
@samnewman#geecon50
S/M TestsBuild Large Tests UAT ProdLarge Tests
Faster Feedback
More Production Like
@samnewman#geecon51
DB
Machine CI Node
Large Tests Environment
S/M TestsBuild Large Tests UAT ProdLarge TestsLarge Tests
@samnewman#geecon51
DB
Machine CI Node
Large Tests Environment
DB
Machine
UAT Environment
Machine
S/M TestsBuild Large Tests UAT ProdLarge Tests UAT
@samnewman#geecon51
DB
Machine CI Node
Large Tests Environment
DB
Machine
UAT Environment
Machine
Master DB
Machine
Production Environment
Machine Machine Machine
Slave DB
S/M TestsBuild Large Tests UAT ProdLarge Tests Prod
@samnewman#geecon52
Faster Feedback
More Production Like
S/M TestsBuild Large Tests UAT ProdLarge Tests
@samnewman#geecon52
Faster Feedback
More Production Like
S/M TestsBuild Large Tests UAT ProdLarge Tests
@samnewman#geecon53
@samnewman#geecon54
@samnewman#geecon55
Ansible
Puppet
Chef
@samnewman#geecon55
Ansible
Puppet
Chef
@samnewman#geecon55
Ansible
Puppet
Chef
AWS
@samnewman#geecon55
Ansible
Puppet
Chef
AWS
Digital Ocean
@samnewman#geecon55
Ansible
Puppet
Chef
AWS
Digital Ocean
OpenStack
@samnewman#geecon55
Ansible
Puppet
Chef
AWS
Digital Ocean
OpenStack
VMWare
@samnewman#geecon55
Ansible
Puppet
Chef
AWS
Digital Ocean
OpenStack
VMWare
Vagrant
@samnewman#geecon55
Ansible
Puppet
Chef
AWS
Digital Ocean
OpenStack
VMWare
Vagrant
Immutable Servers
@samnewman#geecon55
Ansible
Puppet
Chef
AWS
Digital Ocean
OpenStack
VMWare
Vagrant
Immutable Servers
Fast Spin-up
@samnewman#geecon55
Ansible
Puppet
Chef
AWS
Digital Ocean
OpenStack
VMWare
Vagrant
Immutable Servers
Fast Spin-up
Provider Agnostic
@samnewman#geecon55
Ansible
Puppet
Chef
AWS
Digital Ocean
OpenStack
VMWare
Vagrant
Immutable Servers
Fast Spin-up
Provider Agnostic
Feedback Can Suffer
@samnewman#geecon55
Ansible
Puppet
Chef
AWS
Digital Ocean
OpenStack
VMWare
Vagrant
Immutable Servers
Fast Spin-up
Provider Agnostic
Feedback Can Suffer
Cycle Time
@samnewman#geecon56
Prod
Prod
Prod
Prod
@samnewman#geecon56
Prod
Prod
Prod
Prod
Packer Images
@samnewman#geecon56
Prod
Prod
Prod
Prod
“Machine”
Service
Packer Images
@samnewman#geecon56
Prod
Prod
Prod
Prod
“Machine”
Service
“Machine”
Service Service
Packer Images
@samnewman#geecon56
Prod
Prod
Prod
Prod
“Machine”
Service
“Machine”
Service Service
Packer Images
@samnewman#geecon57
S/M TestsBuild Large Tests UAT ProdLarge Tests
@samnewman#geecon57
S/M TestsBuild Large Tests UAT ProdLarge Tests
AWS
@samnewman#geecon57
S/M TestsBuild Large Tests UAT ProdLarge Tests
AWS
VMWare
@samnewman#geecon57
S/M TestsBuild Large Tests UAT ProdLarge Tests
AWS
VMWare
Vagrant
@samnewman#geecon57
S/M TestsBuild Large Tests UAT ProdLarge Tests
AWS
VMWare
Vagrant
@samnewman#geecon57
S/M TestsBuild Large Tests UAT ProdLarge Tests
AWS
VMWare
Vagrant
@samnewman#geecon57
S/M TestsBuild Large Tests UAT ProdLarge Tests
AWS VMWareVagrant
@samnewman#geecon58
“Machine”
Service
@samnewman#geecon58
“Machine”
Service
Much Easier To Reason About
@samnewman#geecon58
“Machine”
Service
Much Easier To Reason About
Easier To Provision (Or Decommission)
@samnewman#geecon58
“Machine”
Service
Much Easier To Reason About
Easier To Provision (Or Decommission)
Fewer Side-effects
@samnewman#geecon58
“Machine”
Service
Much Easier To Reason About
Easier To Provision (Or Decommission)
Fewer Side-effects
Cost & Management Overhead!
@samnewman#geecon58
“Machine”
Service
Much Easier To Reason About
Easier To Provision (Or Decommission)
Fewer Side-effects
Cost & Management Overhead!
AWS
Digital Ocean
OpenStack
@samnewman#geecon58
“Machine”
Service
Much Easier To Reason About
Easier To Provision (Or Decommission)
Fewer Side-effects
Cost & Management Overhead!
AWS
Digital Ocean
OpenStack
@samnewman#geecon59
STANDARD VIRTUALISATION
@samnewman#geecon59
Machine
STANDARD VIRTUALISATION
@samnewman#geecon59
Machine
Base OS
STANDARD VIRTUALISATION
@samnewman#geecon59
Machine
Base OS
Hypervisor
STANDARD VIRTUALISATION
@samnewman#geecon59
Machine
Base OS
Hypervisor
VM
STANDARD VIRTUALISATION
@samnewman#geecon59
Machine
Base OS
Hypervisor
VM
OS
STANDARD VIRTUALISATION
@samnewman#geecon59
Machine
Base OS
Hypervisor
VM
OS
Apps
STANDARD VIRTUALISATION
@samnewman#geecon59
Machine
Base OS
Hypervisor
VM
OS
AppsPacker Image
STANDARD VIRTUALISATION
@samnewman#geecon59
Machine
Base OS
Hypervisor
VM
OS
Apps
VM
OS
AppsPacker Image
STANDARD VIRTUALISATION
@samnewman#geecon59
Machine
Base OS
Hypervisor
VM
OS
Apps
VM
OS
Apps
VM
OS
AppsPacker Image
STANDARD VIRTUALISATION
@samnewman#geecon60
@samnewman#geecon61
CONTAINER VIRTUALISATION
@samnewman#geecon61
Machine
CONTAINER VIRTUALISATION
@samnewman#geecon61
Machine
Base OS
CONTAINER VIRTUALISATION
@samnewman#geecon61
Machine
Base OS
Container
CONTAINER VIRTUALISATION
@samnewman#geecon61
Machine
Base OS
Container
OS
CONTAINER VIRTUALISATION
@samnewman#geecon61
Machine
Base OS
Container
OS
Apps
CONTAINER VIRTUALISATION
@samnewman#geecon61
Machine
Base OS
Container
OS
Apps
Container
OS
Apps
CONTAINER VIRTUALISATION
@samnewman#geecon61
Machine
Base OS
Container
OS
Apps
Container
OS
Apps
Container
OS
Apps
CONTAINER VIRTUALISATION
@samnewman#geecon61
Machine
Base OS
Container
OS
Apps
Container
OS
Apps
Container
OS
Apps
CONTAINER VIRTUALISATION
Linux Only
@samnewman#geecon61
Machine
Base OS
Container
OS
Apps
Container
OS
Apps
Container
OS
Apps
CONTAINER VIRTUALISATION
Same Kernel
Linux Only
@samnewman#geecon61
Machine
Base OS
Container
OS
Apps
Container
OS
Apps
Container
OS
Apps
CONTAINER VIRTUALISATION
Same Kernel
Linux OnlyFine-grained control
@samnewman#geecon61
Machine
Base OS
Container
OS
Apps
Container
OS
Apps
Container
OS
Apps
CONTAINER VIRTUALISATION
Same Kernel
Linux OnlyFine-grained control
Very fast to provision
@samnewman#geecon62
@samnewman#geecon63
DOCKER
@samnewman#geecon63
DOCKER
Machine
@samnewman#geecon63
DOCKER
Machine
Base OS
@samnewman#geecon63
DOCKER
Machine
Base OS
Docker
@samnewman#geecon63
DOCKER
Machine
Base OS
Docker
Apps
@samnewman#geecon63
DOCKER
Machine
Base OS
Docker
Apps Apps
@samnewman#geecon63
DOCKER
Machine
Base OS
Docker
Apps Apps Apps
@samnewman#geecon63
DOCKER
Machine
Base OS
Docker
Apps Apps Apps
Docker Image Registry
@samnewman#geecon63
DOCKER
Machine
Base OS
Docker
Apps Apps Apps
Docker Image Registry
@samnewman#geecon64
@samnewman#geecon64
@samnewman#geecon65
S/M TestsBuild Large Tests UAT ProdLarge Tests
@samnewman#geecon65
S/M TestsBuild Large Tests UAT ProdLarge Tests
Docker Image
@samnewman#geecon65
S/M TestsBuild Large Tests UAT ProdLarge Tests
Docker Image Registry
Docker Image
@samnewman#geecon65
S/M TestsBuild Large Tests UAT ProdLarge Tests
Docker Image Registry
Docker Image
@samnewman#geecon65
S/M TestsBuild Large Tests UAT ProdLarge Tests
Docker Image Registry
Docker Image
@samnewman#geecon66
@samnewman#geecon66
Be aware of - and balance - your test Pyramid
@samnewman#geecon66
Be aware of - and balance - your test Pyramid
Understand the balance between testing & rapid remediation
@samnewman#geecon66
Be aware of - and balance - your test Pyramid
Understand the balance between testing & rapid remediation
Deploy one thing at a time
@samnewman#geecon66
Be aware of - and balance - your test Pyramid
Understand the balance between testing & rapid remediation
Deploy one thing at a time
Consider consumer-driven contracts over integration tests
@samnewman#geecon66
Be aware of - and balance - your test Pyramid
Understand the balance between testing & rapid remediation
Deploy one thing at a time
Consider consumer-driven contracts over integration tests
Explore image-based deployments to reduce environment differences
