CONTINUOUS DELIVERY: A JOURNEY NOT A DESTINATION€¦ · TESTING WITH FEATURE TOGGLES 26 describe...

CONTINUOUS DELIVERY: A JOURNEY NOT A DESTINATIONAkshay Karle & Badrinath Janakiraman @snap_ci | https://snap-ci.com

1

2

The essence of my philosophy to software delivery is to

build software so that it is always in a state where it

could be put into production. We call this Continuous

Delivery because we are continuously running a

deployment pipeline that tests if this software is in a

state to be delivered. - Jez Humble

http://martinfowler.com/delivery.html

3

…high-performing organizations are still deploying

code 30 times more frequently, with 50 percent

fewer failures than their lower-performing

counterparts.

https://puppetlabs.com/sites/default/files/2014-state-of-devops-report.pdf

4

5

frequent, reliable releases of high quality

valuable software

6

SNAP, BACK IN 2012…

Built on 9 components (microservices, anyone?)

A message-queue for fast background job processing

Chef-server for infrastructure provisioning

Relational & NoSQL databases

Extensive Nagios monitoring

Completely automated infrastructure creation

Build and deployment pipelines for each component

So clearly - we had it nailed. Right?7

8

CONSEQUENCES

Likelihood of build setup passing 50%

Likelihood of changes actually triggering a build 50%

Likelihood that someone knew why 0%

Likelihood of 4 SSH sessions firing up debug 100%

Likelihood that the infrastructure was busted 50%

Frequency of production upgrades Twice a month

Consequence of upgrades Site unavailable!

Feedback to users about what was going on Minimal9

10

frequent, reliablereleases of high quality

valuable software

11

Our journey so far..

Queues per build

Messages lost, redelivery problems

Single, centralized queue

No need to manage queues

Delayed::Job

one Nagios to rule them all

too many components

too many metrics and alerts

ignored alerts when things actually did go wrong

separated application monitoring

basic server monitoring

aggregated logs

we don’t need to manage anything

Serverbecame cognizant of what state would go into chef

reliable and faster chef runs

nothing to manage

user build specific configuration data

dependency on the chef server for deployments

had to manage the server

Solo

Single mutable environments

Too many components

READONLY MODE Long running deployments

Loosing build requests

Loosing customers

Fewer components

Blue-green deployments

BLUE-GREEN DEPLOYMENTS IN SNAP

17

VZHOSTBuild Server

web server

VZHOSTBuild Server

web server

L B DATABASE

COUPLED MODULES TO MONOLITH TO REAL MODULES

Highly coupled initial state

Collapsed moving parts to a monolith

Deploying frequently highlights rates of change

This was used to make new modules

Much more decoupled state today

18

SO, WHAT’S THE CURRENT STATE?

Build setups almost always pass…

…but if there are failures, the team gets alerted

Deployments happen every other day

New releases happen once every 2~3 weeks

Near zero downtime

19

20

OUR CURRENT WORKING STYLE

Appropriate sequencing of stories

Using feature toggles

Structured evolution of data and app

Identical environments as trip-wires

21

SEQUENCING OF STORIES

22

short (idealized) cycle time, more technical risk

time

time

longer (idealized) cycle time, less technical risk

FEATURE TOGGLES

Hide unfinished UI elements

Control backend behaviour

Test with feature toggles

Avoid multi-component feature toggles

23

FRONTEND

24

<% if feature_enabled?(:parallel_stage) %> <!-- show new html --><% else %> <!-- show old html --><% end %>

edit.html.erb

BACKEND

25

if feature_enabled?(:parallel_stage) # new logicelse # old logicend

babysitter.rb

TESTING WITH FEATURE TOGGLES

26

describe "multiple jobs" do describe "feature enabled" do before(:each) do with_feature_enabled(:parallel_stage) end it 'should not show the job tabs when there is only one job' do end end describe "feature disabled" do before(:each) do with_feature_enabled(:parallel_stage) end it 'should not show the job tabs but should show the logs' do end endend

stage_run_spec.rb

MIGRATION OF DATA

Consider existing schema as well as data

Incremental changes

Rollbacks

Zero downtime releases

27

INTRODUCE JOBS

28

jobs

id

POPULATE ALL EXISTING STAGES

29

stages

id

started_at

completed_at

result

…

jobs

id

stage_id

…

COPY ATTRIBUTES TO JOB

30

stages

…

started_at

completed_at

result

…

jobs

…

started_at

completed_at

result

…

SWITCH THE APPLICATION TO START USING THE JOB MODEL

31

# After switchclass Stage def result results = jobs.collect { |job| job.result } return :failed if results.any?(:failed) :passed endend

# Before switchclass Stage attr_reader :resultend

# in transitionclass Stage def result if feature_enabled?(:parallel_stage) results = jobs.collect { |job| job.result } return :failed if results.any?(:failed) :passed else result end endend

REMOVE UNUSED ATTRIBUTES

32

stages

id

started_at

completed_at

result

…

jobs

…

started_at

completed_at

result

…

IDENTICAL ENVIRONMENTS/AUTOMATION

33

staging

buildn+1

upgradesmoketests�

ver. n

To production

34

buildn+1

upgradesmoketests�

ver. n+1

To production

staging

35

buildn+1

upgradesmoketests�

ver. n+1 staging

To production

36

�

ver. n+1 staging

buildn+1

upgradesmoketests

buildn+2

upgradesmoketests� To production

37

�

ver. n+2 staging

buildn+1

upgradesmoketests

buildn+2

upgradesmoketests� To production

38

�

ver. n+2 staging

buildn+1

upgradesmoketests

buildn+2

upgradesmoketests� To production

SUMMARY

Start with getting working software into hands of users

Layer on qualities of reliability, automation, frequent releases etc

Kick the can down the road

Don’t be afraid of “re-design”/”re-work”

Let CD guide your architecture and design - not the other way around

39

Akshay Karle, @akshay_karle

THANK YOU

Badrinath Janakiraman, @badrij

@snap_ci | https://snap-ci.com