The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato 2016

The Anatomy of Continuous Deployment

At Scale

John Viner (Dev Manager)Mario Visic (Lead Developer of Delivery Engineering)

100 Deploys a Week at Envato

Who is Envato?

A two-sided Marketplace for creative assets that has paid out $400 million to our

“Authors” in the last 10 years

7 Sites where digital professionals can buy web templates, video, audio, code, graphics and 3d

models 7 million registered users10 million items for sale

Themeforest.net

Show me the Tech Stack

Ruby on Rails (100K loc, 25 web servers)

MySql (175 tables)

Memcached + RedisRackspace / VMWare

Amazon AWS

Show me the Numbers!

Peak of 25,000 requests p/min through Rails160m Page Views per Month

500,000 Database calls p/min Serving 200TB of data a month

22,000 Unit Tests, 2,000 Integration Tests

How do we deliver change?

By the numbers … for the month of June 2016

~ 40 developers3000 total builds (2.5 builds p/dev/day)

500 Pull Requests Closed400 deployments (18 p/day avg, 0.5 p/dev/day, peak 25)

4.5 hour cycle time (from first commit to deploy - working hours)

Statistics over Time

Size of a Pull Request (median)37 lines changed3 files changed

Cycle Time of a PR (first commit to deploy)average: 10.8 hours (working hours)median: 2.6 hours (working hours)

Deployments8400 deployments in 2 years

Pretty Charts - Builds

Deployments

Pull Requests

PR Cycle Time (First Commit to Deployment)

Building Envato Market

Feature request

• 2 week sprints• Work with product team to break down work• No task larger than two days• Feature toggle can be used to push incomplete features

Feature request

Implementation

• Write the tests (sometimes first)• Write the code• Peer review• Get it into production

Implementation

Tests live alongside code, stored in Github

Implementation

Github Pull Request are the unit of change, 2 thumbs up by reviewers means the code is good to ship to production

Implementation

Implementation

Buildkite is our CI tool, a merge on master automatically runs a build. A successful build automatically triggers a production

deploy

Deployment

• Entirely Automated• Post deploy monitoring• Quick rollback• Multiple time per hour

Deployment

Samson (open source) is our deployment tool, successful master builds triggers a deploy which is logged in Samson.

Deployment

We can run manual deployments from Samson and view all PRs in a given deployment.

Deployment

We can trace back to specific changes in a deploy.

Deployment

… including the amount of lines changed. A typical PR changes only 37 lines of code across 3 files

Deployment

Why do we do this?

Lowers Risk

Size of Change is Small means …Easier to identify any failureEasier to rollback any failure

Reduces human errors due to reliance on automationSecurity patches can be deployed within minutes

And also

Reduces Waste:Fewer handovers from Development to QA to Ops

Increases Product Agility:New features go to customers sooner to get earlier feedback

Increases developer ownership and engagement:More direct connection between your creation and it being used

How to get there

Start with Culture

Foster an Ownership Culture in Developers

http://webuild.envato.com/expectations/collaboration/4-i-take-responsibility-for-syste.html

Investment in automation of unit tests, functional tests

Tests to cover legacy systems

Add Tests whenever un-tested code is touched

Set Targets and Measure Test Coverage and # Tests

Automate Deployments and Enable Feature Toggles

Automate your manual run sheet one line at a time

Start with Push Button

Feature Toggles help deploy features without showing them

Start Small

Use a less risky or complicated system first

Investment in monitoring and rollback

Knowing that the system is working within seconds becomes critical

Developers need to be “on the pager”

Invest in the ability to Roll Back

Optimise for Mean Time to Recover

Thank you!