CONTINUOUS DELIVERY:

HOW RIGHTSCALE RELEASES

WEEKLY

Speakers

• Tim Miller

• VP of Engineering

• Daniel Onorato

• Director of QA

• Robert Sulway

• Director of Development

Agenda

• Defining Continuous Delivery

• Motivations and Strategies

• Defining the Weekly Release

• Keys to Success

• Automating Is a “Must”

• The Dashboard

• (Surprising?) Benefits of Shorter Release Cycles

Your Roadmap to Application Agility

Code Build Integrate Test Release Deploy Operate

Agile Development

Continuous Integration

Continuous Delivery

Continuous Deployment

DevOps

POLLING QUESTIONS

Broker Cloud Services with RightScale

Self-Service Cloud Analytics

RightScale Cloud Portfolio Management

Cloud Management

Design

Virtualized

Environments

Public

Clouds

Other

Services

Private

Clouds

Automate

Multi-Cloud Orchestration & Governance

Operate Deploy Report Optimize

• Complex Cloud Management code base, lives in over 200 repositories

• Some code going back 7 years

• Running on ~600 servers in Production

• 7-8 week release cycles

• Developers, QA & Ops all set in their ways, process gets layered on top of

process

Landscape

6

• Get planned features to market more quickly

• Respond to urgent requests

• Reduce WIP

• Master Branch Instability – not getting tested during

regression

• Increase quality, decrease escalations

• Details of features “fresh in minds” on release

• Mindset: My feature is going into production soon

• Think about deployment!

Motivations

• Go Big

• Teams would need to innovate

• Required to shed the old and do things differently

• Add in some constraints to make things more interesting

• Can’t lower the quality bar in any way

• Can’t add human resources to achieve the goal

• Can’t reduce velocity

• If your are going to do it weekly, can’t have downtime during releases

Strategies

8

• Spoiler Alert: These became benefits

• The biggest challenge was getting past “We can’t..” • …because it would lower quality bar, QA always chasing the next

release

• ...roll out deployments in parallel, need to roll out serially

• …because it will take too much time/people for release process

The Challenges

What it is:

• A vehicle to release a collection of features on a weekly basis when

they are “done”

What it is not:

• A mandate to plan, design, implement, test and deploy a given feature

in a week’s time

Defining the Weekly Release

• Process

• Relentless adherence to our “contracts”

• Contract for the definition of “done” for features

• Contract for the “ready to ship” weekly features collection

• Dashboards (Jira/Travis) to see daily status of “ready to ship”

• Environments

• Branching strategy

• Automation to stand up functional test environments for given features

• Staging environment to test the weekly features collection as a whole

Keys to Success

Keys to Success...continued

12 12

• Automation

• Write once, run anywhere system level test automation (feature test,

stage integration, production)

• Matrix driven automation test execution for release deployment

validation in production with dashboard

• Automation to facilitate rolling production deployment

• Organization

• Embedded QA Engineering on scrum teams

• Tightly integrated Dev/Ops team

Contracts We Live By

• For Libraries:

• Unit test coverage in (ci) for new feature, either measured or manually verified

to exist by QA

• For Apps:

• Unit test coverage running in (ci) for new feature(either measured or manually

verified to exist by QA)

• Functional test coverage in (ci) for new feature (either measured or manually

verified to exist by QA)

• Automated system test coverage run on a mini-rs and ready for merge into

master test collateral repo

• Note: we will require either the unit, functional or system test. The QA Eng/Devs will

determine required coverage at the time a feature is worked.

• Adequate end user documentation has been created.

• Product owner sign-off for feature

Definition of “done”

• For both Libraries and Apps:

• Downtime window is not required

• Compatible with a rolling system restart

• Documented migration instructions exist (if necessary)

• Documented deployment instructions exist (if necessary)

• Any required ServerTemplate updates must be coded and integrated on

a mini-rs

Definition of “done”

A weekly features collection is deemed “ready to ship” when:

• Verification the new feature(s) unit test(ci), functional test(ci) and/or

system level tests (staging) are passing on staging branch

• Verification of a full nightly regression run on staging env from staging

branch with no impact on existing features and pass/fail trends

• All regression issues are resolved

Definition of “ready to ship”

Branching Strategy

Weekly View

Automation is a Must

Managing Many Environments

The RightScale DevOps Process

Mini-RightScale

Dev Environment

Staging

Environment Production

Environment

Deployment

Automation

Deployment

Automation

Pull

code &

test

Deployment

Automation

Operations Automation

Pull

code &

test

Pull

code &

release

Monitor

alarm

react

Develop

Templates

• Write once, run anywhere test collateral

• Sharing test frameworks and collateral across products

• Tying test frameworks into matrix based invocation system.

Streamlining QA Automation

• Automation frameworks and collateral support running a given

test in each of our possible test targets without updating test

collateral once written. (mini-rs, staging, production).

• $ bundle exec rake SPEC=ui/functional/server_array_functional.rb

SETTINGS=mini-rs,nightly_settings SHARD=93

• $ bundle exec rake SPEC=ui/functional/server_array_functional.rb

SETTINGS=staging,nightly_settings SHARD=93

• $ bundle exec rake SPEC=ui/functional/server_array_functional.rb

SETTINGS=us_3,nightly_settings SHARD=3

Write once, run anywhere test collateral

• Gemify (Ruby) test collateral repos to allow sharing among

various automation frameworks.

• Can speed up test execution in many cases

• Speeds up test suite implementation

• Example UI test using API test resources for setup

Sharing Test Collateral Repos

require 'rester'

require 'right_api_15’

describe "Verifies CRUD operations on the new ELB browser:", :crud => true do

before(:all) do

Session.login(:account_name => :test_account)

url = config(:LOCATION)

email = config(:EMAIL)

password = config(:PASSWORD)

RightApi15.default_login(login_info)

end

context "Setup" do

it "creates Network required for this test" do

@clouds.each do |cloud|

@api_created['network'] = NetworkManager::Network.create(:cloud => cloud, :cidr_block => '13.0.0.0/16’)

end

…

it "creates a load balancer using network/subnet with no name" do

@clouds.each do |cloud|

cloud_href = RightApi15::MetaData.get_cloud_href(:cloud_name => cloud)

network = @api_created['network'].select {|n| n.cloud_href == cloud_href}.first

subnet = @api_created['subnet'].select {|sn| sn.href.include?(cloud_href)}.first

security_group = @api_created['security_group'].select {|sg| sg.cloud_href == cloud_href}.first

LoadBalancer.create(:cloud => cloud,

:load_balancer_name => "#{@name}-api",

:VPC => { :vpc_option => network.resource_uid,

:subnets => subnet.resource_uid,

…

Matrix Based Automation Test Invocation

The Dashboard

Our “Ready to Ship” Dashboard

Regression and Feature Bug Status

Automated Test Trends

Regression Test Trends

CI Dashboard

Have Your Cake…

Benefits of shorter release cycles

34 34

• Master code branch very stable at all times

• Fewer regression bugs found in integration cycle

• Automation test coverage guaranteed to grow and exist (definition of

done)

• Forced us to keep pushing on automation and tying systems together to

lessen the human resource tax associated with each release (work in

progress)

• Forced us to engineer feature updates/migrations such that they can be

performed in a rolling fashion with no system maintain window requiring

down time.

• Releases happen in the middle of the our normal work day as opposed to

Friday nights / Weekends. Everyone much happier with this.

• Features are deployed very close in time to there development. Details

still fresh in our minds

35

Next Steps and Q&A

E-book:

Continuous Integration and Delivery in the Cloud: How We

Do It at RightScale: How RightScale Does It

www.rightscale.com/ci-cd-ebook