Continuous Delivery: How RightScale Releases Weekly

  • Published on
    16-Jan-2017

  • View
    455

  • Download
    0

Embed Size (px)

Transcript

  • CONTINUOUS DELIVERY:

    HOW RIGHTSCALE RELEASES

    WEEKLY

  • Speakers

    Tim Miller

    VP of Engineering

    Daniel Onorato

    Director of QA

    Robert Sulway

    Director of Development

  • Agenda

    Defining Continuous Delivery

    Motivations and Strategies

    Defining the Weekly Release

    Keys to Success

    Automating Is a Must

    The Dashboard

    (Surprising?) Benefits of Shorter Release Cycles

  • Your Roadmap to Application Agility

    Code Build Integrate Test Release Deploy Operate

    Agile Development

    Continuous Integration

    Continuous Delivery

    Continuous Deployment

    DevOps

  • POLLING QUESTIONS

  • Broker Cloud Services with RightScale

    Self-Service Cloud Analytics

    RightScale

    Cloud Management

    Design

    Virtualized

    Environments

    Public

    Clouds

    IaaS+/PaaS

    Services

    Private

    Clouds

    Bare

    Metal

    Automate

    Multi-Cloud Orchestration & Governance

    Operate Deploy Report Optimize

    5

  • Complex code base, lives in over 200 repositories

    Some code going back 7 years

    Running on ~700 servers in Production

    7-8 week release cycles

    Developers, QA & Ops all set in their ways, process gets layered on top of

    process

    Landscape

    6

  • Get planned features to market more quickly

    Respond to urgent requests

    Reduce WIP

    Master Branch Instability not getting tested during

    regression

    Increase quality, decrease escalations

    Details of features fresh in minds on release

    Mindset: My feature is going into production soon

    Think about deployment!

    Motivations

  • Spoiler Alert: These became benefits

    The biggest challenge was getting past We cant.. because it would lower quality bar, QA always chasing the next

    release

    ...roll out deployments in parallel, need to roll out serially

    because it will take too much time/people for release process

    The Challenges

  • Go Big

    Teams would need to innovate

    Required to shed the old and do things differently

    Add in some constraints to make things more interesting

    Cant lower the quality bar in any way

    Cant add human resources to achieve the goal

    Cant reduce velocity

    If you are going to do it weekly, cant have downtime during releases

    Strategies

    9

  • Process

    Relentless adherence to our contracts

    Contract for the definition of done for features

    Contract for the ready to ship weekly features collection

    Dashboards (Jira/Travis) to see daily status of ready to ship

    Environments

    Branching strategy

    Automation to stand up functional test environments for given features

    Staging environment to test the weekly features collection as a whole

    Keys to Success

  • Keys to Success...continued

    11 11

    Automation

    Write once, run anywhere system level test automation (feature test,

    stage integration, production)

    Matrix driven automation test execution for release deployment

    validation in production with dashboard

    Automation to facilitate rolling production deployment

    Organization

    Embedded QA Engineering on scrum teams

    Tightly integrated Dev/Ops team

  • Contracts We Live By

  • What it is:

    A vehicle to release a collection of features on a weekly basis when

    they are done

    What it is not:

    A mandate to plan, design, implement, test and deploy a given feature

    in a weeks time

    Defining the Weekly Release

  • For Libraries:

    Unit test coverage in (ci) for new feature, either measured or manually verified

    to exist by QA

    For Apps:

    Unit test coverage running in (ci) for new feature(either measured or manually

    verified to exist by QA)

    Functional test coverage in (ci) for new feature (either measured or manually

    verified to exist by QA)

    Automated system test coverage run on a mini-rs and ready for merge into

    master test collateral repo

    Note: we will require either the unit, functional or system test. The QA Eng/Devs will

    determine required coverage at the time a feature is worked.

    Adequate end user documentation has been created.

    Product owner sign-off for feature

    Definition of done

  • For both Libraries and Apps:

    Downtime window is not required

    Compatible with a rolling system restart

    Documented migration instructions exist (if necessary)

    Documented deployment instructions exist (if necessary)

    Any required ServerTemplate updates must be coded and integrated on

    a mini-rs

    Definition of done

  • A weekly features collection is deemed ready to ship when:

    Verification the new feature(s) unit test(ci), functional test(ci) and/or

    system level tests (staging) are passing on staging branch

    Verification of a full nightly regression run on staging on staging branch

    with no impact on existing features and pass/fail trends

    All regression issues are resolved

    Definition of ready to ship

  • Branching Strategy

  • Weekly View

  • Automation is a Must

  • Managing Many Environments

  • The RightScale DevOps Process

    Mini-RightScale

    Dev Environment

    Staging

    Environment Production

    Environment

    Deployment

    Automation

    Deployment

    Automation

    Pull

    code &

    test

    Deployment

    Automation

    Operations Automation

    Pull

    code &

    test

    Pull

    code &

    release

    Monitor

    alarm

    react

    Develop

    Templates

  • Write once, run anywhere test collateral

    Automation frameworks and collateral support running a given test in

    each of our possible test targets without updating test collateral once

    written. (mini-rs, staging, production).

    $ bundle exec rake SPEC=ui/functional/server_array_functional.rb

    SETTINGS=mini-rs_93,nightly_settings SHARD=93

    $ bundle exec rake SPEC=ui/functional/server_array_functional.rb

    SETTINGS=us_3,nightly_settings SHARD=3

    Automated feature tests add to collection of regression tests once

    authored

    Tying test frameworks into matrix based invocation system.

    Streamlining QA Automation

  • Gemify (Ruby) test collateral repos to allow sharing among

    various automation frameworks.

    Can speed up test execution in many cases

    Speeds up test suite implementation

    Example UI test using API test resources for setup

    Sharing Test Collateral Repos

  • require 'rester'

    require 'right_api_15

    describe "Verifies CRUD operations on the new ELB browser:", :crud => true do

    before(:all) do

    Session.login(:account_name => :test_account)

    url = config(:LOCATION)

    email = config(:EMAIL)

    password = config(:PASSWORD)

    RightApi15.default_login(login_info)

    end

    context "Setup" do

    it "creates Network required for this test" do

    @clouds.each do |cloud|

    @api_created['network'] = NetworkManager::Network.create(:cloud => cloud, :cidr_block => '13.0.0.0/16)

    end

    it "creates a load balancer using network/subnet with no name" do @clouds.each do |cloud|

    cloud_href = RightApi15::MetaData.get_cloud_href(:cloud_name => cloud)

    network = @api_created['network'].select {|n| n.cloud_href == cloud_href}.first

    subnet = @api_created['subnet'].select {|sn| sn.href.include?(cloud_href)}.first

    security_group = @api_created['security_group'].select {|sg| sg.cloud_href == cloud_href}.first

    LoadBalancer.create(:cloud => cloud,

    :load_balancer_name => "#{@name}-api",

    :VPC => { :vpc_option => network.resource_uid,

    :subnets => subnet.resource_uid,

  • Matrix Based Automation Test Invocation

  • The Dashboard

  • Our Ready to Ship Dashboard

  • Regression and Feature Bug Status

  • Automated Test Trends

  • Regression Test Trends

  • CI Dashboard

  • Have Your Cake

  • Benefits of shorter release cycles

    33 33

    Master code branch very stable at all times

    Fewer regression bugs found in integration cycle

    Automation test coverage guaranteed to grow and exist (definition of

    done)

    Forced us to keep pushing on automation and tying systems together to

    lessen the human resource tax associated with each release (work in

    progress)

    Forced us to engineer feature updates/migrations such that they can be

    performed in a rolling fashion with no system maintain window requiring

    down time.

    Releases happen in the middle of the our normal work day as opposed to

    Friday nights / Weekends. Everyone much happier with this.

    Features are deployed very close in time to there development. Details

    still fresh in our minds

  • 34

    Next Steps and Q&A

    E-book:

    Continuous Integration and Delivery in the Cloud: How We

    Do It at RightScale: How RightScale Does It

    www.rightscale.com/ci-cd-ebook

    http://www.rightscale.com/ci-cd-ebookhttp://www.rightscale.com/ci-cd-ebookhttp://www.rightscale.com/ci-cd-ebookhttp://www.rightscale.com/ci-cd-ebookhttp://www.rightscale.com/ci-cd-ebook

Recommended

View more >