Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

9/9/13

1

Continuous Delivery (DevOps Best Practices)

1

Bob Aiello, Principal Consultant and Author of Configuration Management Best Practices : Practical Methods that Work in the Real World

http://www.linkedin.com/in/BobAiello http://cmbestpractices.com

CM Best Practices Consulting © 2013

Who am I? •  CM/DevOps Lead & Consultant for over 25 years •  Editor-in-Chief at CM Crossroads •  Author of CM Best Practices •  IEEE Management Board •  Tools and process agnostic •  The guy called in the middle of the night when the release doesn’t work!

2 April 9, 2013 http://cmbestpractices.com © 2013

9/9/13

2

Goals of this Course •  Understand Continuous Delivery •  Configuration Management roots •  Control Dependencies & Configuration •  Continuous Integration •  Build and Deployment Automation •  Deployment Pipeline is an Art!

Agile Release Train


DevOps Focus •  Understand DevOps Best Practices •  A Little History of DevOps •  Scope of DevOps and how to get started •  The People side of DevOps Establish your own plan for DevOps!


9/9/13

3

And Don't Forget •  Delivery Ecosystem •  Components & Dependencies •  Test vs Verification & Validation (V&V) •  Don't forget the Data •  Establish IT governance and compliance So what is Continuous Delivery?


Continuous Delivery

6

•  Methodology for getting software from development to release •  Focus on the Deployment Pipeline •  Rapid incremental deployment •  Minimize Risk •  Many small deployments better than big bang

http://cmbestpractices.com © 2013 April 9, 2013

9/9/13

4

Continuous Integration

7

•  What is Continuous Integration •  Why does CI work? •  Martin Fowler reminds us to test •  Let's consider the ergonomics


Lessons from Aviation

8

•  Cockpit of a plane •  Controls are easy to read •  Traceability •  Designed to avoid mistakes How does this relate builds?


9/9/13

5

Ergonomics of the Build

9

•  “Bob-proof” your build •  Implicit verification and validation •  Avoid the possibility of mistakes •  Each step should be easy to understand •  One step should not break the stream •  Use dashboards and reports to communicate status http://cmbestpractices.com © 2013 April 9, 2013

Knight Capital Group

10

•  August 1st 2012 trading disaster •  Related to NYSE systems upgrade •  Resulted in a $440 million loss •  Loss grew as customers left the firm •  Knight Capital Group merged with GETCO holding company DevOps doesn't cost $440 million http://cmbestpractices.com © 2013 April 9, 2013

9/9/13

6

Introducing the Trusted Base

11

•  Ensure that you know exactly what you built •  Verify that the release gets deploy •  Ensure that there are no unauthorized changes Understanding Continuous Integration


Features of CI

12

•  Source Code Management •  Building the Code •  Database integration •  Testing •  Source code inspection •  Deployment Controlling isolation


9/9/13

7

Controlled Isolation

13

•  Developers work in sandboxes •  Deliver (actually published) your changes •  Rebase before you deliver •  Private builds •  Manage variants Consider the ergonomics


Ergonomics of CI/CD

14

•  Small incremental changes •  Early Warning •  Reduce risk •  Easier to triage •  Easier to backout What about life support systems?


9/9/13

8

I was once asked...

15

•  What if you were upgrading a life support system and your loved one was impacted •  How do we ensure that there are no mistakes •  Join me in Detroit in July How do we keep CI lean?


Keeping CI Lean

16

•  Too many builds •  Too much noise •  Tag interesting builds What about pilots and landings?


9/9/13

9

Pilots Abort Landings

17

•  Recently I was on a plane when another plane was on the same runway •  You want to fail early when necessary •  Abort bad builds and identify the cause What are the best practices?


Seven Practices, Duvall p. 39

18

•  Commit code frequently •  Don't commit broken code •  Fix broken builds immediately •  Write automated developer tests •  All tests & inspections must pass •  Run private builds •  Avoid getting broken code


9/9/13

10

Chicago Board Options Exchange

19

•  Planned systems upgrade •  Problem with staging software •  Employees reported there was a problem •  CBOE did not fail over Could DevOps have helped the CBOE?


What is DevOps?

20

•  New Term for... •  Portmanteau •  Agile Systems Administration •  Agile Operations •  Group of Principles Now that we cleared that up!


9/9/13

11

New Term

21

•  Group of concepts •  Been around for a while •  Use case is compelling •  Stimulating discussion •  Necessary to meet demand


Portmanteau

22

•  Combination of two words •  Development •  Operations Development and Operations have very different goals


9/9/13

12

Conflict Between Dev & Ops

23

•  Development focused on delivering new functionality •  Operations is focused on providing continuous (reliable) services •  Manage risk! One time I was asked to break the rules


Trying to make the deadline

24

•  Trading system was tested and passed •  Few bugs discovered •  I was asked to deliver a different version than was tested How does DevOps help balance?


9/9/13

13

DevOps is also

25

•  Emerging Best Practices •  Collaboration between Dev & Ops •  Application and Systems Deployment •  Software and Systems Development But is DevOps Agile?


What about Agile?

26

•  Agile Systems Administration •  Agile Operations •  Waterfall needs DevOps too! Release Antipatterns...


9/9/13

14

Release Antipatterns

27

l  Deploying software manually l  Deploying to a production-like environment only after development is complete l  Manual configuration of production environment. So what really is DevOps?


DevOps is Really...

28

•  Developer and Operations collaboration •  Crossfunctional team •  Knowledge Management •  Better communication Time to get rid of silos http://cmbestpractices.com © 2013 April 9, 2013

9/9/13

15

What is Ops?

29

•  Blanket term •  Systems engineers •  Systems administrators •  Operations staff


What's honesty got to do with all this?

Agile on What We Know

30

•  Don't try to define requirements we do not yet understand •  Last responsible moment •  Requirements documents that are unusable Deming says, “drive out fear”


9/9/13

16

What is a Deployment Pipeline?

31

•  Build once •  Deploy the same way to every environment •  Smoke is essential •  Deploy to a copy of Production •  Manage the pipeline Environment management http://cmbestpractices.com © 2013 April 9, 2013

Deployment Pipeline

32

A deployment pipeline is … an automated implementation of your application’s build, deploy, test and release process Jez Humble and David Farley’s Continuous Delivery, p 3.


9/9/13

17

Aim of the Pipeline •  Makes building, deploying, testing and releasing software visible to everyone involved •  Improves feedback so that problems are identified, and so resolved, as early in the process as possible •  Enables teams to deploy and release any version of their software to any environment at will through a fully automated process (p. 4)

33 http://cmbestpractices.com © 2013 April 9, 2013

Antipatterns

34

•  Deploying Software Manually •  Deploying to Production-like environment only after Development is complete •  Manual Configuration of Production Environments Continuous Deployment, p. 7 – 10


9/9/13

18

Agile Release Train (ART) Making each product a successful and routine event – an event that is indeed planned and eagerly anticipated yet one one that happens almost on autopilot Dean Leffingwell’s Agile Software Requirements, p. 299

35 http://cmbestpractices.com © 2013 April 9, 2013

How Do We Implement? Are deployment pipelines practical? How do we figure out the details? Is it worth the time and effort? What are the benefits? What are the risks? It's really all about knowledge... 36 http://cmbestpractices.com © 2013 April 9, 2013

9/9/13

19

Knowledge Management

37

•  There are always a few experts •  They are not working when system glitch •  Building a Knowledge Management System •  But What if some people do not want to share? (caution silos ahead)


Beware of Silos

38

•  The SAs see file systems •  The DBAs have a different view •  WebSphere Admins •  InfoSec helped us secure (so much nothing worked) DevOps is about sharing knowledge!


9/9/13

20

Where did DevOps start?

39

•  O'Reilly Velocity Conference 2008 •  Web Performance and Operations •  “Infrastructure as Code”


Need for Rapid Change

40

•  2009 Presentations on developer / operations collaboration at large shops along with safe rapid change of Web environments


9/9/13

21

DevOps Days

41

•  Patrick Debois – DevOpsDays in 2009 •  Tools (actually toolchains) have brought together “the three layers of what you need for agile movement (principles, process and practices)”


Let's Get Into Some Details

42

•  How do we implement? •  How do we make pragmatic choices? •  How do we do this in the real world? My experience taking down NYSE


9/9/13

22

I Was Once Accused

43

•  Promoting the wrong shell scripts •  Taking down the NYSE •  Stopping the World Economy Principles of Software Delivery


Continuous Delivery

44

•  Configuration Management focus •  Version control •  Dependency and configuration control Principles of Software Delivery


9/9/13

23

Software Delivery Principles

45

•  Create repeatable, reliable process •  Automate as much as possible •  If it hurts, do it more often! •  Build quality in from the beginning (Deming) any more?


More Principles

46

•  Done means released •  Everyone is responsible for the delivery process •  Continuous improvement •  Version control is key So what is CM?


9/9/13

24

Configuration Management

47

•  Configuration Identification •  Status Accounting •  Change Control •  Configuration Audit Tracking and Controlling Changes to Configuration Items


Configuration Identification

48

•  Provides a specific and unique identity to each configuration item (e.g. binary, config file, documentation) •  Selecting the configuration items for a system and recording their functional and physical characteristics (Sevocab)


9/9/13

25

Status Accounting

49

•  Tracking the status of a configuration item throughout its lifecycle. •  Recording and reporting of information needed to manage a configuration effectively (Sevocab) http://cmbestpractices.com © 2013 April 9, 2013

Change Control

50

•  Establishing checkpoints including gatekeeping (e.g. Production, QA, UAT) and configuration control. •  Identifying, documenting, approving or rejecting, and controlling changes to the project baselines (Sevocab) http://cmbestpractices.com © 2013 April 9, 2013

9/9/13

26

Configuration Audit

51

•  Inspect and identify the exact version of any configuration item (physical & functional) •  Independent examination of the configuration status to compare with the physical configuration (Sevocab) http://cmbestpractices.com © 2013 April 9, 2013

Functional description of CM

52

•  Easier to understand in the context of a lifecycle •  Consisting of six core CM functions •  Closely matches the job descriptions of the people doing the work •  Can be tailored to your needs So what are the six functions?


9/9/13

27

CM Functions

53

•  Source Code Management •  Build Engineering •  Environment Configuration •  Change Control •  Release Engineering •  Deployment Let's start with a brief overview http://cmbestpractices.com © 2013 April 9, 2013

My buddy from Harvard

54

•  Builds are too complex to automate •  Some folks do not want to see automation as being possible •  You may have to shadow or ask to drive •  Document the procedures and then Script your build...


9/9/13

28

Build Principles

55

•  Create a script for each stage of the build process •  Use the right technology to deploy (find out what others are doing) •  Use your operating systems native tools


More Build Principles

56

•  Idempotent – reliable and no side effects •  Evolve your deployment system incrementally •  Start with “attended automation”


9/9/13

29

Some other tips

57

•  Relative paths (watch your paths) •  Eliminate manual steps •  Traceability from binary to source •  Test targets should not fail the build What do I do with binaries?


Managing Binaries

58

•  Binaries can be rebuilt •  Based upon baselines •  Verifiable (I hope) •  Don't belong in the VCS with source •  Definitive Media Libraries •  Release Repos Managing Variants in the Code


9/9/13

30

Version Control Features

59

•  Provides history and security •  Model the architecture •  Reduce complexity •  Model the process More on streams


Source Code Management

60

•  Control of every configuration item (e.g. source code, config, binaries, compile and runtime dependencies). •  Much more than just checkin and checkout (version control) •  Provides sanity to the development process (reduces cognitive complexity)


9/9/13

31

Terminology

61

•  Configuration items (CIs) include binaries, source code, config files and even documents •  ISO 1007 notes end user function •  Bob says, “anything where getting the wrong version would be bad”


What is Control?

62

•  In CM, control is managing the evolution of a CI throughout its lifecycle •  Change Control •  Configuration Control Is control really the right word? http://cmbestpractices.com © 2013 April 9, 2013

9/9/13

32

Principles

63

•  Code is locked down and can never be lost •  Code is baselined marking specific milestones •  Managing variants using branches •  Code changed on a branch can be merged http://cmbestpractices.com © 2013 April 9, 2013

More Principles

64

•  Processes are repeatable Agile and Lean •  Traceability and tracking of all changes •  Improves productivity and quality


9/9/13

33

Best Practices

65

•  How do we establish source code management that adheres to these principles? •  Better question is how does CM add value and help facilitate the development effort?


Streams

66

•  Provides a clear usage paradigm •  Model components and architecture •  Control flow of changesets •  Snapshots create baseline of code


9/9/13

34

Streams

67

l  Ability to load a particular snapshot l  Strong security authorization and entitlements l  Complete history and traceability How about task based development?


Defect & Task Tracking

68

•  Track changesets to workitem •  Traceability to who made the change •  Makes release notes a breeze to create •  Ties back to requirements and test cases •  Allows for ALM and workflow automation http://cmbestpractices.com © 2013 April 9, 2013

9/9/13

35

InfoSec Scans Code

69

•  Source Code Inspection •  Are coding practices creating risk? •  Are passwords being hardcoded •  Scan for complexity •  Code quality Managing globally distributed teams http://cmbestpractices.com © 2013 April 9, 2013

Globally Distributed team

70

•  Managing work for a globally distributed team •  Effective communication •  Better coordination •  Traceability •  Visibility


9/9/13

36

Build on Commit

71

•  Nightly builds often enough •  Build on demand •  Pre-flight (private) builds •  Build framework Tame the complexity and communicate via dashboards


Deploy to Environment

72

•  Run automated tests •  Monitor the environment •  Build the Ops Knowledgebase •  Building our deployment framework Infrastructure as code


9/9/13

37

Infrastructure as Code

73

•  Provisioning Servers •  Fundamental in the Cloud •  What about private clouds? •  Managing the OS Puppet and chef


Puppet/Chef

74

•  Automate provisioning, patching and configuration of operating system and application components •  Systems integration framework •  Scalable and extensible •  Used in other deployment frameworks www.puppetlabs.com www.opscode.com


9/9/13

38

CIS Benchmark

75

•  Center for Internet Security (CIS) •  Consists of hundreds of recommended configurations •  Code is included to verify the configuration This is all about taming complexity


Taming Complexity

76

•  Understand the technology •  Automate everything •  Do it more often •  Move upstream •  Build a framework By the time we get to Production...


9/9/13

39

Build Once

77

•  Build once – deploy everywhere •  Ensure bits are identical •  Build based upon baseline •  Embed immutable version IDs •  Configuration audit Automated deployments


Deployment Frameworks

78

•  Starts with scripting •  Many dependencies •  Taming complexities •  Test each step Traceable, Repeatable Process


9/9/13

40

Deploy the Same Every Environment

79

•  Write a deployment framework •  Practice the deploy •  Well oiled machine •  Repeatable and traceable DevOps Focus


DevOps

80

•  Moving automation upstream •  Communicating with stakeholders •  Building knowledge •  Infrastructure as code Smoke testing is required


9/9/13

41

SmokeTest

81

•  Test the deploy itself •  Put in the first trade •  Verify what changed •  Work with QA & Testing Environments need to be similar to Production


Deploy into Copy of Production

82

•  You need a dress rehearsal •  Need to verify automation works •  Need to know the deploy will work •  Manage risks and unknowns Deploys need to be verifiable


9/9/13

42

Changes Through the Pipeline

83

•  Every commit triggers •  Build and deploy automation •  Testing the release What are the recommended practices?


Pipeline Practices

84

•  Only build binaries once •  Deploy the same way to every environment •  Smoke test Changes should propagate instantly continuously The process itself must be testable


9/9/13

43

Verification and Validation

85

•  Does it meets requirements? •  Are the requirements correct? •  Deming – build in quality •  Each step is testable When problems occur...


Stop the Line

86

•  Need to detect defects early •  Stop the process immediately •  Easier to diagnose •  Easier to fix Kanban for the Deployment Pipeline


9/9/13

44

Kanban

87

•  Push •  Pull •  Implement through workflow automation Delivery Environment as an Ecosystem


Delivery Ecosystem

88

•  Understanding Operations •  Managing Infrastructure •  Server Provisioning & Configuration •  Managing Middleware •  Virtualization & Cloud DevOps should focus on Ops


9/9/13

45

Operations

89

•  Key stakeholder •  Often outgunned and kept in the dark •  Building a knowledgebase •  Automating detection and response How can Ops get ahead of the curve?


Operations in DevOps

90

•  Infrastructure as Code •  Provisioning servers •  Monitoring the environment •  Monitoring events InfoSec is also key


9/9/13

46

InfoSec

91

•  Key stakeholder •  Often misinformed •  Policies don't secure systems •  Many incidents show this is a problem area Securing the trusted base


Securing the Trusted Base

92

•  Builds are baselined •  Version IDs are embedded •  Configuration audit •  Non-repudiation Security is quality


9/9/13

47

Build Security In

93

•  Security should be considered from the beginning •  Security and quality are tightly coupled •  Provision servers using standards •  Control & Detect unauthorized changes Manage components & dependencies


Managing Components

94

•  Code should be designed into components •  Reduces complexity •  Interfaces are essential •  Part of environment management Managing Big Builds


9/9/13

48

Managing Big Builds

95

•  Big builds may require multiple pipelines •  Treat the team as internal products •  Handle this as COTs Configuration can be complex


Managing Configuration

96

•  Many ways to handle this •  Configuration files (httpd.conf) •  Properties files (.properties) •  XML as configuration (server.xml) •  Default as production (so you don't forget!) Managing Dependencies


9/9/13

49

Managing Dependencies

97

•  Maven and Ivy help identify dependencies •  Need to be able to identify versions •  Monitor and detect issues •  Often controlled through data But you have to test


Testing Topology

98

•  Unit Testing •  Functional •  Regression •  Integration •  User Acceptance What about non functional testing?


9/9/13

50

Non-‐functional Testing

99

•  Capacity •  Performance •  Scalability Shoemakers children...


Testing the Pipeline

100

•  You need to test the automation including build, package and deployment •  Fail early! •  Trust but verify Don't forget the data


9/9/13

51

Internal Audit Requirements

101

•  Managing baselines •  Traceability •  Change control •  Seperation of controls Regulatory Requirements


Conducting an Assessment

102

•  What is going well •  What can be improved? Assess to industry standards and frameworks


9/9/13

52

Regulatory

103

•  Section 404 of the Sarbanes-Oxley Act of 2002 •  SSAE-16 (formerly SAS-70) •  Finra •  Office of the Currency Standards and Frameworks


Industry Standards

104

•  IEEE 828 •  EIA 649-B •  ISO 12207 or 15288 •  ISO 9001 Frameworks also provide guidance


9/9/13

53

Frameworks

105

•  Cobit for Sox Compliance •  ITIL for IT Service Management •  CMMI (less common in financial services)


Globally Distributed team

106

•  Managing work for a globally distributed team •  Effective communication •  Better coordination •  Traceability •  Visibility


9/9/13

54

The CD/CI/CM Process

107

•  Should be Lean •  Processes need to be reviewed •  Tailor down or tailor up •  More collaboration and consensus building •  Use standards and frameworks

April 9, 2013 http://cmbestpractices.com © 2013

Assessment

108

•  First step is to assess current practices - “As-Is” •  Compare to industry standards and frameworks •  Determine “To-Be” •  Create a plan for improving your CM processes

April 9, 2013 http://cmbestpractices.com © 2013

9/9/13

55

IT Governance & Compliance

109

•  IT Governance needs to be in alignment with corporate governance •  Financial reports needs to be accurate •  Separation of controls •  Security measures to prevent unauthorized access •  Audit in place for intrusion detection http://cmbestpractices.com © 2013 April 9, 2013

Sox Compliance

110

•  Section 404 of the Sarbanes Oxley Act of 2002 •  Using ISACA Cobit 4.1 •  34 high level IT controls •  PCI compliance •  SAS-70


9/9/13

56

ISO 9001

111

•  Establishes the quality management system •  ISO 90003 is the software standard in the 9000 family of standards •  Uses ISO 12207 (or 15288) to specify lifecycle processes •  ISO 10007 for CM •  IEEE 828, EIA 649-A, Mil Std coming!


Which Standards?

112

•  IEEE 828 – CM Planning •  EIA 649-A – Non compliance •  ISO 90003 to support QMS •  Full lifecycle ISO 12207 Tailor !


9/9/13

57

Moving Upstream

113

•  Dev to CM to QA to Ops •  Cross functional focus •  Speed up development •  Build a great deployment architecture •  Give it to Devs as a service!


Frameworks

114

•  ITIL v3 including CMDBs, federated CMDBs, CMS, DML… •  Cobit for SOX •  CMMI ->>>> Agile


9/9/13

58

How Do We Improve

115

•  CSI is well - continuous •  Inclusive •  Transparent •  Learning from mistakes Retrospectives are essential http://cmbestpractices.com © 2013 April 9, 2013

Retrospective

116

•  After action review •  Need open and honest evaluation •  Opportunity to improve the process •  Drives the entire release process


9/9/13

59

Plan for Improvement

117

•  Improve training and use case for source code management •  Improvement build automation •  Setup or improve continuous integration •  Automate package and deployment •  Create procedures for configuration audit April 9, 2013 http://cmbestpractices.com © 2013

CM/Devops

118

•  Flexible technical background •  Good knowledge of development •  Knowledge of QA/Ops •  Strong automation skills •  Some systems administration •  Ability to work across silos


9/9/13

60

Toolsmith/Devops

119

•  Strong technical background •  Strong scripting skills •  Diving deep into the tools including troubleshooting •  Understands toolchains and finds flexible solutions •  Process orientation – focus on traceability http://cmbestpractices.com © 2013 April 9, 2013

Goals of this Course •  Understand Continuous Delivery •  Configuration Management roots •  Control Dependencies & Configuration •  Continuous Integration •  Build and Deployment Automation •  Deployment Pipeline is an Art!

Agile Release Train


9/9/13

61

DevOps Focus •  Understand DevOps Best Practices •  A Little History of DevOps •  Scope of DevOps and how to get started •  The People side of DevOps Establish your own plan for DevOps!


And Don't Forget •  Delivery Ecosystem •  Components & Dependencies •  Test vs Verification & Validation (V&V) •  Don't forget the Data •  Establish IT governance and compliance So what is Continuous Delivery?


9/9/13

62

Continuous Delivery

123

•  Methodology for getting software from development to release •  Focus on the Deployment Pipeline •  Rapid incremental deployment •  Minimize Risk •  Many small deployments better than big bang


Continuous Delivery (DevOps Best Practices)

124

Bob Aiello, Principal Consultant and Author of Configuration Management Best Practices : Practical Methods that Work in the Real World

http://www.linkedin.com/in/BobAiello http://cmbestpractices.com

CM Best Practices Consulting © 2013

Technology

Continuous Delivery: Rapid and Reliable Releases with DevOps Practices