DevOps
Daniel Zoltak & Shiva N
Use Cases, Architectures & Best Practices
Agenda
• Culture
• Technology
• Your experiences
• Breakdown the barriers • Work as one team end to end • Support business and IT agility • Collaboration & Communication • Treat Infrastructure as code • Automate everything • Test everything • Measure & monitor everything
Agenda…
Culture
Technology
Competing Forces
Business
Development Operations
Build it faster Keep it stable
Change is inevitable
• Change is the root cause of most outages
• This leaves two approaches: - Make it a big deal
- Small iterative non events
“Do painful things more frequently, so you can make it less painful…”
- Adrian Cockcroft, Architect, Netflix
Deploying More Frequently Lowers Risk
Smaller Effort “Minimized Risk”
Frequent Release Events: “Agile Methodology”
Time
Cha
nge
Rare Release Events: “Waterfall Methodology”
Larger Effort “Increased Risk”
Time
Cha
nge
Knowledge is power
Requirements Gathering Release Development
and Check In Test and QA
Some Learning
Lots of Learning
Minimal Learning
Cycle Time
“How long would it take your organization to deploy a change that involves just one single line of code?” “Do you do this on a repeatable, reliable basis?”
- Mary & Tom Poppendieck, Lean Software Development
Waterfall
Analyze Design Build Test Deploy
Agile Sprint
Test Deploy
Agile-er
A D B A D B
Sprint Sprint
A D B
Sprint
A D B T D
Sprint Sprint
A D B T D A D B T D
Agile-er Dev QA Ops
Dev QA Ops
Dev/QA/Ops Dev/QA/Ops Dev/QA/Ops
What is it?
• A philosophy? Cultural change? Paradigm shift ?
• Alignment of development and IT operations with better communication and collaboration ?
• Improvement in software deployment ?
• Breaking down the barriers between development and IT operations ?
• Akin to Agile software development applied to infrastructure and IT operations
• Set of tools and processes • It’s all of the above!
Code
Wal l
Developer IT Operations
DevOps
Building Teams
• Make DevOps everyone’s job - No silo/walled off DevOps teams
• Encourage developers to participate openly in the automation of operations code
• Embolden operations participation in testing and automation of application code
• “Take pride” in how fast and frequent you deploy
DevOps Team Anti Patterns
Dev Ops
Separate Silos
Dev Ops DevOps
Separate DevOps Silo
Dev Ops DevOps
We don’t need Ops http://www.slideshare.net/matthewskelton/devops-patterns-team-topologies
DevOps is a Re-Org! - Adrian Cockcroft, Architect, Netflix
RE
DevOps Team Patterns
Dev Ops
Collaboration Dev Ops
Embedded
Dev Ops
DevOps-as-a-service
DevOps Dev Ops DevOps
Temporary DevOps team
http://www.slideshare.net/matthewskelton/devops-patterns-team-topologies
Business Case Requirements Use Case Features Plan Go to market
Business
Design Code Refactor Unit Test Bug Fix Deliver
Developers (application)
Provision Configure Orchestrate Deploy Report Monitor
IT Operations (infrastructure)
Agile Development
DevOps • Continuous Integration • Continuous Deployment • IT Automation • Application Management
• Iterative development • Scrum, sprints, stories • Velocity
Business Agility
IT Agility
Business and IT agility
Collaboration and Communication
Standups Pairing
Rotation Visibility
Everyone knows what production looks like.
Building a DevOps Culture
• Encourage a fail fast, learn quick mindset • Foster innovation and accountability • Promote open and honest sharing of lessons
learned • Build trust across organizational boundaries
- Include all the stakeholders
• Expect that it will be a work-in-progress
The DevOps Stack
The DevOps Stack
Continuous Deployment
Delivery Pipelines
Deployment Automation
Continuous Integration
Automated Testing
Configuration Management
Agi
le
Com
mun
icat
ion
• Infrastructure as code
• IT Automation
• Continuous Integration - Application
§ Compile, test, optimize (code coverage)
- Infrastructure
§ Logical, valid, secure
DevOps Practices
• Continuous Deployment - Application - Rollout & Rollback
• Version control integration
• Application and Infrastructure version management
• Monitoring and logging
DevOps Practices
Continuous Integration
• Every developer check-in initiates a build - Builds are fully automated
- Tests validate every check-in
- Always check-in to the trunk
• Instant feedback for developers - The build must remain fast
• Minimizes the impact of broken builds - Bugs don’t accumulate
• Drives the culture of small, frequent releases
Continuous Integration Tools
Automated Testing
• Robust, automated testing drives confidence in the entire process
• Testing is not viewed as a “phase” of a project - Testing starts on day one
• View failures as successes • Don’t overlook testing the infrastructure
- Are you ready for the Simian Army?
• Testing is everyone’s responsibility
Configuration Management
• Everything must go into version control
• Implement an effective strategy for branching and merging
• Don’t let tools get in the way
Deployment Automation • Invest in streamlining and unifying all aspects of the
deployment process • An application’s environment and configuration should be
managed through code - Remove any dependencies on human intervention
• Every manual step represents an opportunity for future failure - Console tweaks = future bugs
• Value repeatability - How fast can you completely rebuild your environment from scratch?
One Step Environment Creation
• Require a single build process for all environments - Try to build on horizontal concepts
• Expect the environments to be constantly evolving - Fragile automation becomes fragile environments
• “The longer you wait to have a common environment build process, the harder it is to create one”
Version Control
Build/ Compile
Code Dev
Unit Test App Code
IT Ops
DR Env
Test Env
Prod Env
Dev Env
Application Write App
Code
Infrastructure
CloudFormation
tar, war, zip yum, rpm Deploy
App
Package Applicatio
n
Deploy application only
Deploy infrastructure only
AMI
Build AMIs
Validate Template
s
Write Infra Code
Deploy Infras
Automate Deployme
nt
Artifact Repository
Continuous Integration / Deployment & Automation
Delivery Pipelines
Commit Unit Test
System Test
• Defines the automated lifecycle of delivery • Enables collaboration between the groups • Provides everyone with visibility into the flow of change
Commit Unit Test
System Test
UAT Staging Prod Commit Unit Test
System Test
Continuous Deployment
• Code that makes it through the pipeline goes straight to production
• Requires continuous integration and delivery
• Relies heavily on small, constantly tested releases
• Often a stretch goal for many organizations
Continuous Deployment vs Delivery
• Deploy in-place - Deploy all at once (Service outage)
- Rolling updates
Deployment Approaches
Demo
• Blue-Green Deployment - Discrete environment
- Multiple environments from branches
- Support A/B testing
- “Rolling DNS”
• Alternate Blue-Green (Red-Black?) deployment - Attach and detach Auto Scaling groups from Elastic Load balancers
- Avoid messing with DNS
Deployment Approaches
Blue-Green Deployment
Amazon Route 53
EC2 Instances
ELB
100%
DynamoDB MySQL RDS
Instance ElastiCache Cache Node
• Stand up duplicate infrastructure and slowly cut traffic over to it - Shift via DNS - Simplifies feature testing - Simplifies roll back
• As we shift more traffic over, let auto-scaling grow/shrink our instances - Shut down the old when no traffic there
Blue-Green Deployment
Amazon Route 53
EC2 Instances
ELB
EC2 Instances
ELB
90% 10%
DynamoDB MySQL RDS Instance
ElastiCache Cache Node
• Stand up duplicate infrastructure and slowly cut traffic over to it - Shift via DNS - Simplifies feature testing - Simplifies roll back
• As we shift more traffic over, let auto-scaling grow/shrink our instances - Shut down the old when no traffic there
Blue-Green Deployment
Amazon Route 53
EC2 Instances
ELB
EC2 Instances
ELB
0% 100%
DynamoDB MySQL RDS Instance
ElastiCache Cache Node
• Stand up duplicate infrastructure and slowly cut traffic over to it - Shift via DNS - Simplifies feature testing - Simplifies roll back
• As we shift more traffic over, let auto-scaling grow/shrink our instances - Shut down the old when no traffic there
Blue-Green Deployment
• Stand up duplicate infrastructure and slowly cut traffic over to it - Shift via DNS - Simplifies feature testing - Simplifies roll back
• As we shift more traffic over, let auto-scaling grow/shrink our instances - Shut down the old when no traffic there
Amazon Route 53
EC2 Instances
ELB
EC2 Instances
ELB
0% 100%
DynamoDB MySQL RDS Instance
ElastiCache Cache Node
ASG V1
ELB
Red-Black Deployment
RDS
Red-Black Deployment
ELB
ASG V1 ASG V2
UAT
RDS
Red-Black Deployment
ELB
ASG V1 ASG V2
RDS
Red-Black Deployment
ELB
RDS
ASG V2
Agile Everywhere
• Agile tenets are core to all dimensions of the DevOps lifecycle
- Small, iterative releases
- Quality is everyone’s job
- Customer focused
- Fast, constant feedback
- Done means “released”
Automation Everything
• Automation is vital to smaller, more frequent releases • All steps that add human intervention also add
fragility - “Small compromises can undermine your success”
• Use knowledge gathered from frequent, automated releases to constantly evolve the process
If you have to login to production, your DevOps-fu is sub optimal
• Scalability (anything manual is not scalable) • Reliability • Reproduction/Duplication
• Environment consistency • Auditability/Record Keeping
• Security • Governance
Infrastructure as code – why?
"WebServer": { "Type": "AWS::EC2::Instance", "Metadata" : { "AWS::CloudFormation::Init" : { "config" : { "packages" : { "yum" : { "httpd" : [], "php" : [], "php-mysql" : [], "php-gd" : [], "php-xml" : [], "php-mbstring" : [], "mysql" : [] } }, "sources" : { "/var/www/html" : "http://ftp.drupal.org/files/projects/drupal-7.8.tar.gz", "/home/ec2-user" : "http://ftp.drupal.org/files/projects/drush-7.x-4.5.tar.gz" },
AWS CloudFormation
template
Here’s some infrastructure as Code
Describing Infrastructure with Code
Developers &
Operations Internal Git CI Server
Pre-commit Hook
Testing Environment Subnet
CI Workers
Dev Environment VPC Subnet
DEV WEB ELB Dev Stack
Tier 1 Dev Stack
Tier 2
Dev MySQL DB Instance
DEV APP ELB
VPN TUNNEL
VPN facing VPC Subnet
Internet Gateway
VPN Endpoint
Dev Admin Instance
NAT Instance
Amazon S3
Amazon DynamoDB
Amazon SQS
Amazon CloudFront
Amazon Route 53
This entire infrastructure stack can be constructed, configured, and deployed with code:
Infrastructure Code
The code that describes infrastructure should inherit the same values applied to application code
• Not JUST revision control • Make use of bug tracking/ticketing systems • Peer reviews of changes before they happen • Establish infrastructure code patterns/designs • Test infrastructure changes like code changes
Declarative Approach to: • Provisioning • Configuration • Orchestration • Reporting
Elastic Beanstalk
CloudFormation
OpsWorks
Automation and configuration management
Automation Tools AWS Tools
• CloudFormation • Elastic Beanstalk • OpsWorks
Third-Party Tools • Chef • Puppet
Monitoring & Metrics : The DevOps Pulse
• Track every resource possible • Alert on services, availability, and response times • Capture, learn, and improve
• Share access with everyone on the team • Plan metrics and monitoring into the DevOps lifecycle
If it moves graph it, or it didn’t happen
Monitoring/Logging Tools • Logging
- Logstash Ø Check out ELK!
- Sumo Logic - Splunk - CloudWatch logs
• Metrics - CloudWatch - Ganglia - StatsD + Graphite + Grafana - NewRelic
- Monitoring - Nagios - Munin - Sensu - Hyperion
Discussion
Thank You