From 0 to DevOps in 80 daysLessons learnt from shifting an on-prem to a cloud cultureBernd Greifeneder, CTO

http://dynatrace.com/trial

From the DevOps Webinar with Gene & Mark

Mark TomlinsonPerformance Sherpa

@mark_on_task

Andi GrabnerPerformance Advocate

@grabnerandi

Gene Kim, CTOResearcher and Author

@RealGeneKim

High Performers Are …

200x 2,555xmore frequent deployments faster lead times than their peers

Source: Puppet Labs 2015 State Of DevOps Report: https://puppet.com/resources/white-paper/2016-state-of-devops-report

More Agile

3x 24xlower change failure rate faster Mean Time to Recover

More Reliable

@RealGeneKim

24 “Features in a Box” Ship the whole box!

Very late feedback

F r u s t r a t i o n !

@grabnerandi

@RealGeneKim

Continuous Innovation and Optimization1 “Feature at a Time”

Optimize Before DeployImmediate Customer Feedback

@grabnerandi

2011: APM about to be disrupted!

• Migrate from On-Prem to VM, Cloud, Containers and PaaS• Architectures include micro-services, on-demand scaling,

self-healing• ”Cloud Natives“ demand SaaS based solutions• Bridging the gap between ”New Stack“ and “Enterprise

Stack“• Digital Transformers demand Analytics for Biz, Dev, Ops &

Sec• Many new players on the market

@grabnerandi

From 0 to DevOps in 80 daysLessons learnt from shifting an on-prem to a cloud cultureBernd Greifeneder, CTO

http://dynatrace.com/trial

2 major releases/yearcustomers deploy & operate on-prem

26 major releases/year170 prod deployments/dayself-service online sales SaaS & Managed On-Prem

2011 2016

ManualCICDFcontinuousintegration

continuous delivery & deploymentand feedback

NOC lessons learntContinuous Integration faster than classic OpsAutomation and APIsOne delivery stack across the pipeline

DevOps

Continuous Delivery

is a culture that emphasizes the collaboration of various teams involved in software delivery

is an approach about building, testing and releasing software reliable, faster and more frequent

Utmost goal: minimize cycle time

timefeature cycle time

minimize Users

This is where youcreate value!

feature cycle time time

minimize Users

Utmost goal: minimize cycle time

Deploy faster!!

Failfaster??

Right focus?!

The “War Room”

Facebook – December 2012

ruxit theme 2014.05.15Production facts – Oct 2016

450 AWS EC2 instances

>2 yearsvalue to customers in production

99.998%cluster availability since June’15 NO24/7 OPs Team

170deployments per working day2 weeksrelease cycle

20 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Shift-Left QualityQuality/Performance matters in Dev/Staging as well!Make Dev/CSA/PM dependent from Quality in trunk!

Sure glad the hole isn’t at our end.

Confidential, Dynatrace, LLC

http://dynatrace.github.io/ufo

ruxit theme 2014.05.15more responsibilityfor developers

production toolingmonitoring and diagnosticsproactive analytics/fixingmore security

ruxit theme 2014.05.15Make also architects responsible for production

26 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

iPhone 6

failing early improves quality

late feedbacksucks

27 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Shift-Left QualityQuality/Performance matters in Dev/Staging as well!Make Dev/CSA/PM dependent from Quality in trunk!

DevOps = start thinking like an Ops before Commit

28 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Shift-Left QualityQuality/Performance matters in Dev/Staging as well!Make Dev/CSA/PM dependent from Quality in trunk!

DevOps = start thinking like an Ops before Commit

Shift-Right Metricsenable DEVs defining quality metrics make DEVs to the primary consumers of their metrics

29 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Developer will never do that!Operator’s job

DevOps NoOpsFrom 0 to & Continuous Delivery in 80 days

ruxit theme 2014.05.15call our DevOpsNoOps

automated

ruxit theme 2014.05.15Automation high availability global

Global AWS Infrastructure

ruxit theme 2014.05.15The rise of the chaos monkeys

ruxit theme 2014.05.15

35 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Daily deployments from trunk the whole DEV team depends on

36 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

believe in the mission impossible

6 monthsmajor/minor release+ intermediate fix-packs+ weeks to months rollout delay

sprint releases (continuous-delivery)1h : code to production

New “multimodal” dynamics

Apps of innovation

Apps of records

Apps of EngagementApps of

Engagement

Explore new revenue modelsLook ahead, see beyond present. Non-IT led.

New hyper-scale, hyper-dynamic apps.

Public cloud and open source biasNew cloud platforms and micro-services stacks.

Decisions made in hours/days, no POC.

DevOps AssumedExpect release cycles measured in hours.

Tight teaming between biz, dev, ops.

ruxit theme 2014.05.15

Thank you!

Food for thoughts: • In which of the following stages are you and what's next?

• Classic siloed, Continuous Integration, DevOps?• Is technology or process/culture the hurdle?

• Do you have the right monitoring strategy?

http://dynatrace.com/trial

Confidential, Dynatrace LLC

Performance management for the digital customer age

DevOps Tooling @

Dev Stage Daily

Continuous Delivery & Feedback

Production Stage Bi-Weekly & On Demand

Acceptance Stage Bi-Daily

Deploy

Develop & Fixing

Acceptance & Performance

& Load Tests & Monitor

Deploy

Fixing

Acceptance & Performance

& Load Tests & Monitor

Deploy

Hotfixing

Monitor

Release

Every 2 weeks version is pushed to next stage = 2 weeks release cycle

Unit+Integration testing & Build

Unit+Integration testing & Build

Unit+Integration testing & Build

SaaS

Week 1 Week 2 Week 3 Week 4

Continuous Delivery & Feedback

Dev Stage Daily Production Stage Monthly & On Demand

Acceptance Stage Bi-Daily

Week 1 Week 2 Release

Every 4 weeks version is pushed to next stage = 4 weeks release cycle

Managed - On Premises

Deploy

Develop & Fixing

Acceptance & Performance

& Load Tests & Monitor

Deploy

Fixing

Acceptance & Performance

& Load Tests & Monitor

Deploy

Hotfixing

MonitorUnit+Integration testing & Build

Unit+Integration testing & Build

Unit+Integration testing & Build

Week 3 Week 4 Week 5 Week 6 Week 7 Week 8

CDF Tooling Chain - Dev Stage

Acceptance TestsSelenium/Appium Automated Functional Tests

on UI Level Automated E2E Tests on UI LevelBrowser Compatibility Testing (Chrome, IE, FF, Edge, Safari)24/7 Deployment Checks

Jira Manual Regression TestingAWS, VMware, VirtualBox Test Environments

DevelopGradle + Artifactory build/deployment automationSVN + Git Version ControlEclipse / IntelliJ IDEAQuickbuild + Jenkins Continuous Integration

Unit & Integration Tests (fully automated)Unit Tests +Integration Tests

Quickbuild, Eclipse/IntelliJ IDEA for local execution before commit

Memory/CPU Overhead tests Dynatrace AppMon

Code Quality and Coverage ChecksSonar

Memory leak detectionBullseye, Valgrind

Virus Scan KasperskyOpen Source License compliance checks

Blackduck

Facts/Numbers28.000 Unit Tests + 3.000 integration test executions / hour

Deployment (fully automated)Quickbuild Automated daily deployment of Trunk Builds

using Ansible and Puppet or callingDynatrace Ruxit CloudControl, that uses

AWS Cloud Formation (whole infrastructure as code!)Selenium/Appium

Automated Customer-Like DeploymentDeployment Checks on UI Level

Performance & Load TestsCluster Workload SimulatorEclipse Mem Analyzer (MAT) + Eclipse Thread Dump AnalyzerJava Flightrecorder

MonitoringDynatrace Real-User, Service and

Infrastructure monitoringWebChecks, AWS Monitoring, Log Analytics & Monitoring

OpsGenie and common HipChat Room to escalate detected Problems directly to

DevelopmentSecurity 24/7 OSSEC Host intrusion monitoringDeploy

Develop & Fixing

Unit+Integration testing & Build

Acceptance & Performance & Load Tests & Monitor

CDF Tooling Chain - Acceptance Stage

Deploy

Fixing

Unit+Integration testing & Build

Acceptance & Performance & Load Tests & Monitor

Facts/Numbers (Dev + Acceptance Stage)~700 Automated UI Tests~60 hours UI Test execution per Build ~20 parallel running test sets executed on ~30 execution machines up to ~5 hours per test set~15 different OS (Windows, Linux) Unit & Integration Tests (fully

automated)Unit Tests +Integration Tests

Quickbuild, Eclipse/IntelliJ IDEA for local execution before commit

Virus Scan Kaspersky

Deployment (fully automated)Quickbuild Automated deployment of Sprint Builds using Ansible and Puppet or calling Dynatrace Ruxit CloudControl, that uses AWS Cloud Formation

(whole infrastructure as code!)Selenium/Appium

Automated Customer-Like DeploymentDeployment Checks on UI Level

Acceptance TestsSelenium/Appium Automated Functional Tests

on UI Level Automated E2E Tests on UI LevelBrowser Compatibility Testing (Chrome, IE, FF, Edge, Safari)24/7 Deployment Checks

Jira Manual Regression TestingAWS, VMware, VirtualBox Test Environments

FixingGradle + Artifactory build/deployment automationSVN + Git Version ControlEclipse / IntelliJ IDEAQuickbuild + Jenkins Continuous Integration

Performance & Load TestsCluster Workload SimulatorEclipse Mem Analyzer (MAT) + Thread Dump AnalyzerJava Flightrecorder

MonitoringDynatrace Real-User, Service and

Infrastructure monitoringWebChecks, AWS Monitoring, Log Analytics & Monitoring

OpsGenie and common HipChat Room to escalate detected Problems directly to DevelopmentSecurity 24/7 OSSEC Host intrusion monitoring

CDF Tooling Chain - Production Stage

Deploy

Hotfixing

Unit+Integration testing & Build

Monitor

FixingGradle + Artifactory build/deployment automationSVN + Git Version ControlEclipse / IntelliJ IDEAQuickbuild + Jenkins Continuous Integration

MonitoringDynatrace Real-User, Service and

Infrastructure monitoringWebChecks, AWS Monitoring, Log Analytics & Monitoring

OpsGenie and common HipChat Room to escalate detected Problems directly to DevelopmentSecurity 24/7 OSSEC Host intrusion monitoring

Monthly and on demand Vulnerability Scans (KPMG Linz)Closed Bug Bounty Program at HackerOne

Unit & Integration Tests (fully automated)Unit Tests +Integration Tests

Quickbuild, Eclipse/IntelliJ IDEA for local execution before commit

Virus Scan Kaspersky

Deployment (fully automated)Quickbuild Automated deployment of Sprint Builds using Ansible and Puppet or calling Dynatrace Ruxit CloudControl, that uses AWS Cloud Formation

(whole infrastructure as code!)Selenium/Appium

Automated Customer-Like DeploymentDeployment Checks on UI Level