The Return of the Dull The Return of the Dull Stack EngineerStack Engineer

Kris Buytaert

@krisbuytaert

Kris BuytaertKris Buytaert● I used to be a Dev,I used to be a Dev,● Then Became an OpThen Became an Op● CTO and Open Source Consultant CTO and Open Source Consultant

@@inuits.euinuits.eu● Everything is a freaking DNS Everything is a freaking DNS

ProblemProblem● Evangelizing devopsEvangelizing devops● Organiser of #devopsdays, Organiser of #devopsdays,

#cfgmgmtcamp, #loadays, ….#cfgmgmtcamp, #loadays, ….

C(L)AMSC(L)AMS● CultureCulture

● (Lean)(Lean)

● AutomationAutomation

● MeasurementMeasurement

● SharingSharing

Damon Edwards and John Damon Edwards and John WillisWillis

A global movement to improve the A global movement to improve the quality of software delivery leveraging quality of software delivery leveraging

Open Source experience, started in Gent Open Source experience, started in Gent in 2009in 2009

Because the old days: Because the old days: ● ““Put this Code Live, here's a tarball” Put this Code Live, here's a tarball”

NOW! NOW!

● What dependencies ?What dependencies ?

● No machines available ?No machines available ?

● What database ?What database ?

● Security ?Security ?

● High Availability ? High Availability ?

● Scalability ?Scalability ?

● My computer can't install this ?My computer can't install this ?

10 days into operations10 days into operations

Topic Evolution : CultureTopic Evolution : Culture

● Agile, Kanban, LeanAgile, Kanban, Lean

● Startups as examplesStartups as examples

● UnicornsUnicorns

● Poster ChildsPoster Childs

● Agile, Fake Agile, Less, Agile, Fake Agile, Less, Safe,Safe,

● Enterprises as Enterprises as examplesexamples

● Engage managementEngage management

● BurnoutBurnout

Topic Evolution : AutomationTopic Evolution : Automation

● Cfengine, Puppet, ChefCfengine, Puppet, Chef

● Mcollective, Noah, Mcollective, Noah, RundeckRundeck

● Hudson, JenkinsHudson, Jenkins

● EC2, OpenStack, EC2, OpenStack, Cloudstack, Cloudstack, Eucalyptus,OpenNebulEucalyptus,OpenNebula, openQRMa, openQRM

● Bash in Yaml “Basic”Bash in Yaml “Basic”

● Ansible, Rundeck, Ansible, Rundeck, ChoriaChoria

● Jenkins, GitlabCI, Jenkins, GitlabCI, CircleCI, TravisCICircleCI, TravisCI

● EC2, GCE, Azure, EC2, GCE, Azure, OpenStack,OpenStack,

Topic Evolution : AutomationTopic Evolution : Automation

● Xen, KVM, openVZ, lxcXen, KVM, openVZ, lxc

● VagrantVagrant

● Linux-HA, CoroSyncLinux-HA, CoroSync

● Ceph, Gluster, DRBD,Ceph, Gluster, DRBD,

● Docker , Docker, Docker , Docker, Docker, Docker, Moby, Docker, Docker, Moby, Rkt, lxcRkt, lxc

● CR-IOCR-IO

● Docker , Docker, Docker , Docker, Dockrer,Mesos, Dockrer,Mesos, KubernetesKubernetes

● NFSNFS

Topic Evolution : Monitoring Topic Evolution : Monitoring

● Nagios Nagios

● CactiCacti

● MuninMunin

● RRDRRD

● Zenoss, ZabbixZenoss, Zabbix

● #monitoringsucks#monitoringsucks

● IcingaIcinga

● ELKELK

● Graphite + Grafana Graphite + Grafana

● CollectdCollectd

● PrometheusPrometheus

● #monitoringlove#monitoringlove

Culture vs ToolsCulture vs Tools

These DaysThese Days● ““Put this Code Live, here's a Docker Put this Code Live, here's a Docker

Container ” Container ”

● No machines available ?No machines available ?

● What database ? Where to store the What database ? Where to store the data ?data ?

● Security ? What distro is this even ? Security ? What distro is this even ? Bad Cows ? Bad Cows ?

● How do we monitor his ?How do we monitor his ?

● Backups ?Backups ?

● How did you build this ?How did you build this ?

Tools will not fix your Tools will not fix your broken culturebroken culture

Tooling hype is not helpingTooling hype is not helping

Hype SeasonHype Season

● Conference Visitors numbers skyrocketConference Visitors numbers skyrocket

• OpenStack 3000+ marketeersOpenStack 3000+ marketeers

• DockerCon : 1000+ devs & unicornsDockerCon : 1000+ devs & unicorns

• Kube Con : 1500+ unicorns & vendors Kube Con : 1500+ unicorns & vendors

● Everybody claims to be doing it , nobody Everybody claims to be doing it , nobody is actually doing it.is actually doing it.

Why adopt shiny new stuff ?Why adopt shiny new stuff ?

● Attention Span ?Attention Span ?

● Resume building ?Resume building ?

• 0.5 year = novice in 20 techs0.5 year = novice in 20 techs

vsvs

• 7 year = expert in 4 techs7 year = expert in 4 techs

Works on My MachineWorks on My Machine

New != BetterNew != Better● It’s called Bleeding edge for a It’s called Bleeding edge for a

reasonreason

● Missing FeaturesMissing Features

● OperatabilityOperatability

● StabilityStability

Hello world todayHello world today● Micro service architecture Micro service architecture

● Displaying content in a fileDisplaying content in a file

● Kubernetes 2.6+ StorageKubernetes 2.6+ Storage

● OpenStack OpenStack

Complexity is the Complexity is the enemy of reliabilityenemy of reliability

U are not facebookU are not facebook

U are not LinkedinU are not Linkedin● Kafka is awesomeKafka is awesome

● But you have 100 messages an hourBut you have 100 messages an hour

Welcome to EuropeWelcome to Europe

Why don’t ops adopt ?Why don’t ops adopt ?● They do .. They do ..

• It just takes more timeIt just takes more time

• More worriesMore worries

• More careMore care

Improve your sleepImprove your sleep

Put your {devs,managers,qa} Put your {devs,managers,qa}

ON CALL !ON CALL !

Why don’t ops adopt ?Why don’t ops adopt ?● We already fixed that problemWe already fixed that problem

● We have a different use caseWe have a different use case

● That won’t work because That won’t work because

• AA

• BB

• CC

● Cool lets try it ..Cool lets try it ..

Puppet AdoptionPuppet Adoption● Focus on the wrong partFocus on the wrong part

• Expert in PuppetExpert in Puppet

• Doesn’t understand apache configDoesn’t understand apache config

Operations trying to adoptOperations trying to adopt

$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$● We don’t have timeWe don’t have time

● Initial Budget was too low, Initial Budget was too low, unrealisticunrealistic

● Existing Technical DebtExisting Technical Debt

● FirefightingFirefighting

● Time pressured projectTime pressured project

Operations

Developers

Security

Workload is invertedWorkload is inverted● X devs work on 1-2 concurrent X devs work on 1-2 concurrent

projectsprojects

● X/20 ops/security work on all the X/20 ops/security work on all the projects the devs touched over the projects the devs touched over the past 5 yearspast 5 years

● AND need to build the next gen AND need to build the next gen platformplatform

● Grumpy Grumpy

• Woken up one Woken up one to many times to many times for the wrong for the wrong reasonreason

● Old , aka Old , aka ExperiencedExperienced

Typical Ops DudeTypical Ops Dude

Devops = CoolDevops = Cool● All the cool kids are doing thisAll the cool kids are doing this

● We want to be coolWe want to be cool

● People only want to work for cool People only want to work for cool companiescompanies

● Banks etc are boringBanks etc are boring

FOMOFOMO● We need to do something CoolWe need to do something Cool

• Docker Docker DockerDocker Docker Docker

• Bi ModalBi Modal

Broken StrategiesBroken Strategies

● Hiring Devops EngineersHiring Devops Engineers

● Building Devops TeamsBuilding Devops Teams

● Docker Docker Docker => VM’s as Docker Docker Docker => VM’s as ContainersContainers

s/ops/oops/g;s/ops/oops/g;

Tools will not fix your Tools will not fix your broken culturebroken culture

Tooling hype is not helpingTooling hype is not helping

YOLOYOLO● StartupStartup

● VCVC

● Exit StrategyExit Strategy

● BusinessBusiness

● CustomerCustomer

● SurvivalSurvival

Late AdoptionLate Adoption● You need to do IACYou need to do IAC

● We don’t need IACWe don’t need IAC

● You really need to do IACYou really need to do IAC

● IAC is too difficult for us ..IAC is too difficult for us ..

● Really you need to do IAC Really you need to do IAC

● We’re doing Ansible now …We’re doing Ansible now …

0 day software0 day software0900: “Whow Hashicorp released 0900: “Whow Hashicorp released Consul X.Y”Consul X.Y”

Let’s have this in prod by this Let’s have this in prod by this afternoonafternoon

Ego driven development Ego driven development

R&D vs ProdR&D vs Prod● Schedule time to Research new topicsSchedule time to Research new topics

● Not all of them are irrelevantNot all of them are irrelevant

● But limit the number of patterns you need to manageBut limit the number of patterns you need to manage

● e.g Limited number of OS Variants, Deployment e.g Limited number of OS Variants, Deployment Patterns, Storage BackendsPatterns, Storage Backends

Stop DreamingStop DreamingThere is no way your 3 person ops team can manage There is no way your 3 person ops team can manage and scale VMWare, EC2, Azure, OpenStack, RHEV, and scale VMWare, EC2, Azure, OpenStack, RHEV, Gluster, DRBD, LibVirt,NFS, Ceph, Mongo, MySQL, Gluster, DRBD, LibVirt,NFS, Ceph, Mongo, MySQL, PgSQL, Cassandra, mcollective, Rabbit, collectd, PgSQL, Cassandra, mcollective, Rabbit, collectd, dhcpd, sonar, aptly,ActiveMQ, Kafka, Nomad, dhcpd, sonar, aptly,ActiveMQ, Kafka, Nomad, zookeeper, haproxy, iptables, Consul, Etcd, Puppet, zookeeper, haproxy, iptables, Consul, Etcd, Puppet, Cacti, corosync, Docker Distribution, vault, bind, Cacti, corosync, Docker Distribution, vault, bind, owncloud, openswan,Nexus, Redmine, diamond, owncloud, openswan,Nexus, Redmine, diamond, Jenkins,Mule, GoCD, Chef, memcached, foreman, PHP, Jenkins,Mule, GoCD, Chef, memcached, foreman, PHP, Java,Ruby, nextcloud, Kubernetes, pulp, Spring Boot, Java,Ruby, nextcloud, Kubernetes, pulp, Spring Boot, Ansible, Django, Apache Storm, redis, Nginx, Apache Ansible, Django, Apache Storm, redis, Nginx, Apache httpd, Jboss, Jetty, Hadoop, Elastic, Graphite, Grafana, httpd, Jboss, Jetty, Hadoop, Elastic, Graphite, Grafana, freeswan, Graylog, Apache Spark, Bamboo, Jirafreeswan, Graylog, Apache Spark, Bamboo, Jira

Stop DreamingStop DreamingThere is no way your 3 person ops team can manage There is no way your 3 person ops team can manage and scale and scale VMWare,VMWare, EC2, EC2, Azure,Azure, OpenStack, OpenStack, RHEV,RHEV, Gluster, DRBD, LibVirt,Gluster, DRBD, LibVirt,NFS,NFS, Ceph, Mongo, MySQL, Ceph, Mongo, MySQL, PgSQL, PgSQL, Cassandra,Cassandra, mcollective, Rabbit, collectd, mcollective, Rabbit, collectd, dhcpd, sonar, aptly,ActiveMQ, dhcpd, sonar, aptly,ActiveMQ, KafkaKafka, , Nomad,Nomad, zookeeper, haproxy, iptables, Consul, Etcd, Puppet, zookeeper, haproxy, iptables, Consul, Etcd, Puppet, Cacti,Cacti, corosync, corosync, Docker Distribution,Docker Distribution, vault, bind, vault, bind, owncloud, openswan,Nexus, Redmine,owncloud, openswan,Nexus, Redmine, diamond, diamond, Jenkins,Jenkins,Mule, GoCD, Chef, memcached,Mule, GoCD, Chef, memcached, foreman, PHP, foreman, PHP, Java,Ruby, nextcloud, Kubernetes, pulp, Java,Ruby, nextcloud, Kubernetes, pulp, Spring Boot,Spring Boot, Ansible, Django, Apache Storm, redis, Nginx, Apache Ansible, Django, Apache Storm, redis, Nginx, Apache httpd, httpd, Jboss, Jetty,Jboss, Jetty, Hadoop,Hadoop, Elastic, Graphite, Grafana, Elastic, Graphite, Grafana, freeswan, Graylog, Apache Spark, Bamboo, Jirafreeswan, Graylog, Apache Spark, Bamboo, Jira

Full Stack EngineerFull Stack Engineer● Knows the MEAN StackKnows the MEAN Stack

● Is still confused about Is still confused about the Javascript the Javascript framework he wants to framework he wants to use yesterdayuse yesterday

● Is familiar with all the Is familiar with all the previously mentioned previously mentioned toolstools

● Understands the Linux Understands the Linux KernelKernel

● Debugs Java CodeDebugs Java Code

● Understands Networking Understands Networking and Securityand Security

● Understands deploymentUnderstands deployment

● Tunes DatabasesTunes Databases

Dull Stack EngineerDull Stack Engineer

Dull Stack EngineerDull Stack Engineer● Knows the stack in depthKnows the stack in depth

● Invests in people Invests in people

● Knows his coworkers in depthKnows his coworkers in depth

● Is there for the business, not for the Is there for the business, not for the fun of the new toolfun of the new tool

$value$value● Devops is about generating value Devops is about generating value

for your organisation , togetherfor your organisation , together

● What’s the value of shiny $tool vs What’s the value of shiny $tool vs boring tool ?boring tool ?

Back to the rootsBack to the roots● Collaborating for a better platformCollaborating for a better platform

● Open SourceOpen Source

● EmpathyEmpathy

● Common SenseCommon Sense

ContactContactKris Buytaert kris.buytaert@inuits.euKris Buytaert kris.buytaert@inuits.eu

Further ReadingFurther Reading@krisbuytaert @krisbuytaert http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/http://www.inuits.eu/http://www.inuits.eu/

Find Inuits inFind Inuits in

Brasschaat,GhentBrasschaat,Ghent,,Rotterdam,PraguRotterdam,Prague,e,Kiev,BrnoKiev,Brno