From pets to cattle - powered by CoreOS, docker, Mesos & nginx

MUC_DATE_THEME_INITIALS

From pets to cattle –powered by CoreOS, docker, Mesos & nginxThomas Schneider

MUC_DATE_THEME_INITIALS 2

Ι IntroΙ Pets vs. cattleΙ Where we started ...Ι Learnings & dead-endsΙ ... where we ended upΙ Technology stackΙ Showtime, baby!

Agenda


Intro

Ι Background§ ~10 years experience in IT§ Mainly software development§ Some system engineering, automation & operations§ @zooplus:

§ Lead engineer, build & runtime platform§ Trying to encourage DevOps culture

Ι Get in touch§ [email protected]§ github.com/schneidexe


Pets vs. cattle


Where we started …

Ι Done via shellΙ .sh scripts, scp, rsyncΙ Mainly .jar/.war filesΙ 2 out of 3 apps

deployed differentlyΙ Locally != dev != prodΙ Dev create & test, but

no idea how it runsΙ Ops deploy & run, but

have no idea what

Ι Monolithic code-baseΙ CI partly automatedΙ Manually configured

teamcityΙ 10+ all (=one) purpose

agentsΙ Up to 10+ builds in

parallelΙ Maven binary repository

(nexus)Ι Source repository

(subversion)

Build Deploy

Ι 300+ apps/jobsΙ Lot’s of bare-metalΙ Static environmentΙ Servers used by multiple

apps, upgrades blockedΙ Outdated machinesΙ Monitoring: ELK,

collectd (hard to get all)Ι High error noise levelΙ Ops == FirefightersΙ Devs blocked by InfraΙ Biz scaled, errors $$$

Run

Application stack is limited and tailored to environment. Build and release process slow, error-prone and inflexible. Intransparent infrastructure, extension is slow & painful.


Where we started …


Learnings & dead-ends

Ι Handle diversity§ Horizontal scaling is not our primary issue§ Handling tons of different apps it is!

Ι Immutability & automation§ Everything is (made from) code, build immutable artifacts§ Automation is not only for speed but for knowledge§ Re-creation over re-configuration

Ι Bring the pain forward§ Do not do it all by yourself, get infra and devs (and biz) on-board§ Think reverse: from prod to dev

Ι Keep it simple and fast§ Get buy-in from devs, they have to understand it§ Teams had/have to adopt several times (after all that years of stable infra!)§ Simple is fun, Speed is fun - if it’s fun people will use it



Ι Docker helps, but does not solve all of your problems§ It’s not just about ‘docker build’ & ‘docker run’§ Think about scaling, monitoring and management of your apps on prod

Ι Stay focused§ What do YOU need? (not Google, Netflix & co)§ There are new products around every week, do 2-3 POCs, then stick with your descision§ Keep it modular, have a plan in mind how to migrate/replace parts§ Be not scared of throwing away a few things

Ι Persistence§ Try not to mix stateful and stateless things, externalize data§ A database might be more a pet than cattle



Ι Puppet§ Not the best tool for deploying your app§ Re-configuration can be tricky (systems become almost identical)§ Not immutable (unless you really nail every dependency)

Ι Fleet (0.9.x)§ nice features like side-kicks, low overhead§ no resource management, too low level§ stability issues

Ι Mixing frameworks on Mesos agent nodes§ Isolation can get tough if patterns are too different e.g. jobs and services§ Not enough resources for big jobs on service nodes§ Spike utilization of batch jobs or builds can impact overall host performance

Ι Graphite and containers§ cannot handle metrics with too much dynamics (30k different containers in 1 week)


... where we ended up

Ι GUIs and REST APIsΙ Deploy

§ Services§ Jobs§ Containers§ Machines

Ι Unified deployment & management

Ι Cloud-agnosticΙ You build it, you run it!*

Ι Distributed code-baseΙ full CI/CD life-cycle can

be automatedΙ Pre-configured jenkins

masterΙ Disposable, customized

jenkins slavesΙ Scalable buildsΙ Multi-format binary

repository (Artifactory)Ι Source repository (git)

Build Deploy

Ι Flexible resourcemanagement

Ι Health-checks & self-healing

Ι Environment configΙ Service discovery &

routingΙ Horizontal scalingΙ House-keepingΙ Out-of-the-box

monitoring (metrics, logs)

Run

Build, deploy and run any application with high flexibility & low effort (Jenkinsfile, Dockerfile, Deployment .json). Same release process for all applications. High transparency on infrastructure*.


... where we ended up


Technology stack: Service Discovery & Routing, PaaS, CaaS, IaaS

SD/SR Nixy Mesos-DNS

PaaSMarathon Chronos Jenkins

Mesos Zookeeper

CaaS

IaaSdeploy-API + cloud-init


Technology stack: Monitoring

SD/SR

journal+

filebeat

dockerbeatPaaS

CaaS

hostbeat

IaaSvCenter


Technology stack: Deploy API

Ι Fast cloud-like provisioning of VMs (resources: cpu, mem, disk, net)Ι Lightweight bootstrapping

with cloud-initΙ Focus on cattle machinesΙ Re-create over re-configure

$ curl -X POST \--form "image=coreos-1081.3.0" \--form "application=docker" \--form "env=dev112" \--form "cpu=8" \--form "mem=16" \--form "disk=100" \--form "[email protected]" \"http://deploy.zooplus.de/api/v1/machines"

coreos:units

- name: docker.servicedrop-ins:

- name: docker-opts.confcontent: |

[Service]Environment='DOCKER_OPTS=host=tcp://0.0.0.0:2375'


Technology stack: Docker

Ι Automated reproducible builds with DockerfileΙ Immutable imagesΙ Bundles app and

dependenciesΙ Common artifact format

Ι Standardized way of deployment, monitoring, etc.Ι Isolation of applicationsΙ Resource allocation

$ cat DockerfileFROM repo.zooplus.de/centos:7RUN yum install –y java-1.8.0_47 && \

yum clean allADD shop.jar /shop.jarCMD java –jar shop.jar

$ docker build –t shop . Building image shopStep 1 : FROM repo.zooplus.de/centos:7---> 9b92a6d1f7de

...

$ docker run shopStarting shop...


Technology stack: Mesos

Ι Resource managerΙ Task distribution Ι “Whole DC as single machine”Ι All tasks run in docker

containers

Ι Web UI for status/utilization and debugging (logs, task state)Ι Usually no direct interaction


Technology stack: Jenkins

Ι Automation engineΙ Jenkins 2.xΙ Jenkinsfile and multi-

branch supportΙ Post-commit hooksΙ Immutable slaves

§ Running on mesos/docker§ Customized§ Highly scalable

Ι Jenkins master docker image § Spawn test instance in <1min

(builds should run on prod)§ Bootstrap with DSL

folder(”catalog") { }multibranchWorkflowJob(catalog/app') {

branchSources {git {

remote('ssh://[email protected]:22/cat/app.git')credentialsId('ef406810-be3c-4f2a-ad65-6239706d1766')

}}

}


Technology stack: Marathon

Ι Task scheduler for mesosΙ Distributed init systemΙ Long runnnig apps/services

Ι Rest API: submit apps via .json

Ι GUI: manage apps & manual config

Ι Health checks & self-healingΙ Multi-app deploymentsΙ Rolling updatesΙ Horizontal scaling


Technology stack: Chronos

Ι Task scheduler for mesosΙ Distributed cron systemΙ Batch jobs

Ι Rest API: submit jobs via .json

Ι GUI: job details & status, manual execution

Ι Scheduling§ Time-based§ Dependency-based


Technology stack: Nixy/Nginx/Mesos-DNS

Ι Nixy§ Service catalog from marathon§ REST-like API§ Event-based§ Configures nginx based on

templatesΙ Nginx

§ State-of-the-art web server§ Used as service router§ SSL termination§ Proxy for HTTP, TCP and UDP§ Access control & public

exposureΙ Mesos-DNS

§ Service catalog from mesos§ Convention-over-configuration

naming pattern§ used for “internal” services

"Apps": {"/finance/jenkins": {"Tasks": [["ops85-150.web.zooplus.de:20357"],["ops85-150.web.zooplus.de:20358"]

],"Frontends": [{"Type": "http","Data": ["finance-jenkins"]

}]

}}

$ host jenkins-finance.marathon.prod.zooplus.netjenkins-finance.marathon.prod.zooplus.net has address 192.168.85.150


Technology stack: journal & beats

Ι Hostbeat§ Ships host metrics in beats

format§ Like collectd

Ι Dockerbeat§ Ships container metrics in

beats format§ Metadata: env, labels

Ι Journal/Filebeat§ Ships every single log line from

journald to ELK§ Docker uses journal log-driver

to ship stdout/stderr§ Apps should log in JSON-lines

Ι ELK/Graphite§ Elastic search: event-data§ Graphite: TSD/metrics

Ι Nagios


Technology stack: Cluster structure


Technology stack:Questions?


Showtime, baby!


Thank You!

… and yes, we’re hiring! ;)

Data & Analytics

From pets to cattle - powered by CoreOS, docker, Mesos & nginx