Docker CoreOSSolving a Real Pain in the PaaS

Ensemble+

THE PROBLEM

I present to you...

Lot’s of Tools, Few Solutions

● No Master Blueprints or Distributions

● Docker is Not Enough

● Docker Hub is a Mess

● Fleet has Obvious Limitations

● Managing Systemd Units is Cumbersome

● Logging is a Real Pain

● Data Persistence is Necessary

● Kubernetes & Mesos are Overkill *

Chatty Micro Services

But You Run Fat Stacks...

Redis, Memcache, Varnish, MongoDB,

CouchDB, Storm, Jenkins, NodeJS, MySQL,

Postgres, Apache, Nginx, Tomcat, Mongrel,

Mongrel2, Sinatra, Riak, Graphite, Cassandra,

Hadoop, ActiveMQ, Solr, RabbitMQ,

Elastic Search, Logstash, Kibana,

HAProxy, Postfix, and Graphite

What Not To Do

☒Manually Create Unit Files

☒ Blindly Use Docker Hub Images

☒ Run Data Volumes without a Backup Plan

☒Manual Configuration or Conf Management

☒Modify CoreOS Outside of Cloud Init

☒ Execute Multiple Processes per Container

☒ Expose Lots of Public Ports

Docker Madness

Ehh?? What am I looking at?68ce1cebbf07 741583d7dbd1 "/usr/sbin/apache2ct 10 days ago Up 10

days 80/tcp tp_web-2

3f26d14fe567 0eeeca18f709 "/bin/bash /sync.sh 10 days ago Up 10

days tp_web_backups

a77bcb74d75c 741583d7dbd1 "/usr/sbin/apache2ct 10 days ago Up 10

days 80/tcp tp_web-1

98f15fac74ca 0eeeca18f709 "/bin/bash /sync.sh 10 days ago Up 10

days tp_mysql_backups

8019d38dd49b cloudposse/library:btsync "\"/bin/sh -c '/usr/ 10 days ago Up 10

days 44444/tcp tp_btsync

652e815800d7 cloudposse/library:mysql "/entrypoint.sh mysq 10 days ago Up 10

days 3306/tcp tp_mysql

21a18d390135 0eeeca18f709 "/bin/bash /sync.sh 10 days ago Up 10

days vps_akira_backups

751f682f9606 1c3ff222069c "/bin/sh -c 'bootstr 10 days ago Up 10

days 0.0.0.0:30000->22/tcp vps_osterman

3da29f6eda58 1c3ff222069c "/bin/sh -c 'bootstr 10 days ago Up 10

days 0.0.0.0:30002->22/tcp vps_akira

02c

Fleet-ing Commands

WTF Fleet is Not much better…osterman_web@1.service 3ea56e45.../167.114.0.38 active

running

osterman_web@2.service 3ea56e45.../167.114.0.38 active

running

osterman_web_announce.service 3ea56e45.../167.114.0.38 active

exited

osterman_web_backups.service 3ea56e45.../167.114.0.38 inactive dead

registrator_skydns.service 3ea56e45.../167.114.0.38 active

running

tp_btsync.service 3ea56e45.../167.114.0.38 active

running

tp_mysql.service 3ea56e45.../167.114.0.38 active

running

tp_web@1.service 3ea56e45.../167.114.0.38 active

running

tp_web@2.service 3ea56e45.../167.114.0.38 active

running

vps_akira.service 3ea56e45.../167.114.0.38 active

running

vps_akira_backups.service 3ea56e45.../167.114.0.38 active

running

Sample Apache Systemd Unit File

[Unit]

Description=Apache Web Server

After=docker.service

Requires=docker.service

[Service]

User=core

EnvironmentFile=/etc/environment

TimeoutStartSec=0

ExecStartPre=-/usr/bin/docker kill ib_web-1

ExecStartPre=-/usr/bin/docker rm ib_web-1

ExecStartPre=/usr/bin/docker --debug=true pull \

registry.hub.docker.com/cloudposse/library:apache-php

(Continued)

ExecStart=/usr/bin/docker run --name ib_web-1 \

--rm \

--dns=${COREOS_PRIVATE_IPV4} \

--dns-search=ib.cloudposse.local \

--volume /media/sdb/ib_web:/var/www/html \

-e "DB_USER=wordpress" \

-e "DB_PASS=XXXXXXXX" \

-e "DB_HOST=web.mysql.ib.cloudposse.local" \

-e "DB_NAME=wordpress" \

-e "SERVICE_NAME=ib/web" \

-e "SERVICE_ID=1" \

registry.hub.docker.com/cloudposse/library:apache-php

ExecStop=-/usr/bin/docker kill ib_web-1

ExecStop=-/usr/bin/docker rm ib_web-1

The Missing Pieces

♟Fleet Orchestration

♟Systemd Unit Templates

♟Basic Building Blocks

♟Naming Conventions

♟ Introspection

♟Overlay Networking

Low and behold!

THE OPPORTUNITY

What if I told you….

There was a way you could quickly spin up

docker containers for everything, running

coreos in the cloud or bare metal, with

multiple environments or stages, using

simple commands on your laptop

it would just work? =)

Well….

YOU CAN

Entertain...

THE POSSIBILITY

The Blue Print

● Ensemble Services Distribution✓ 60+ Essential Services (e.g nginx, varnish, mysql)

✓ HTTP Router with Vulcand

✓ RFC1918 Overlay Network

✓ Service Composition, Orchestration

✓ Library Unit Templates, Images

✓ DNS Based Service Discovery & Registration

✓ Logging to PaperTrail, DataDogHQ, LogEntries

✓ Remote VPN Access

✓ Backups & BitTorrent Data Replication

Ensemble Command Line

Usage: bin/ensemble options [action] [service] [unit]

Some of the Supported actions:

build Compile the m4 templates for each service

commit Execute "git commit" modified units for each service

push Execute "git push" to master committed units for each service

kickstart Shortcut for stop+destroy+start specified units

restart Shortcut for stop+start specified units

view Display the current unit definition

template Display the template definition

status Display status of all units in cluster

stats Display realtime performance metrics of specified units in cluster

journal Stream stdout logs for specified unit from each server in the cluster

shell Attach to the container and execute a shell (/bin/bash)

Actions can be combined using +

Command Line Examples

# Build all configurations for all services

ensemble --config deployment.yml build

# Build and deploy only the “ib” service

ensemble --config deployment.yml build+kickstart ib

# Give me the operating status of the “ib” service

ensemble --config deployment.yml status ib

# Open up a shell inside the web container

ensemble --config deployment.yml shell ib web@1

# Display real-time memory and network utilization of the “ib” service

ensemble --config deployment.yml stats ib

# Commit changed configurations and push them to master

ensemble --config deployment.yml commit+push ib

Tons of Ready Made Templates

datadog.service.m4, logentries.service.m4, logspout.service.m4,

boundary_agent.service.m4, newrelic_sysmond.service.m4,

nodejs.service.m4, apache.service.m4, openvpn.service.m4,

docker_registry.service.m4 duplicity.service.m4,

haproxy_with_discover.service.m4, os_swap.service.m4

jenkins_master.service.m4, jenkins_swarm.service.m4,

kiwiirc.service.m4, memcache.service.m4, skydns.service.m4

mysqldump.service.m4, mysql_galera.service.m4, generic.timer.m4,

mysql_galera_cluster.service.m4, mysql_galera_garbd.service.m4,

mysql_galera_lb.service.m4, mysql_server.service.m4,

btsync.service.m4, postfix.service.m4, redis.service.m4,

registrator.service.m4, vps.service.m4, vulcand.service.m4,

os_mount.service.m4

Ensemble Configuration (YAML)fleet_tunnel: "cluster.cloudposse.local" # Cluster to administer

fleet_ssh_timeout: 20 # How long to wait before giving up

template_path: "~/Dev/ensemble/units" # Where to find the templates

template_engine: "m4" # How to rewrite the templates

target_path: "~/Dev/cluster-configs/units" # Where to send the rendered units

globals: # Global variables can be used anywhere in the YAML

config

cluster_name: "cluster-1" # Sample definition

container_storage: "@(mount_dir)/sdb" # Where we’ll store persistent data (it will be backed

up!)

mount_dir: "/media" # Mount point on CoreOS for persistent storage

services: # Services are composed of a collection of units

- include: "services/core.yml" # CoreOS extensions deployed via units

- include: "services/jenkins.yml" # Jenkins CI with Jenkins Swarm

- include: "services/osterman.yml" # My own website and virtuals

- include: "services/tp.yml" # TP Deployment

- include: "services/isc_staging.yml" # Staging environment for ISC

- include: "services/isc_prod.yml" # Production environemnt for ISC

Pretty Service Configurations

# Osterman Associates

name: "osterman" # The name of this service

globals: # Local service globals # Set/Override New Globals

mysql_user: "wordpress"

mysql_pass: "XXXXXXXXXX"

mysql_host: "mysql.@(service).@(local_domain)"

mysql_port: 3306

mysql_database: "wordpress"

mysql_root_pass: "XXXXXXXXXXXXXX"

mysql_backups_volume: "@(container_storage)/@(service)_mysqldump:/backups"

mysql_volume: "@(container_storage)/@(service)_mysql:/var/lib/mysql"

web_volume: "@(container_storage)/@(service)_web:/var/www/hosts"

Minimal Unit Configurations# Database instance for wordpress

- name: "mysql" # Name of this unit

template: "mysql_server.service.m4"

type: "service"

define:

"DOCKER_VOLUME": "@(mysql_volume)"

"DOCKER_IMAGE": "@(docker_registry)/cloudposse/library:mysql"

"MYSQL_USER": "@(mysql_user)"

"MYSQL_PASS": "@(mysql_pass)"

"MYSQL_HOST": "@(mysql_host)"

"MYSQL_DATABASE": "@(mysql_database)"

Clustering Made Easy

# Apache web service for wordpress multisite

- name: "web@%i"

template: "apache.service.m4"

type: "service"

instances: 5

# Define all the virtual hosts referencing another definition

- name: "vhost@www.osterman.com"

from: "vhost@%i"

- name: "vhost@osterman.com"

from: "vhost@%i"

Screencast

● Overview

● Sample Configurations (YAML, m4)

● Demonstration of COOL Commands

● Data Dog, Paper Trail, Boundary

BAM! That’s all I got...

Ask Erik Osterman

erik@cloudposse.com

(415) 535-8615