Perspectives on Docker10 things not to forget before

deploying Docker in production

Docker Meetup @ RightScaleRaphael Simon & Thorsten von Eicken

October 21st 2014

Docker from Theory to ProductionHow to cruise …… and what to avoid

Containers vs. Virtual MachinesDifferences:● Size● Boot time● Performance● Isolation● Compatibility

Containers vs. ProcessesM

EM

PG

Mre

gs

proc

textbook process

ME

Mre

gs

proc

/etc

/lib

/bin

real process

ME

Mre

gs

proc

/etc

/lib

/bin

container

net

⇒Containers are processes with env, not mini-VMs

Docker Benefits @RightScale

1. Dev & ops contract

Dev responsible for app containers contents

Ops responsible for container surrounds

Not against devops, but ops handles >30 apps

2. App portability

Make it easier for our customers to run an app anywhere

VMs need to be customized for each cloud (or bare metal)

Rather than installing apps each time, just drop down containers

Host 1

VM 1Tenant A

Containers in a VM

Container 1Runs app 1Tenant A

Container 2Runs app 2Tenant A

VM 1Tenant B

Container 3Runs app 3Tenant B

Container 4Runs app 4Tenant B

● Containers are produced by development

● VMs are produced and managed by ops

● Hosts are managed by the cloud provider

Do not trust containers to provide a hard security boundary

10 Things not to Forgetbefore deploying Docker in production

Docker captures stdout/stderrdocker logs command prints combined stdout/errdocker logs -f : running tail (from the beginning!)

1. Logging – how Docker does it

$ docker run --name hello -d busybox echo hello Santa Barbaraa3c0caa675e106cc0cf208dade762afcc341ed5b9ea8f3d75b6e2092745a5faa$ docker logs hellohello Santa Barbara$

1: Logging – how not to do it● Log to stdout/stderr and collect them in the VM

○ Not all apps log to stdout/err, many don’t add timestamps○ No log rotation (can use logrotate with copytruncate)○ No tailing to ship the logs

● Run syslog daemon inside the container○ Containers ≠ VMs○ Configuration hell

1: Logging – solutions● Bind-mount /tmp/dev -> /dev

○ Can’t bind-mount /dev/log!○ Move /dev/log to /tmp/dev/log○ See http://jpetazzo.github.io/2014/08/24/syslog-docker/

● Fix docker daemon to handle logging○ Fixing stdout/err is happening (#7195)○ Ready to add support for syslog source, but not active

container dockersyslog

filestdout/err json

2: Monitoring – how not to do it● Monitoring daemon inside

each container○ Container ≠ VM○ Monitoring daemons require privs○ Configuration/management hell

Monitoring – how to do it● Collect stats in VM using container-aware monitoring

○ Stats are in /sys/fs/cgroup/…See: Docker doc article on run-time metrics

○ Docker support: cAdvisor, DataDog, … ?

● Or just monitor at the process level$ docker run --name hello -d busybox sleep 603a804b088b432035c5cee541f4baef3cc728d27dded3378fd253c6b4abeb077a$ cat /sys/fs/cgroup/cpuacct/docker/3a804b088b432035c5cee541f4baef3cc728d27dded3378fd253c6b4abeb077a/cpuacct.usage_percpu630924 4774818 7494614 3622216

3. Secrets – how not to do it

DEMO

3. Secrets – Solutions

$ cat Makefile# Setup context then build imagebuild: Dockerfile

git clone git@github.com:rightscale/docker_demodocker build -t raphael/demo .rm -rf docker_demo

$ cat DockerfileFROM rightscale/ruby-212ADD docker_demo /docker_demo

Setup context prior to build:

3. Secrets – Take away● Each Dockerfile ADD and RUN command results in a

new committed layer● All image layers (built or pulled) are readily

accessible● For now: Make sure to remove any unnecessary

credential from the context prior to building● In the future: Take advantage of “nested builds”,

see #7115

4. Container access● Launch image manually with

custom command to troubleshoot● Inspect files inside running container● Launch shell into running container

using docker exec (new in 1.3)

$ docker exec -it hopeful_shockley /bin/sh# ps -ax PID TTY STAT TIME COMMAND 1 ? Ss+ 0:00 /bin/bash ← Main container process 43 ? S 0:00 /bin/sh 49 ? R+ 0:00 ps -ax

5. Aufs vs. btrfs● aufs corruption of container filesystems,

scope unknown, issue #7229● btrfs seems to work better (default in CoreOS)● btrfs “requires” separate partition

$ mkfs.btrfs /dev/xvdb$ mount /dev/xvdb /mnt$ mkdir -p /mnt/docker$ ln -sf /mnt/docker /var/lib/docker$ sed -i -e '/DOCKER_OPTS/s/.*/DOCKER_OPTS="-s=btrfs"/' /etc/default/docker

$ restart docker

6. Got Infinite disk space?● Container logs grow indefinitely

○ Use logrotate with copytruncate

● Containers accumulate indefinitely○ Becomes an issue if containers are frequently

restarted due to upgrades or crashes○ Use docker run --rm

■ but then how do you troubleshoot?

○ Write script to docker rm old unused containers?

7. Huge Containers – how not to do itOverlays don’t go awayFROM ubuntu:14.04RUN apt-get updateRUN apt-get install -y libjpegRUN apt-get install -y libjpeg-dev build-essential gcc 109 MBADD source /build 5 MB?WORKDIR /build -RUN ./configure 0 MBRUN make 100 MB?RUN make installCMD /usr/local/bin/myexe

Use a tools container, share build results via volumeIn the future: “nested builds” #7115, “squash” #4232 ?

FROM ubuntu:14.04VOLUME /opt/app

ADD src /buildWORKDIR /buildRUN apt-get updateRUN apt-get install -y libjpeg-dev build-essential gccRUN ./configureRUN makeRUN make installRUN mkdir -p /opt/app

RUN cp -r /build/out/* /opt/app/

7. Huge Containers – solutions

8. Very slow container downloads● Downloading docker images is very slow● The problem isn’t bandwidth… see #7291● Caching can help depending on use-case

Boot time steps Docker RightScript Launch and boot 53s 49sPrep VM environment 36s 16sInstall & launch zookeeper, redis, kafka, mariadb, graphite, statsd 4m57s 1m5s

Install ruby n/a 54sInstall & launch custom apps 2m23s 3m3sTOTAL 8m50s 6m8s

9. BackupsUserguide: backup-restore-or-migrate-data-volumes● Create DB container with /data volume● Backup /data “anytime” from the VM● Or launch 2nd backup container with --volumes-from

➣ Simple in a 1-off server, but how to automate in general?

10. Docker Clusters● Does Docker Cluster software solve all these issues?● Kubernetes, Mesos, Fleet, …

○ apparently not (yet?)● But, they require an overlay network…

VM 1

Container 1Runs app 1172.16.4.3

VM 2

Container 2Runs app 1172.16.4.6

10.0.0.1 10.0.0.2

Wrapping upWhy docker?● dev-to-CI-to-prod workflow● portability: same container in different VMs

Putting it into production:● simple for one-off apps● still WIP for system-wide deployment

Overall very promising and great to work withMost pain points are actively being worked on

Perspectives on Docker10 things not to forget before

deploying Docker in production

— the end —

Raphael Simon & Thorsten von Eicken