L X C T O D O C K E R V I A C O N T I N U O U S D E L I V E R Y !@ M I C H A E L N E A L E

@CLOUDBEES

A B O U T M E & C L O U D B E E S

• co-founder

• cloudbees: continuous delivery company

• Best known for Jenkins!

S U M M A R Y

• Challenge: 3 disparate ways of containerisation

• evolution from cgroups to LXC - finally to docker

• a work in progress

• How Docker fits in with CD

• Lessons learned about containers in anger

O U R C H A L L E N G E - 3 E V O LV E D S TA N D A R D S

A P P S

J E N K I N S M A S T E R S

B U I L D E X E C U T O R S

• PaaS - more freedom for users

• Multi tenancy needs

• Containment

• use: cgroups, lxc, erlang control plane.

A P P S

• Non trivial to host: plugins, user config.

• multi tenancy (again)

• persistent volumes !!

• openvpn servers/clients and more

• rapid iteration

• existing solution was: chef + lxc

J E N K I N S M A S T E R S

• flexible build environments (user customisation)

• clean workspace (no processes)

• lxc + zfs + rest api

• covers os-x (virtualised kvm)

• network isolation (builds do all sorts of things)

• multi tenancy (large, we call it “mansion”)

• shorter duration containers

B U I L D E X E C U T O R S

• Why ZFS?

• pooling of disks

• pool of slaves (consistent hash==get a server with your stuff cached on it from last build)

• future: docker + zfs tempting

• C.O.W semantics (similar to aufs/devicemapper)

• shared pools

• Highly variable workload (season, time of day)

B U I L D E X E C U T O R S

http://developer-blog.cloudbees.com/2014/03/hi-speed-linux-builds-for-devcloud.html

http://developer-blog.cloudbees.com/2013/05/inside-linux-

containers-lxc-with.html?q=lxc

Evidence

• > 3 years LXC in production

• 1000’s of servers

• … but 3 different ways of doing similar-ish things evolved, so…

C O N TA I N E R S

U N I F Y A L L T H R E E : D O C K E R

A P P S

J E N K I N S M A S T E R S

B U I L D S L AV E S

source: if you don’t know xkcd leave now please

N O W W E H AV E 4 S TA N D A R D S

Q U I C K A S I D E

• continuous delivery refresher

C O N T I N U O U S D E P L O Y M E N T

• ftp *.php to server

C O N T I N U O U S D E L I V E RY

• Feature ready, feature deployed

• risk proportional to time between deploys

• more deploys == lower risk

• You all know this right?

C D & D O C K E R

• like wine & cheese (or a whale and containers?)

• docker image == unit of deployment

• triviality of concurrent docker images

L I M I T S O F C D - E G S U R V E Y M O N K E Y

• Developer commits to Git

• Jenkins builds app war file

• Ephemeral app is launched

• UAT is run against this (selenium)

• Ephemeral app is destroyed

• If DB schema changes, wait for approval

• blue/green deploy to production

C D T I M E L I N E

B L U E G R E E N

• source: martin fowler

• In docker terms: NEVER update an image

• docker build -t app:$BUILD_NUMBER!

• docker build -t app_$BUILD_NUMBER!

• maybe: -t app:latest? (:production can be prod tag)

N E X T P H A S E O F O U R J O U R N E Y

• Deploying lots of Jenkins

• From cgroups -> lxc -> docker over time

D E P L O Y I N G L O T S O F J E N K I N S V 1

• Initial: cgroups (and ephemeral users):

P R O V I D O R E M A S T E R H O S T

assign user/port

new acct/wake up

setup

U S E R D ATA

S N A P S H O T

create volume for master+slave

W H AT W E D I D N E X T

• Chef

• Needed some structure around setup

• Smart people were doing it

• recipes were a hit with the team

A L S O I N T R O D U C E D L X C F O R M A S T E R S

• Tweakable image

• Superior containment

• OpenVPN and other network improvements

D E P L O Y I N G V 2

P R O V I D O R E M A S T E R H O S T

assign user/port

new acct/wake up

chef on setup

U S E R D ATA

S N A P S H O T

C H E F R E P O

C H A L L E N G E S W I T H C H E F B O O T

• LXC image not often updated (cost/time)

• ∴ more work done on startup (via chef)

• ∴ slower, error prone, runtime dependency on chef repo

C H A L L E N G E S U P D AT I N G I M A G E

• Sans docker: lots of large files to push and pull

• Starting “cold” each time

• No registry

• No Dockerfile

• No cache of images

• No easy dev. desktop experience

L X C - > D O C K E R

• AOT image preparation

• Layered FS makes cost affordable

• don’t leave it to (Jenkins/app) boot time

• Developers can do it on desktop

• “refresh driven development”

• truly portable images

• “socialise container bugs”

N O T E : O N C O N TA I N E R S A N D M E M O R Y

• our old model

• thinking: swap bad, kill apps when out of memory

• turns out people hate this

• no swap accounting

• docker memory (-m parameter)

• swapaccount=1 kernel param (eg pvgrub/ec2: /boot/grub/menu.lst)

• docker way: the right way (account for swap, prevent accidents)

S U M M A R Y:

• Docker turns into perfect solution

T H E I D E A L D O C K E R F I L E

• Prefer native packaging

• Dockerfile essentially series of package installs

• and small minor steps

• anything > 1 line bash is a OS package

T H E I D E A L D O C K E R F I L E

• No one agreed with me.

C H E F - S O L O

• compromise: chef-solo ahead of time

• Docker image built ahead of time

• Dockerfile generated based on chef-solo

• re-use of chef recipes, but done at the “correct” time

N E W H I G H L E V E L W O R K F L O W

R E C I P E C H A N G E D O C K E R B U I L D

S M O K E T E S T

S 3

B E E S C L O U D VA L I D AT E

( S 3 ) P U L L O N

E X I S T I N G

N E W A M I P R E -S E E D E D

U N - H I B E R N AT E /N E W G E T L AT E S T

tag:$BUILD_NUM

R E S U LT

• Cold build of AOT image - 6 minutes (old 15) - in .au

• Warm build in seconds

• build on desktop or CI

• though: registry in .au can be PAINFUL

N E X T C H A L L E N G E : D I S T R I B U T I N G I M A G E S

• We didn’t have per-node (lxc) image caching

• Download on launch or refresh (s3 fast enough)

• AMIs refreshed “reasonably” often, servers rotated

• Docker gave us that for free, but…

R E G I S T R Y ?

• Continuous Delivery generates a LOT of artifacts

• every build

• Want to keep everything (sort of)

• Need a perfect scalable store for large binary objects

• Solomon guarantees index backed up on 12 continents**

• Need to quickly get images on boxes

G E T T I N G I M A G E S O N B O X E S

• Options

• Twitter “murder” (clever use of bittorrent)

• We host (mostly) on ec2: s3 reliably 50-200MB/s

• Reduce permutations of images required on a box

• Private registry

R E G I S T R Y

• Needs to be very HA

• It’s a runtime production grade dependency

• High throughput for large blobs (push and pull)

• We run on EC2 (US-east-1 and EU-west-1)

R E G I S T R Y

D O C K E R R E G I S T R Y S 3

Many instances Free scalabilitywork :(

R E G I S T R Y

D O C K E R S 3

Many instances Free scalability

load/save

Also: fully offline from public registry

S 3

• no servers are good servers

• S3 is magic.

• someone claimed it runs on servers, lies. Can’t be true.

• ideally only deploy time dependency is on S3 (or similar)

• Is safe to consider it HA

D O G E S T R Y

• https://github.com/blake-education/dogestry

• think: docker save -> rsync layers to S3 bucket

• future: s3 client app “plugin” for docker CLI

• temporary measure

D O G E S T R Y

• dogestry push s3://ops-goodies/docker-repo/?region=us-west-2 hipache!

• dogestry pull s3://ops-goodies/docker-repo/?region=us-west-2 hipache

S 3 S T O R A G E

R E G I S T R Y - F U T U R E O P T I O N S

• shift to private registry or s3 client

• private registry “client” on each node with /etc/hosts hack (backed by s3)

• just use index.docker.io for private

• For Now: Using private registry where it makes sense

• Now: dogestry/s3

F I N A L H I G H L E V E L W O R K F L O W

R E C I P E C H A N G E D O C K E R B U I L D

S M O K E T E S T

S 3 /D O G E S T R Y

S T O R E L AT E S T I M A G E N A M E

push s3 jenkins_$build_number

S 3 P U L L /P R E - S E E D

approve image

S U P E R V I S I O N

• Currently using runit with “docker run”

• Fits in with existing systems, log shipping etc.

• Behaves itself

• Systemd also good

• Future: docker “native” daemonisation

P E R B U I L D R E P O V S TA G

• ie “docker pull jenkins:$BUILD_NUMBER” vs

“docker pull jenkins_$BUILD_NUMBER”

• Why not use tags for builds?

• Massive repo growth, every build (Cont. Delivery)

• ever need to export whole repo? purge tags?

• historical impact on pull (resolved now?)

F I N A L L E S S O N S L E A R N E D …

• Multi tenancy needs (user-namespace !!!)

• Solutions

• whitelisted dockerfiles to base images on

• produce docker image for user

• desires

• securely allow users full freedom (uid=0 mapping)B U I L D

E X E C U T O R S

M U LT I T E N A N C Y

C H A L L E N G E

O N E D O C K E R I M A G E P E R A P P ?

• A) One docker image per app, or

• B) Generic docker image, with app bind-mount in at runtime

!

• Advantage B: less of an explosion of images. Faster to get new box “seeded”. JVM apps, node apps, Go apps all work well this way.

D O C K E R A S PA C K A G I N G S Y S T E M

• Demos of on-prem products

• Continuously deliver (push) images for field engineers

• change to demo content - new image

• change to version - new image

J E N K I N S - C I . O R G

• Now docker powered

• Managed with puppet, for example

• https://github.com/jenkins-infra/bind

• https://github.com/jenkins-infra/jenkins-infra/blob/dns/dist/profile/manifests/bind.pp

• puppet module: https://github.com/garethr/garethr-docker

S A M P L E D O C K E R P I P E L I N E

https://github.com/michaelneale/docker-pipeline-demo

S A M P L E D O C K E R P I P E L I N E

docker run -p 8080:8080 —privileged cloudbees/jenkins

S A M P L E D O C K E R P I P E L I N E

docker run -p 8080:8080 -v /var/run/docker.sock:/var/run/docker.sock cloudbees/jenkins

D O C K E R F I L E A S B U I L D F I L E

• Why not? Canonical build definition

• Run unit tests

• output “xunit" reports (allows test trend reporting)

• clone an opensource project, “docker build .” - reflex.

H A S H D O C K E R F I L E C O N T E N T

• hash contents of docker file

• name image as hash

• only build if hash not found in docker images

N O V E L C I U S E S O F D O C K E R

• By Groupon: .ci.yml: specify env image or default to Dockerfile in project root, include services

• https://github.com/jenkinsci/DotCi

C O N TA I N E R C H A L L E N G E S

• containers and CPU

• Slice up large box == false appearance of more CPU power

• LD_PRELOAD “fake” value?

• cgroups cpuset cumbersome

!

• Security of multitenancy

• and docker killed it!

• linux is my IDE, my platform

• apps delivered as compositions of linux tools and system

• many languages

• images actually are “portable”

• future is bright

PA A S I S D E A D

Q U E S T I O N S

• @michaelneale & github/michaelneale

• www.michaelneale.net

!

• #docker-dev #jenkins IRC as michaelneale

• cloudbees.com

• developer-blog.cloudbees.com