Deploying OpenStack Using Docker in Production

Overview• The Pain of Operating Openstack• Possible Solutions• Why Docker Works• Why Docker Doesn’t Work• Docker @ TWC• Lessons Learned

• Docker in production in July 2015• First service was Designate• Added Heat, Nova and Keystone• Nova using Ceph and Solidfire Backends• Neutron in progress• Glance and Cinder later this year• Using Docker 1.10 and Docker Registry V2

Docker & OpenStack @ TWC

• Started with packages for deployments• Don’t like big-bang upgrades• Want to be able to carry local patches• Want to run mixed versions of services• Smaller upgrades, more often

How Did We End Up Here?

Why Not Packages?• Built packages for Keystone• Worked for local patches• Worked for updating stable branches• Doesn’t work for mixed releases• Limited by distro python packaging• Packaging workflow is a pain• Packages slow down your workflow• Package may not exist yet

Why Not Python Virtual Envs?• Deployed Designate with Virtual Envs• Mirrored Python packages internally• Built Virtual Envs on servers• Was slow to deploy• Still have to install/manage non-Python deps

Why Docker?

Everyone Else Is Doing It?

• Reproducible builds• Easy to distribute artifacts• Contains all dependencies• Easy to install multiple versions of an image

Why Docker?

• Restarting docker restarts containers • Intermittent bugginess• Complex services are hard to fit into Docker• Requires new tooling for build/deployment/etc

Why Not Docker?

Docker @ TWC: Images• Building base images using debootstrap• Build openstack-dev image based on that

–Contains all common deps• Image per OpenStack Service• Per service base requirements.txt and a frozen one• Frozen requirements.txt is used for image builds• Uses upper-constraints.txt for frozen requirements1

1. https://github.com/openstack/requirements/blob/master/upper-constraints.txt

Docker @ TWC: Image Tags• Tag should:

– Identify OpenStack service version– Identify tooling version–Be automatically generated–Be unique

Docker @ TWC: Image Tags

5.0.1-9-g0441ca8.16.dd35404

5.0.1-9-g0441ca8 16 dd35404

git-describe for Heat Tooling # commits Tooling commit hash

Docker @ TWC: Image Distribution• Using Docker Registry V2• Registry using file backend for local storage• Publish to master registry via Jenkins• Replicate to registry mirrors via rsync• Mirrors provide read-only access to images• No dependency on production environment

Docker @ TWC: Deployments• Images installed with puppet-docker• Managed with twc-openstack/os_docker• Worked with Puppet OpenStack project to add

hooks for software and service management• The os_docker module uses these to extend

OpenStack Puppet modules

Docker Registry Scaling• Docker recommends (almost requires) TLS for

registry• We deploy to 20 hypervisors in parallel• 8 vCPU Docker Registry• Supports concurrent 40 pulls * 500mb images• Size your registry for concurrent pulls * image size

Beware Docker Networking• We use --net host for all containers• Many services *require* --net host• Docker always creates bridge and NAT rules• NAT rules aren’t tied to a specific interface• Docker picks unused network range

–But can’t see VM IP addresses• Found this out on first Nova Compute deploy

OpenStack Upgrades With Docker• Allows upgrading single services!• Allows staging the upgrade images ahead of time• Not exciting

Why Not Kolla?• At the time didn’t meet our requirements:

–Didn’t support plugins, no source build–These things are resolved, or being resolved

• Great reference for running OpenStack with Docker• Recommended

Questions?Clayton O’Neill

– clayton.oneill@twcable.com– IRC: clayton– Twitter: @clayton_oneill

Eric Peterson–eric.peterson1@twcable.com– IRC: ducttape_–Twitter: @_ducttape