Transcript
Page 1: Building a PaaS with Docker, Consul and Python

Building a PaaS with Docker, Consul and Python

by Konstantin Nazarov

Page 2: Building a PaaS with Docker, Consul and Python
Page 3: Building a PaaS with Docker, Consul and Python

I work in Tarantool And I've built a PaaS this way

Page 4: Building a PaaS with Docker, Consul and Python

I'll share how to build one• It is simple

• The experience is highly portable

• You can start small and grow iteratively

• Fits your requirements

• The tech stack is widely known

Page 5: Building a PaaS with Docker, Consul and Python

1. docker host

2. docker host

3. orchestrator

4. consul

5. web UI

Page 6: Building a PaaS with Docker, Consul and Python

Why?

Page 7: Building a PaaS with Docker, Consul and Python

Why build your own PaaS?

• If you develop your own product and sell it

• Small initial investment to solve lifecycle problems

• A way to fit your requirements exactly

• Keep your operations lean

• To enable fast experiments

Page 8: Building a PaaS with Docker, Consul and Python

When not to build it

• When you run off-she-shelf software

• If you have large monolithic services

Page 9: Building a PaaS with Docker, Consul and Python

What a PaaS should do

• Run the code you give to it

• Abstract away the OS

Page 10: Building a PaaS with Docker, Consul and Python

Let's build it

Page 11: Building a PaaS with Docker, Consul and Python

• We will build progressively

• On each step there will be a working system

Let's build it

Page 12: Building a PaaS with Docker, Consul and Python

A few building blocks

Page 13: Building a PaaS with Docker, Consul and Python

1. Python• Because it is simple

• There are bindings for almost all existing stuff

Page 14: Building a PaaS with Docker, Consul and Python

2. Docker

• Has remote HTTP API to run stuff

• Has convenient packaging format

Page 15: Building a PaaS with Docker, Consul and Python

2. Docker: the good

• Simple

• Well documented

Page 16: Building a PaaS with Docker, Consul and Python

2. Docker: the bad

• Bugs

• Weak networking

Page 17: Building a PaaS with Docker, Consul and Python

2. Docker: Alternatives

• fabric (yes, as an RPC)

• gearman

• nomad

Page 18: Building a PaaS with Docker, Consul and Python

3. Consul

• Fault-tolerant key-value storage

• Service registry with active checks

• Easily deployed

Page 19: Building a PaaS with Docker, Consul and Python

3. Consul: alternatives

• etcd

• zookeeper

Page 20: Building a PaaS with Docker, Consul and Python

MK1: Smart command-line

client to Docker/Consul

Page 21: Building a PaaS with Docker, Consul and Python

Why?

• quick to implement

• very high value compared to effort

Page 22: Building a PaaS with Docker, Consul and Python

What it should do

• run

• inspect

• upgrade

• rm

• ps

Page 23: Building a PaaS with Docker, Consul and Python

Example usage

$ mypaas run git://gitserver/project.git v1.2

fddf3f

$ mypaas upgrade fddf3f v1.3

$ mypaas rm fddf3f

Page 24: Building a PaaS with Docker, Consul and Python

How

• Docker API is exposed on physical servers

• Physical servers are registered in consul

• CLI connects to a known consul host

• Docker API is used to build app container "in place"

Page 25: Building a PaaS with Docker, Consul and Python

How to choose physical nodes

• By maximum memory requirements

• By conventional CPU units

• By number of services running

Page 26: Building a PaaS with Docker, Consul and Python

Upgrading versions

• Stop the running container

• Start new container inheriting volumes from the old

• On success, remove old container

• On failure, restart old container

Page 27: Building a PaaS with Docker, Consul and Python

Result

• A working PaaS

Page 28: Building a PaaS with Docker, Consul and Python

Improving MK1: health checks

• Consul API can be used to register health checks

• Consul can run commands in docker containers

• The CLI can poll consul for service statuses

Page 29: Building a PaaS with Docker, Consul and Python

Wiring things together via network

Page 30: Building a PaaS with Docker, Consul and Python

Docker networking hell

• There are overlay networks (UDP encapsulation)

• And macvlan (adding new MAC addresses to eth)

• And openvswitch with DPDK

• And god knows what else (BGP routing anyone?)

Page 31: Building a PaaS with Docker, Consul and Python

Let's use plain bridges

br0

Page 32: Building a PaaS with Docker, Consul and Python

Let's use plain bridges

• Docker IPAM doesn't know about other nodes

• IP conflicts are possible

• So we have to use our own IPAM

• Write allocated IPs to consul KV

• Set IPs explicitly

Page 33: Building a PaaS with Docker, Consul and Python

IPAM

Page 34: Building a PaaS with Docker, Consul and Python

MK2: running as a service

Page 35: Building a PaaS with Docker, Consul and Python

Why?• Limiting access to production servers

• Active monitoring of business logic

• Concurrent access

Page 36: Building a PaaS with Docker, Consul and Python

How?

• flask

• flask-restful

• gevent

Page 37: Building a PaaS with Docker, Consul and Python

Why these tools?

• Everything fits in one app

• Orchestration code is easier to write in async mode

• Quick to implement

• Your CLI becomes an HTTP API client

Page 38: Building a PaaS with Docker, Consul and Python

Separate state• Orchestrator itself should be stateless

• Orchestrator should show the system overview

• You probably need basic auth at this step

Page 39: Building a PaaS with Docker, Consul and Python

MK3: admin UI

Page 40: Building a PaaS with Docker, Consul and Python

Why?

• Easier to manage

• Easier to debug

Page 41: Building a PaaS with Docker, Consul and Python

Example

Page 42: Building a PaaS with Docker, Consul and Python

How?

• flask templates

• bootstrap

Page 43: Building a PaaS with Docker, Consul and Python

What to have here

• Overview of physical servers and their state

• Overview of your services and their state

• CRUD

Page 44: Building a PaaS with Docker, Consul and Python

MK4: delayed tasks and active checks

Page 45: Building a PaaS with Docker, Consul and Python

Why?

• Distributed cron

• Data extraction

• Backups

Page 46: Building a PaaS with Docker, Consul and Python

Tools

• gevent

• Docker exec API

Page 47: Building a PaaS with Docker, Consul and Python

How?

• On server start, spawn a worker fiber

• In the fiber, poll consul and run your code

• Or start worker fibers on demand

• Send notification emails upon completion

Page 48: Building a PaaS with Docker, Consul and Python

MK5: metrics and time series

Page 49: Building a PaaS with Docker, Consul and Python

Why?

• Historical data matters for problem solving

• See how well your new code is behaving over time

Page 50: Building a PaaS with Docker, Consul and Python

How?

• install prometheus and hook it up to consul

• use prometheus API to query aggregates

Page 51: Building a PaaS with Docker, Consul and Python

Exporting metrics to prometheus

• Either add support to your service

• Prometheus protocol is very simple!

• Or collect via the orchestrator

Page 52: Building a PaaS with Docker, Consul and Python
Page 53: Building a PaaS with Docker, Consul and Python

Recap

Page 54: Building a PaaS with Docker, Consul and Python

Recap• docker-python - running stuff

• python-consul - storing information about running stuff

• flask - serving admin UI

• flask-restful - providing HTTP API

• bootstrap - making your web page less ugly

• gevent - running delayed tasks and async code

• prometheus - storing time series

Page 55: Building a PaaS with Docker, Consul and Python
Page 56: Building a PaaS with Docker, Consul and Python

Thanks!Konstantin Nazarov

[email protected]

Building orchestrators is not that hard.

@racktear

http://bit.ly/paas-bom


Recommended