Upload
konstantin-nazarov
View
855
Download
4
Embed Size (px)
Citation preview
Building a PaaS with Docker, Consul and Python
by Konstantin Nazarov
I work in Tarantool And I've built a PaaS this way
I'll share how to build one• It is simple
• The experience is highly portable
• You can start small and grow iteratively
• Fits your requirements
• The tech stack is widely known
1. docker host
2. docker host
3. orchestrator
4. consul
5. web UI
Why?
Why build your own PaaS?
• If you develop your own product and sell it
• Small initial investment to solve lifecycle problems
• A way to fit your requirements exactly
• Keep your operations lean
• To enable fast experiments
When not to build it
• When you run off-she-shelf software
• If you have large monolithic services
What a PaaS should do
• Run the code you give to it
• Abstract away the OS
Let's build it
• We will build progressively
• On each step there will be a working system
Let's build it
A few building blocks
1. Python• Because it is simple
• There are bindings for almost all existing stuff
2. Docker
• Has remote HTTP API to run stuff
• Has convenient packaging format
2. Docker: the good
• Simple
• Well documented
2. Docker: the bad
• Bugs
• Weak networking
2. Docker: Alternatives
• fabric (yes, as an RPC)
• gearman
• nomad
3. Consul
• Fault-tolerant key-value storage
• Service registry with active checks
• Easily deployed
3. Consul: alternatives
• etcd
• zookeeper
MK1: Smart command-line
client to Docker/Consul
Why?
• quick to implement
• very high value compared to effort
What it should do
• run
• inspect
• upgrade
• rm
• ps
Example usage
$ mypaas run git://gitserver/project.git v1.2
fddf3f
$ mypaas upgrade fddf3f v1.3
$ mypaas rm fddf3f
How
• Docker API is exposed on physical servers
• Physical servers are registered in consul
• CLI connects to a known consul host
• Docker API is used to build app container "in place"
How to choose physical nodes
• By maximum memory requirements
• By conventional CPU units
• By number of services running
Upgrading versions
• Stop the running container
• Start new container inheriting volumes from the old
• On success, remove old container
• On failure, restart old container
Result
• A working PaaS
Improving MK1: health checks
• Consul API can be used to register health checks
• Consul can run commands in docker containers
• The CLI can poll consul for service statuses
Wiring things together via network
Docker networking hell
• There are overlay networks (UDP encapsulation)
• And macvlan (adding new MAC addresses to eth)
• And openvswitch with DPDK
• And god knows what else (BGP routing anyone?)
Let's use plain bridges
br0
Let's use plain bridges
• Docker IPAM doesn't know about other nodes
• IP conflicts are possible
• So we have to use our own IPAM
• Write allocated IPs to consul KV
• Set IPs explicitly
IPAM
MK2: running as a service
Why?• Limiting access to production servers
• Active monitoring of business logic
• Concurrent access
How?
• flask
• flask-restful
• gevent
Why these tools?
• Everything fits in one app
• Orchestration code is easier to write in async mode
• Quick to implement
• Your CLI becomes an HTTP API client
Separate state• Orchestrator itself should be stateless
• Orchestrator should show the system overview
• You probably need basic auth at this step
MK3: admin UI
Why?
• Easier to manage
• Easier to debug
Example
How?
• flask templates
• bootstrap
What to have here
• Overview of physical servers and their state
• Overview of your services and their state
• CRUD
MK4: delayed tasks and active checks
Why?
• Distributed cron
• Data extraction
• Backups
Tools
• gevent
• Docker exec API
How?
• On server start, spawn a worker fiber
• In the fiber, poll consul and run your code
• Or start worker fibers on demand
• Send notification emails upon completion
MK5: metrics and time series
Why?
• Historical data matters for problem solving
• See how well your new code is behaving over time
How?
• install prometheus and hook it up to consul
• use prometheus API to query aggregates
Exporting metrics to prometheus
• Either add support to your service
• Prometheus protocol is very simple!
• Or collect via the orchestrator
Recap
Recap• docker-python - running stuff
• python-consul - storing information about running stuff
• flask - serving admin UI
• flask-restful - providing HTTP API
• bootstrap - making your web page less ugly
• gevent - running delayed tasks and async code
• prometheus - storing time series
Thanks!Konstantin Nazarov
Building orchestrators is not that hard.
@racktear
http://bit.ly/paas-bom