Data focused Docker clustering

Luke MarsdenFounder, ClusterHQ

clusterhq.comgithub.com/clusterhq/flocker

Background:

Container clusters where volumes are a first class citizen

Motivations:

robustnessscalability

eliminate SPOFsreduce complexity

Modern applications

are complex distributed systems

Containers and PaaS today

capture the easy, data-free bits

We should aim to capture the entire app

enabling larger class of apps to move to containers

Goals

Multi-node/clustered

Ease of operational manageability

Homogeneous serversNo single point of failure

Multi-regionData services the cluster depends on must

be protected by/within the system itself

Containers should be portable, even when they have state

(corollary: the system should expose portable, resilient volumes)

Dependent services or applications should not need to be reconfigured when a service

moves around

Stateful things scale vertically, stateless

things scale horizontally

Proposed implementation

Management layer

Strongly consistent group messaging semantics

Distributed consensus protocolsNo single master

Storage

ZFS: storage analogue to containers for compute

Independently snapshotted and replicatedSharding container volumes across a cluster

Redundancy invariantAutomatic slave allocation

Networking

AwesomeProxyany request to any node

Routes vs. Links

Live migration

Pausing in flight requestsQuiesce filesystem

Replicate last snapshotUnleash requests

Thank you - questions?

Check it out!github.com/clusterhq/flocker

clusterhq.com