33
Docker containers at scale with Apache Mesos Distributed Systems Engineer Apache Mesos PMC Timothy Chen @tnachen [email protected]

Scale your docker containers with Mesos

Embed Size (px)

Citation preview

Page 1: Scale your docker containers with Mesos

Docker containers at scale withApache MesosDistributed Systems Engineer

Apache Mesos PMC

Timothy Chen

@tnachen

[email protected]

Page 2: Scale your docker containers with Mesos

Mesoswhat?    Marathon

Chronos

Demo!

Page 3: Scale your docker containers with Mesos

Status quo is static partitioning and use of virtual machines

Page 4: Scale your docker containers with Mesos

Add some virtual machines

Page 5: Scale your docker containers with Mesos

Provision Hadoop

Page 6: Scale your docker containers with Mesos

Provision a web service

Page 7: Scale your docker containers with Mesos

Moar data, moar Hadoop

Page 8: Scale your docker containers with Mesos

Mesos let us treat a cluster ofnodes...

Page 9: Scale your docker containers with Mesos

As one big computer

Page 10: Scale your docker containers with Mesos

Not as individual machines

Not as VMs

Page 11: Scale your docker containers with Mesos

But as computational resourceslike cores, memory, disks, etc.

Page 12: Scale your docker containers with Mesos

240 million monthly activeusers

500 million tweets per dayUp to 150k tweets per secondMore than 100TB per day of

compressed data

Page 13: Scale your docker containers with Mesos

“Mesos is the cornerstone of our elasticcompute infrastructure — it's how we build allour new services and is critical for Twitter'scontinued success at scale. It's one of the

primary keys to our data center efficiency.” —Chris Fry, SVP of Engineering at Twitter

Page 14: Scale your docker containers with Mesos
Page 15: Scale your docker containers with Mesos
Page 16: Scale your docker containers with Mesos
Page 17: Scale your docker containers with Mesos
Page 18: Scale your docker containers with Mesos

 

Containerization in Mesos, a brief history

Page 19: Scale your docker containers with Mesos

Mesos is...A top-level Apache project A cluster resource negotiator Scalable to 10,000s of nodes Fault-tolerant, battle-tested An SDK for distributed apps

Page 20: Scale your docker containers with Mesos

 

Page 21: Scale your docker containers with Mesos

Mesos for all the things

Page 22: Scale your docker containers with Mesos

 Marathon  Mesoswhat?

Chronos

Demo!

Page 23: Scale your docker containers with Mesos

a self-serve interface to your cluster

Page 24: Scale your docker containers with Mesos

distributed "init" for long-running services

Page 25: Scale your docker containers with Mesos

a private fault-tolerant PaaS

Page 26: Scale your docker containers with Mesos

Marathon does it!Start, stop, scale, update appsNice web interface, APIHighly available, no SPoFNative Docker supportFully featured REST APIPluggable event busRolling deploy / restartApplication health checksArtifact staging

Page 27: Scale your docker containers with Mesos

   Chronos

Mesoswhat?

Marathon

Demo!

Page 28: Scale your docker containers with Mesos

Introducing Chronos

a scheduler for batch and one-off jobs

Page 29: Scale your docker containers with Mesos

Distribute a graph of jobs

Dependency graph for execution

Page 30: Scale your docker containers with Mesos

FeaturesDistributed job schedulerWeb interface, APIHighly available, no SPoFNative Docker supportEasy scheduling with repeating intervals

Page 31: Scale your docker containers with Mesos

Chronos  REST

PUT chronos-node:8080/scheduler/job/job1

GET chronos-node:8080/scheduler/jobs

DELETE chronos-node:8080/scheduler/task/kill/job2

Page 32: Scale your docker containers with Mesos

   

Demo!

Mesoswhat?

Marathon

Chronos

Page 33: Scale your docker containers with Mesos

Thanks!Come and talk to me

P.S., we're hiring!