Building Web Scale Apps with Docker and Mesos by Alex Rukletsov (Mesosphere)

  • Published on
    07-Jul-2015

  • View
    9.741

  • Download
    1

Embed Size (px)

DESCRIPTION

Operating apps at web scale has become the new normal, but has been out of reach for most companies. Join us as we show you how to deploy and manage your Docker containers at scale. See how easy it is to build highly-available, fault-tolerant web scale apps using Docker with the Mesos cluster scheduler. Docker plus Mesos is a new way to scale applications. Together they give you capabilities similar to Googles Borg, the Googleplexs secret weapon of scalability and fault tolerance.

Transcript

  • 1. Building Web Scale Apps withDocker and MesosAlexander RukletsovSoftware Engineer @ Mesosphere

2. Why should we care? 3. Applications in the Cloud EraApp App App AppClient-Server Era:Small apps, big serversAppServ Serv Serv ServCloud Era:Big apps, small serversVirtualizationAggregationServer 4. Major Components Hardware Orchestration, deployment and isolation Cluster and resource management Scale-aware applications, service discovery, etc. 5. Major Components Hardware Orchestration, deployment and isolation Cluster and resource management Scale-aware applications, service discovery 6. Applications in the Cloud EraApp App App AppClient-Server Era:Small apps, big serversAppServ Serv Serv ServCloud Era:Big apps, small serversVirtualizationAggregationServer 7. Divide et impera [et sarcina]Divide and rule [and package]Philip II of Macedon, paraphrased 8. Containers Lightweight Linux execution environment Static application composition Reliable deployment Unit of resource isolation Execution isolation Multi-tenancy without heavyweight VMs 9. Docker Open source Configurable layers Reproducible Version-controlled Plenty of other peoples containers 10. Docker Open source Configurable layers Reproducible Version-controlled Plenty of other peoples containers First-class citizen in Mesos and the Mesosphere stack Kubernetes employs and promotes Docker 11. Gerard Julien/AFPRun everything in containers! 12. What about container management? 13. Unknown 14. Unknown 15. Apache Mesos facts Created in 2009 at UC Berkeley, hardened in Twitter Top-level Apache project Mesosphere, Twitter, and Airbnb are major users /contributors Scales to 10 000s of nodes, production grade Packages and support through Mesosphere Google officially endorsed Mesos for Kubernetes Built-in containerization, including Docker 16. Mesos as a Distributed OS kernel Two level resource scheduling Launch tasks across the cluster Communication between tasks (like IPC) APIs for building native applications (aka frameworks):program against the datacenter APIs in C++, Python, JVM-languages, Go and counting Pluggable CPU, memory, IO isolation Multi-tenant workloads Failure detection Easy failover and HA 17. How Mesos works (HA mode)ApplicationSchedulerZookeeperMesosMasteMr esosMasterMesosMasterMesosMasterMesosSlaveExecutor TaskTaskExecutorFramework TaskTaskTaskExecutorMesosSlave 18. Manage containers with Mesos! ect.nl 19. Mesos + Docker = 20. Native Docker support in Mesosmessage DockerInfo {required string image = 1;// Network options.enum Network {HOST = 1;BRIDGE = 2;NONE = 3;}message PortMapping {required uint32 host_port = 1;required uint32 container_port = 2;optional string protocol = 3; // Protocol to expose (ie: tcp, udp).}optional Network network = 2 [default = HOST];repeated PortMapping port_mappings = 3;optional bool privileged = 4 [default = false];// Allowing arbitrary parameters to be passed to docker CLI.repeated Parameter parameters = 5;} 21. Native Docker support in MesosCommandInfo command;command.set_value("dd if=/dev/zero of=/dev/null");ContainerInfo::DockerInfo dockerInfo;dockerInfo.set_image("mesosphere/inky");ContainerInfo containerInfo;containerInfo.set_type(ContainerInfo::DOCKER);containerInfo.mutable_docker()->CopyFrom(dockerInfo);TaskInfo task;task.set_name("");task.mutable_task_id()->set_value("1");task.mutable_slave_id()->CopyFrom(offer.slave_id());task.mutable_resources()->CopyFrom(offer.resource());task.mutable_command()->CopyFrom(command);task.mutable_container()->CopyFrom(containerInfo);vector tasks;tasks.push_back(task);driver.launchTasks(offer.id(), tasks); 22. Native Docker support in Marathon// nginx-task.json{"container": {"type": "DOCKER","docker": {"image": "nginx","network": "BRIDGE","portMappings": [{ "containerPort": 80,"hostPort": 0,"servicePort": 80,"protocol": "tcp" }]}},"id": "nginx","instances": "1","cpus": "0.25","mem": "256","uris": []}$ cat nginx-task.json | http http://dev1.mesosphere.com:8080/v2/apps 23. Future Docker Swarm API supportAppAppAppMesos FrameworkMesosAppServDockerDocker Swarm APIDockerAppDockerDockerDockerAppDockerAppDockerServ Serv Serv Serv 24. What Mesos contributes Multi-framework: weighted fair sharing, roles, etc. Run Docker containers alongside other popular frameworks(e.g. Spark, Rails, Hadoop, ) Run services and batch apps in the same cluster Advanced scheduling: resources, constraints, global view High resource availability, cluster self-healing Proven at scale, battle-tested in production GUI / CLI cluster management console 25. Ways of running Dockers Marathoneasy to setup, reliable orchestration Multiple Marathonsbenefit from two level scheduling, reservations, frameworkroles Custom frameworkfine-grained management 26. Lhomme est libre au moment quil veut ltreMan is free at the instant he wants to beVoltaire 27. Cluster configuration example #1ServicesMesosAppAppAWS, DigitalOcean, GCEAppsAPIKernelHardwareDockerDockerServices REST APIMarathonAppDockerAppAppDockerApp 28. Cluster configuration example #2MesosAppAppAWS, DigitalOcean, GCEAppsAPIKernelHardwareServicesServices REST APIMarathonAppAppKubernetesDockerDockerDockerAppDockerApp 29. Cluster configuration example #3DockerServices REST APIMarathon (init)Serv Serv Serv ServMesosSpark, MPI,Hadoop, StormMesos SDKAppJava, Python, C++, GoServ Serv Serv ServBatch REST APIChronos (cron)ServApp Recurring Jobs(ETL, backups)HardwareNative Long running BatchAppsAPIKernelAppDocker 30. Cluster configuration example #4DockerAWS, DO, GCEMesosSpark, MPI,Hadoop, StormMesos SDKAppJava, Python, C++, GoServ ServServices REST APIMarathon (init)Batch REST APIChronos (cron)ServApp Recurring Jobs(ETL, backups)HardwareNative Long running BatchAppsAPIKernelAppDockerServ 31. Mesos is not only for big players! Reuters 32. Tiny clusters for everybody Mesosphere on GCE Mesosphere on DigitalOcean Mesosphere on AWS Portable Mesosphere on a USB stick 33. google.mesosphere.com 34. digitalocean.mesosphere.com 35. elastic.mesosphere.io 36. Demo: Mesos-on-Mesosphere 37. Mesos-on-MesosphereTask Task TaskZookeeperMesosMarathon SlaveMesosMasterMesosMasMteersosMasteMresosMasterMesosSlave...Task 38. Mesos-on-MesosphereMesosSlaveMesosSlaveDocker Docker DockerDockerZookeeperMesosMarathon SlaveMesosMasterMesosMasMteersosMasteMresosMasterMesosSlave...MesosMasterDockerMesosSlaveDockerMesosSlaveMesosMaster 39. Summary Complete stack for large (and small) distributed apps Multi-tenancy Resource optimizations Easy to deploy No vendor lock-in 40. Thank You.alex@mesosphere.io#mesos on irc.freenode.net