47
1 © Copyright 2016 EMC Corporation. All rights reserved.

EMC World 2016 - Introduction to Mesos and Mesosphere

Embed Size (px)

Citation preview

Page 1: EMC World 2016 - Introduction to Mesos and Mesosphere

1© Copyright 2016 EMC Corporation. All rights reserved.

Page 2: EMC World 2016 - Introduction to Mesos and Mesosphere

2© Copyright 2016 EMC Corporation. All rights reserved. 2© Copyright 2016 EMC Corporation. All rights reserved.

INTRODUCTION TOMESOS AND MESOSPHEREMAY 2016

Page 3: EMC World 2016 - Introduction to Mesos and Mesosphere

3© Copyright 2016 EMC Corporation. All rights reserved.

• Somik Behera, DCOS Product Manager - Mesosphere – Twitter: @strikesme– Blog: http://strikesme.com

• David vonThenen, Developer Advocate - EMC {code}– Twitter: @dvonthenen– Blog: http://dvonthenen.com

PRESENTERS

Page 4: EMC World 2016 - Introduction to Mesos and Mesosphere

4© Copyright 2016 EMC Corporation. All rights reserved.

• What is Mesos?• What will Mesos do for me?• Who is Mesosphere?• How EMC{code} fits with Mesos• Demo• Q&A

AGENDA

Page 5: EMC World 2016 - Introduction to Mesos and Mesosphere

5© Copyright 2016 EMC Corporation. All rights reserved. 5© Copyright 2016 EMC Corporation. All rights reserved.

WHAT IS MESOS?

Page 6: EMC World 2016 - Introduction to Mesos and Mesosphere

6© Copyright 2016 EMC Corporation. All rights reserved.

• From a resource perspective, it’s a cluster manager:– Pools servers resources to be centrally managed as a single

unit• From an application perspective, it’s a scheduler:

– Dispatches workloads to consume pooled resources• Often described as a Data Center Operating System

(DCOS)

MESOS IS…

Page 7: EMC World 2016 - Introduction to Mesos and Mesosphere

7© Copyright 2016 EMC Corporation. All rights reserved.

• Master– Manages Mesos Agents– Enables fine-grained sharing of resources (CPU, RAM, …)

across frameworks in the form of resource offers.• Zookeeper

– Enables highly reliable distributed coordination (cluster state)

• Agent (aka Slave)– The entity that will be running the task

MESOS COMPONENTS

Page 8: EMC World 2016 - Introduction to Mesos and Mesosphere

8© Copyright 2016 EMC Corporation. All rights reserved.

MESOS ARCHITECTURE

Page 9: EMC World 2016 - Introduction to Mesos and Mesosphere

9© Copyright 2016 EMC Corporation. All rights reserved.

EFFICIENT RESOURCE UTILIZATION

Page 10: EMC World 2016 - Introduction to Mesos and Mesosphere

10© Copyright 2016 EMC Corporation. All rights reserved.

WHO DOESN’T LIKE EFFICIENCY?

Page 11: EMC World 2016 - Introduction to Mesos and Mesosphere

11© Copyright 2016 EMC Corporation. All rights reserved.

• Generic– Anything that can be run on a standard Linux shell– Provides a mechanism for deploying applications packages– Supports Docker workloads

• Framework– A plug-in to Mesos that allows you to implement a sub-

scheduler– specialize a workload by providing custom placement and

resource allocation

APPLICATION SUPPORT

Page 12: EMC World 2016 - Introduction to Mesos and Mesosphere

12© Copyright 2016 EMC Corporation. All rights reserved.

GENERIC APPLICATIONS

Page 13: EMC World 2016 - Introduction to Mesos and Mesosphere

13© Copyright 2016 EMC Corporation. All rights reserved.

MESOS FRAMEWORKS

Page 14: EMC World 2016 - Introduction to Mesos and Mesosphere

14© Copyright 2016 EMC Corporation. All rights reserved.

BRINGING EVERYTHING TOGETHER

Page 15: EMC World 2016 - Introduction to Mesos and Mesosphere

15© Copyright 2016 EMC Corporation. All rights reserved. 15© Copyright 2016 EMC Corporation. All rights reserved.

WHAT WILL MESOS DO FOR ME?

Page 16: EMC World 2016 - Introduction to Mesos and Mesosphere

16© Copyright 2016 EMC Corporation. All rights reserved.

• Bare metal x86 hardware• VMs (your favorite hypervisor)• Public, Private, Hybrid Cloud

MULTIPLE DEPLOYMENT VECTORS

Page 17: EMC World 2016 - Introduction to Mesos and Mesosphere

17© Copyright 2016 EMC Corporation. All rights reserved.

MESOS WILL RUN ON…

Page 18: EMC World 2016 - Introduction to Mesos and Mesosphere

18© Copyright 2016 EMC Corporation. All rights reserved.

• Heterogeneous nodes (different CPU models, memory footprints, etc)• Supports elastic scaling

– Nodes can be added for capacity– Nodes can be removed for maintenance

CLUSTER NODES

Page 19: EMC World 2016 - Introduction to Mesos and Mesosphere

19© Copyright 2016 EMC Corporation. All rights reserved.

• Has been in production use for 3+ years• 10k+ node scale• Recognized by users and the community• Tasks dispatching scale linearly over # nodes (2010)

STABILITY AND PERFORMANCE

Page 20: EMC World 2016 - Introduction to Mesos and Mesosphere

20© Copyright 2016 EMC Corporation. All rights reserved.

POWERED BY MESOS

Page 21: EMC World 2016 - Introduction to Mesos and Mesosphere

21© Copyright 2016 EMC Corporation. All rights reserved.

• Marathon– manages long-running applications.

Provides both a UI and REST API• Chronos

– is a replacement for cron to schedule batch jobs and other workloads.

• Mesos-DNS– enables DNS based service discovery 

• Etc, etc, etc…

RICH APPLICATIONS AND ECOSYSTEM

Page 22: EMC World 2016 - Introduction to Mesos and Mesosphere

22© Copyright 2016 EMC Corporation. All rights reserved.

• Runs a wide variety of workloads• On commodity hardware• Efficient utilization of resources• High Availability / Fault Tolerant• UI and API management

WHAT WILL MESOS DO FOR ME?

Page 23: EMC World 2016 - Introduction to Mesos and Mesosphere

23© Copyright 2016 EMC Corporation. All rights reserved.

OR TRY TO BUILD ONE FROM SCRATCH

Page 24: EMC World 2016 - Introduction to Mesos and Mesosphere

24© Copyright 2016 EMC Corporation. All rights reserved. 24© Copyright 2016 EMC Corporation. All rights reserved.

WHO IS MESOSPHERE?

Page 25: EMC World 2016 - Introduction to Mesos and Mesosphere

25© Copyright 2016 EMC Corporation. All rights reserved.

MESOSPHERE: BORN OF THE HYPERSCALE ERA

Mesosphere is well funded$50M by Tier 1 investors: Andreessen Horowitz & Khosla Ventures

2009 2013 2014 2015

First Mesosphere DCOS lighthouse customersTens of thousands of containers launched

Expanded OperationsSan Francisco (2013)Hamburg (2014)New York (2015)Berlin (2015)

Mesosphere FoundedKey engineering leaders from Twitter, Airbnb - companies behind open-source tech

Apache Mesos built at UC Berkeley

● Core technology from AMPlab

● Corporate large-scale production deployments begin (e.g., Apple, Twitter, Salesforce)

Growing Ecosystem & DCOS General AvailabilityNow a breeze to install modern app services (e.g., Hadoop, Spark, Cassandra)

Page 26: EMC World 2016 - Introduction to Mesos and Mesosphere

26© Copyright 2016 EMC Corporation. All rights reserved.

DELIVERING OUTCOMES FOR ENTERPRISES

DEVELOPER AGILITY

HYPERSCALE OPERATIONS

DATA AGILITY

Highly Efficient & Scalable IT

Distributed Framework Pooling

Container Orchestration

Cont. Integration/Delivery

Developer Cloud

Self-Service Container Runtime

Big Data

Internet Of Things

Reactive applications

Page 27: EMC World 2016 - Introduction to Mesos and Mesosphere

27© Copyright 2016 EMC Corporation. All rights reserved.

THE DATACENTER IS THE NEW SERVER

PHYSICAL (x86) VIRTUAL HYPERSCALEMAINFRAME

SERVER VIRTUAL MACHINE

PARTITION (LPAR)

UNIT OF INTERACTION

● ERP, CRM, PRODUCTIVITY, MAIL & WEB SERVER

● LINUX, WINDOWS

● DATA / TRANSACTION PROCESSING

● UNIX, IBM OS/360DEFINITIVE

APPS AND OS

● ERP, CRM, PRODUCTIVITY, MAIL & WEB SERVER

● HYPERVISOR + GUEST OS

● BIG DATA, INTERNET OF THINGS, MOBILE APPS

● ???

???DATACENTER

NEW FORM FACTOR FOR DEVELOPING AND

RUNNING APPS

● BIG DATA, INTERNET OF THINGS, MOBILE APPS

● DATACENTER OPERATING SYSTEM (DCOS)

Page 28: EMC World 2016 - Introduction to Mesos and Mesosphere

28© Copyright 2016 EMC Corporation. All rights reserved.

MESOSPHERE DCOS ACCELERATES ENTERPRISES TO A MODERN CLOUD STACK WITH THE DATACENTER FORM

Datacenter Operating System (DCOS)

Distributed Systems Kernel (Apache Mesos)

Big Data + Analytics EnginesMicroservices (in containers)

Streaming

Batch

Machine Learning

Analytics

Functions & Logic

Search

Time Series

SQL / NoSQL

Databases

Modern Cloud App Components

Distributed systems kernel to abstract resources

Killer “apps” ecosystem

Cont Integration/Cont Delivery

Framework “app” architecture to consistently run on top of kernel

User Interface (GUI & CLI)

Core system services (e.g., distributed init, cron, service discovery, package mgt & installer, storage)

Any Infrastructure (Physical, Virtual, Cloud)

Page 29: EMC World 2016 - Introduction to Mesos and Mesosphere

29© Copyright 2016 EMC Corporation. All rights reserved.

DCOS BRINGS HYPERSCALE TO ENTERPRISES

Desktop Apps (e.g., Safari, Adobe Photoshop, Itunes)

GUI (Aqua)/CLI

Personal computer

- App store - OpenGL - Advanced UI gestures

BSD Unix

Mobile Apps (e.g., Spotify, Evernote, WhatsApp)

Android GUI

Mobile devices

- Telephony Manager- Battery management- External storage support

Linux

Distributed systems (e.g., Docker, Spark, Hadoop, Cassandra)

GUI/CLI (DCOS CLI)

Full Datacenter / Cloud

- Container orchestration- Distributed batch jobs- Persistent storage mgmt

Apache MesosKernel

Applications

Form Factor

User Interface

OS Services(highlights)

Computing for the masses Mobile apps for the masses Datacenter-scale apps for mainstream enterprises

Benefits

Page 30: EMC World 2016 - Introduction to Mesos and Mesosphere

30© Copyright 2016 EMC Corporation. All rights reserved. 30© Copyright 2016 EMC Corporation. All rights reserved.

HOW EMC {CODE} FITS WITH MESOS

Page 31: EMC World 2016 - Introduction to Mesos and Mesosphere

31© Copyright 2016 EMC Corporation. All rights reserved.

• Containers are ephemeral therefore stateless• No state? Means boring Application• Container vendors started introducing state by way

of local disks• Something better?• EMC {code} brings storage expertise to the table

HOW EMC {CODE} FITS WITH MESOS?

Page 32: EMC World 2016 - Introduction to Mesos and Mesosphere

32© Copyright 2016 EMC Corporation. All rights reserved.

• Enable external storage• Enhance frameworks to embrace and consume

external storage• Run EMC storage platforms as scale out data services

EMC {CODE} GOALS WITH MESOS

Page 33: EMC World 2016 - Introduction to Mesos and Mesosphere

33© Copyright 2016 EMC Corporation. All rights reserved.

• Persistent– Containers can come and go,

but continue where they left off– Application Data, Logs, Audit

Trail, etc• External

– Storage lives outside the Mesos Agent node

– High Availability– Tolerates node failure

PERSISTENT EXTERNAL STORAGE

Page 34: EMC World 2016 - Introduction to Mesos and Mesosphere

34© Copyright 2016 EMC Corporation. All rights reserved.

• REX-Ray– Provides a vendor agnostic storage orchestration engine– AWS, GCE, ScaleIO, XtremIO, many many more– Github: https://github.com/emccode/rexray

• mesos-module-dvdi– Provides a hook to Mesos agent nodes to provision, attach,

and detach external storage– Github: https://github.com/emccode/mesos-module-dvdi

HOW DO WE ACHIEVE THIS?

Page 35: EMC World 2016 - Introduction to Mesos and Mesosphere

35© Copyright 2016 EMC Corporation. All rights reserved.

• We can run “real world” applications• Persistence – DBs• Highly Available, Fault Tolerant• Audit trail – Transaction Logs• Not worry about the “2am call”

STATELESS TO STATEFUL

Page 36: EMC World 2016 - Introduction to Mesos and Mesosphere

36© Copyright 2016 EMC Corporation. All rights reserved.

THE 2AM CALL…

Page 37: EMC World 2016 - Introduction to Mesos and Mesosphere

37© Copyright 2016 EMC Corporation. All rights reserved. 37© Copyright 2016 EMC Corporation. All rights reserved.

DEMO

Page 38: EMC World 2016 - Introduction to Mesos and Mesosphere

38© Copyright 2016 EMC Corporation. All rights reserved.

DEMO 1: TWEETER - ARCHITECTURE

Datacenter Operating System (DCOS)

Containerized Microservice

Kafka Distributed Messaging system

Cassandra Distributed Datastore

Spark Data Analytics Jobs

Distributed Software Load Balancers

“Tweeter” Service

Page 39: EMC World 2016 - Introduction to Mesos and Mesosphere

39© Copyright 2016 EMC Corporation. All rights reserved.

• Running on Amazon EC2– Mesos– Marathon– Mesos DNS

• Production HA Cluster– 3 x Master + Zookeeper Nodes (t2.micros)– 4 x Slave Nodes with Docker Support (t2.medium)

DEMO 2: CONFIGURATION

Page 40: EMC World 2016 - Introduction to Mesos and Mesosphere

40© Copyright 2016 EMC Corporation. All rights reserved.

• Production Website (MVC)– Web UI (Golang)

• Github: https://github.com/dvonthenen/goprojects/tree/master/src/restclient– REST API (Golang)

• Github: https://github.com/dvonthenen/goprojects/tree/master/src/restserver– PostgreSQL (Docker Container) backed by Persistent

External Storage• Using REX-Ray• Using mesos-module-dvdi

• Simple Web App - User Account Management

DEMO 2: PRODUCTION WEBSITE

Page 41: EMC World 2016 - Introduction to Mesos and Mesosphere

41© Copyright 2016 EMC Corporation. All rights reserved.

• Deploy Model, View, Controller using REST API– Observe creation of each component– Observe PostgreSQL DB attaching to EBS volumes– Using browser, log into Account Management UI and

add/delete records– Observe data from Web UI being added to PostgreSQL table

• Simulate Failure– Kill EC2 Instance– Watch task recreation and reattach to EBS volumes

DEMO 2: PRODUCTION WEBSITE

Page 42: EMC World 2016 - Introduction to Mesos and Mesosphere

42© Copyright 2016 EMC Corporation. All rights reserved.

DEMO 2: INITIAL DEPLOY

Page 43: EMC World 2016 - Introduction to Mesos and Mesosphere

43© Copyright 2016 EMC Corporation. All rights reserved.

DEMO 2: SCALE IT UP

Page 44: EMC World 2016 - Introduction to Mesos and Mesosphere

44© Copyright 2016 EMC Corporation. All rights reserved.

Data Persistence in the New Container WorldWednesday 3PM

Joshua BernsteinVP of Technology for ETD

Tobi Knaup CEO & Co-Founder of Mesosphere

Guru Session

Page 45: EMC World 2016 - Introduction to Mesos and Mesosphere

45© Copyright 2016 EMC Corporation. All rights reserved.

@EMCcode@strikesme@dvonthenen

emccode.comcommunity.emccode.com

Come visit us at Booth #1044 or in the vLab

Questions?

Page 46: EMC World 2016 - Introduction to Mesos and Mesosphere

46© Copyright 2016 EMC Corporation. All rights reserved. 46© Copyright 2016 EMC Corporation. All rights reserved.

Page 47: EMC World 2016 - Introduction to Mesos and Mesosphere