YUMING LIANG VOIDBOX - DOCKER ON YARN

WHAT IS HULU

GAMING • CONNECTED TVS • MOBILE • COMPUTER

540 CONTENT PARTNERS

+!75+ DISTRIBUTION PARTNERS 100+

HULU BEIJING

WHAT IS AUDIENCE PLATFORM

WORKFLOW ENGINE

HADOOP ECOSYSTEM: YARN + HDFS + MAP/REDUCE + HBASE + ZOOKEEPER

CONFIGURATION SERVICE

KILLUA / MATRIX

FIREWORK

SPARK

VOIDBOX DOCKER ON YARN

ESTIMATION SERVICE

NESTO BATCH FLOW: ETL + PROCESSING + SERVING

REAL-TIME FLOW: ETL + PROCESSING + SERVING

Lambda Architecture

SEGMENTATION SYSTEM OLAP

AGENDA

•  Why Voidbox"

•  What is Voidbox"•  Architecture"•  How to use Voidbox"

•  Voidbox in Hulu"•  What’s next"

•  Q&A"

WHY VOIDBOX

YARN: DATA OPERATION SYSTEM

CLUSTER

MAP / REDUCE

TEZ SLIDER

HIVE PIG HBASE STORM

USER APPLICATION

Why only big data application? Can we run other application on yarn? What about task processing, web service?

WHY VOIDBOX

YARN: DISTRIBUTED OPERATION SYSTEM

CLUSTER

MAP / REDUCE

TEZ SLIDER

HIVE PIG HBASE STORM

USER APPLICATION

TASK SYSTEM

WEB SERVICE OTHER

Data App Other App

Is there any demand?

VOIDBOX

WHY VOIDBOX

20+ TASK IN DIFFERENT LANGUAGE

20+ VIRTUAL MACHINE

LOW UTILIZATION

MACHINE FAILURE VOIDBOX

WHAT IS VOIDBOX

LIGHTWEIGHT VM

RUNTIME ISOLATION

UNION FILE SYSTEM

RESOURCE MANAGEMENT

SCHEDULING

FAULT TOLERANCE

DISTRIBUTE COMPUTATION

ARBITRARY APPLICATION

FAULT TOLERANCE

VOIDBOX IS A PROGRAMMING FRAMEWORK

WHAT IS VOIDBOX

Core

VOIDBOX MASTER VOIDBOX PROXY

RUNTIME MODE REGISTRY / MGMT

API

Framework

DAG SERVICE TASK

ARCHITECTURE – DOCKER

Host OS Server

bins/libs

App A

App B

Hypervisor

App A

Guest OS Guest OS

bins/libs bins/libs

App B

Docker Engine

App A

bins/libs bins/libs

App B

Docker Image

Docker Registry

DOCKER IMAGE

DOCKER ENGINE

DOCKER CONTAINER DOCKER

CONTAINER DOCKER CONTAINER

Runtime Isolation

bins/libs

App B’

ARCHITECTURE – DOCKER

ARCHITECTURE – VOIDBOX

YARN modules

Resource Manager

Client

NodeManager

HDFS

NodeManager

HDFS

NodeManager

HDFS

YARN application1

Container (MapTask)

MR AppMaster

Container (MapTask)

Client

Application Master

Container YARN application2

ARCHITECTURE – VOIDBOX

YARN modules

Resource Manager

Voidbox Client

NodeManager

NodeManager

NodeManager

Docker Engine

Docker Engine

Voidbox Master

State Server

Container

Voidbox Proxy

Container

Voidbox Proxy

Docker Registry

In GlusterFs

HDFS

HDFS

HDFS

Voidbox Driver

Pull

Voidbox modules

Docker modules

Voidbox is a standard appliction on top of YARN, just like MapReduce

server

server

server

server

ARCHITECTURE – VOIDBOX YARN CLUSTER MODE

YARN modules

Resource Manager

Voidbox Client

NodeManager

NodeManager

Docker Engine

Voidbox Master

Container

Voidbox Proxy

HDFS

HDFS

Voidbox Driver

Voidbox modules

Docker modules

ARCHITECTURE – VOIDBOX YARN CLIENT MODE

YARN modules

Resource Manager

Voidbox Client

NodeManager

NodeManager

Docker Engine

Voidbox Master

Container

Voidbox Proxy

HDFS

HDFS

Voidbox modules

Docker modules

Voidbox Driver

ARCHITECTURE – VOIDBOX YARN LOCAL MODE

YARN modules

Resource Manager

Voidbox Client

NodeManager

NodeManager

Docker Engine

Voidbox Master

Container

Voidbox Proxy

HDFS

HDFS

Voidbox modules

Docker modules

Voidbox Driver

ARCHITECTURE – VOIDBOX FAULT TOLERANCE

•  Work preserving in case ResourceManager restarts –  https://issues.apache.org/jira/browse/YARN-556 target version:2.6.0 –  Improvement of ResourceManager HA

•  Container preserving in case NodeManager restarts –  https://issues.apache.org/jira/browse/YARN-1336 –  Avoid multiple ApplicationMasters –  Avoid out-of-control containers in NM crashed machine

•  ApplicationMaster retry count reset window –  https://issues.apache.org/jira/browse/YARN-611

•  YARN container -> Docker cli -> Docker container –  Add shutdown hook to recycle docker container in case container unexpected exit –  Deal with voidboxproxy exits without running shutdown hook(kill -9 proxy.pid)

ARCHITECTURE – VOIDBOX FAULT TOLERANCE

yarn-site.xml <property> <name>yarn.resourcemanager.work-preserving-recovery.enabled</name> <value>true</value> </property> <property> <name>yarn.resourcemanager.am.max-attempts</name> <value>100</value> </property> <property> <name>yarn.nodemanager.recovery.enabled</name> <value>true</value> </property>

ApplicationSubmissionContext public abstract void setMaxAppAttempts(int maxAppAttempts); public abstract void setAttemptFailuresValidityInterval(long attemptFailuresValidityInterval); public abstract void setKeepContainersAcrossApplicationAttempts(boolean keepContainers);

ARCHITECTURE – VOIDBOX RESOURCE MANAGEMENT

•  Management Center –  Add/remove Docker engine in cluster –  Garbage collection of out-of-control Container(NodeManage crashes) –  History Log Server –  Service discovery

ARCHITECTURE – LOGS

•  VoidboxMaster’s log –  Specify log4j.properties to rotate master’s log when submit an application

•  Docker container’s log

–  Stdout, stderr log •  Copyfile, truncate(0) to rotate docker container’s log •  Rotate log file in host(default:/var/lib/docker/containers/../containerid-json.log)

–  A log dir in Docker image •  Map specify log volume, by user-defined log rotate in Docker image

CORE

ARCHITECTURE – KEY COMPONENTS

FRAMEWORK

APP

CORE

ARCHITECTURE – KEY COMPONENTS

•  Functionality •  Allocate/release container •  Launch/stop task in container •  Multi run mode: yarn-cluster, yarn-client, yarn-local

•  Component •  VoidboxMaster

•  Container resource allocation/release/preserving •  Container context management

•  VoidboxProxy •  Docker container lifecycle management

•  Management Center •  Docker engine management •  Garbage collection FRAMEWORK

APP

CORE

ARCHITECTURE – KEY COMPONENTS

•  DAG framework •  DAGDriver

•  Task scheduler •  Task retry policy

•  DAGContext •  Describe DAG docker job

•  Service framework •  ServiceDriver

•  Replication controller •  Health check, warm up

•  ServiceContext •  Describe service docker job

FRAMEWORK

APP

CORE

ARCHITECTURE – KEY COMPONENTS

•  DAG APP •  DAG programming based on Framework

•  Service APP •  Service programming based on Framework •  Register to load balance

FRAMEWORK

APP

HOW TO USE VOIDBOX

Advanced API:

•  AMClient –  requestResource(cpu, memory) –  releaseResource(container)

•  DockerTaskRunner –  run(task)

P1

P2

C1

C2

C3

How to increase / decrease number of consumers automatically?

HOW TO USE VOIDBOX

HOW TO USE VOIDBOX

P1

P2

Message Queue Monitor

Consumer Manager

Docker

Consumer

Docker

Consumer

Docker

Consumer

AMClient TaskRunner

VOIDBOX IN HULU

VOIDBOX IN HULU

RESOURCE BOUND HIGHLY PARALLELIZABLE

COMPLEX ENVIRONMENT DEPENDENCIES

VOIDBOX APPLICATION

VOIDBOX BENEFITS

•  Ease creating distributed applications –  Build-in service discovery, fault tolerance, scheduling, communication –  Support complex environment dependencies

•  Improve cluster efficiency –  Share cluster with other data application

•  Simplify deployment –  Package only –  No deployment

•  Flexibility –  Programming interface

WHAT’S NEXT

YARN

VOIDBOX

MESOS

MAP REDUCE / SPARK /

AURORA / MARATHON /

KUBERNETES

DOCKER

M / R SPARK HIVE TEZ

HBASE STORM SLIDER

PAAS SAAS DISTRIBUTED APPLICATION

QUESTIONS Q & A

THANK YOU