© Hortonworks Inc. 2015

Deploying Docker applications on YARN

via Slider

HW Slider team

Presenter: Thomas Liu

Page 1

© Hortonworks Inc. 2015

Slider is an incubation project with one goal:

Make it easy to deploy and manage

applications in a Hadoop cluster with YARN

Page 2

© Hortonworks Inc.

Project Status

• In ASF incubation

• 0.70-incubating release this month

• Growing set of application packages

• Working in simplifying packaging

• HDP 2.2 added Slider component for easy on-boarding of

long-lived service on YARN

http://slider.incubator.apache.org/

© Hortonworks Inc.

An application consists of

“Application Package”

JSON configuration files: YARN resources + config

Data persisted in HDFS (May)

Placement history in HDFS

Page 4

© Hortonworks Inc. 2015

Slider: architecture

1. AppMaster

2. AgentProvider

3. Agent

4. AppPackage

5. CLI

Slider

App

config

files

HDFS

YARN Resource Manager

“The RM”

HDFS

YARN Node Manager

Agent Component

HDFS

YARN Node Manager

Agent Component

Slider Application Master

Slider CLI

© Hortonworks Inc. 2015

Slider App Master runs the application

HDFS

YARN Node Manager

HDFS

YARN Node Manager

HDFS

YARN Resource Manager

HDFS

YARN Node Manager

Slider Application Master

Redis Server

Nodejs Server

© Hortonworks Inc.

On-boarding Dockerized Application on YARN via Slider

• Application definition – metainfo.json (contains docker image name)

• Application instance configuration (if any) – app_config.json

• Application resource definition – resources.json

No Application bundle and lifecycle scripts

Page 7

© Hortonworks Inc. 2015

Demo Dockerized node.js app using dockerized redis for persistence

Page 8

© Hortonworks Inc.

metainfo.json – structure of the application

{

"schemaVersion": "2.1",

"application": {

"name": "NODEJS-REDIS",

"components": [

{

"name": "NODEJS",

"type": "docker",

"dockerContainers": [

{

"name": "nodejs”,

"image": "rsahahw/centos-node-redis”,

"ports": [{"containerPort" : "8000” }]

}

]

},

{

"name": "REDIS",

"type": "docker",

"dockerContainers": [

{

"name": "redis",

"image": "tutum/redis",

"ports": [{

"containerPort" : "6379",

"hostPort": "6379”

...

© Hortonworks Inc

resources.json – resource requirement of the application

{

"schema" : "http://example.org/specification/v2.0.0",

"metadata" : {

},

"global" : {

},

"components": {

"NODEJS": {

"yarn.role.priority": "1",

"yarn.component.instances": "1",

"yarn.memory": "512"

},

"REDIS": {

"yarn.role.priority": "2",

"yarn.component.instances": "1",

"yarn.memory": "512"

}

}

}

© Hortonworks Inc.

app_config.json – instance specific configuration of the application

{

"schema": "http://example.org/specification/v2.0.0",

"metadata": {

},

"global": {

},

"components": {

"NODEJS": {

"nodejs.options":"-d -e REDIS_HOST=${REDIS_HOST}",

"nodejs.statusCommand":"/usr/bin/docker ps"

},

"REDIS": {

"redis.options":"-d -e REDIS_PASS=**None**",

"redis.statusCommand":"/usr/bin/docker ps"

}

}

}

© Hortonworks Inc

Commands: create, start, stop, destroy

$ slider create dockerapp1 --resources resources.json -–template appConfig.json -–metainfo metainfo.json

$ slider list

$ slider stop dockerapp1

$ slider destroy dockerapp1

© Hortonworks Inc.

How to get started

• Use

–Pre-requisite – Hadoop 2.6 or above [ or HDP 2.2 ]

–Build Slider [ link ]

–Author slider configuration files for your dockerized app or use

examples

• Contribute

–Explore Slider @ http://slider.incubator.apache.org/

–Few ideas

– Aligning Docker security model with YARN’s

– Docker images with multiple initialization steps

– Active resource monitoring docker containers by YARN

© Hortonworks Inc. 2015

Q&A