Deploying Docker applications on YARN via Slider

  • Published on

  • View

  • Download

Embed Size (px)



Deploying Docker applications on YARN via SliderHW Slider teamPresenter: Thomas LiuPage 1 Hortonworks Inc. 2015Slider is an incubation project with one goal:

Make it easy to deploy and manage applications in a Hadoop cluster with YARNPage 2 Hortonworks Inc. 2015Project StatusIn ASF incubation0.70-incubating release this monthGrowing set of application packagesWorking in simplifying packagingHDP 2.2 added Slider component for easy on-boarding of long-lived service on YARN Hortonworks Inc.An application consists ofApplication PackageJSON configuration files: YARN resources + configData persisted in HDFS (May)Placement history in HDFSPage 4 Hortonworks Inc.Slider: architectureAppMasterAgentProviderAgentAppPackageCLI

SliderApp config filesHDFSYARN Resource ManagerThe RMHDFSYARN Node ManagerAgentComponentHDFSYARN Node ManagerAgentComponentSlider Application MasterSlider CLI Hortonworks Inc. 2015Slider App Master runs the applicationHDFSYARN Node ManagerHDFSYARN Node ManagerHDFSYARN Resource ManagerHDFSYARN Node ManagerSlider Application MasterRedis ServerNodejs Server Hortonworks Inc. 2015On-boarding Dockerized Application on YARN via SliderApplication definition metainfo.json (contains docker image name)Application instance configuration (if any) app_config.jsonApplication resource definition resources.json

No Application bundle and lifecycle scripts

Page 7 Hortonworks Inc.Demo Dockerized node.js app using dockerized redis for persistence Page 8 Hortonworks Inc. 2015metainfo.json structure of the application{ "schemaVersion": "2.1", "application": { "name": "NODEJS-REDIS", "components": [ { "name": "NODEJS", "type": "docker", "dockerContainers": [ { "name": "nodejs, "image": "rsahahw/centos-node-redis, "ports": [{"containerPort" : "8000 }] } ] }, { "name": "REDIS", "type": "docker", "dockerContainers": [ { "name": "redis", "image": "tutum/redis", "ports": [{ "containerPort" : "6379", "hostPort": "6379 ...

Hortonworks Inc.resources.json resource requirement of the application { "schema" : "", "metadata" : { }, "global" : { }, "components": { "NODEJS": { "yarn.role.priority": "1", "yarn.component.instances": "1", "yarn.memory": "512" }, "REDIS": { "yarn.role.priority": "2", "yarn.component.instances": "1", "yarn.memory": "512" } }}

Hortonworks Incapp_config.json instance specific configuration of the application { "schema": "", "metadata": { }, "global": { }, "components": { "NODEJS": { "nodejs.options":"-d -e REDIS_HOST=${REDIS_HOST}", "nodejs.statusCommand":"/usr/bin/docker ps" }, "REDIS": { "redis.options":"-d -e REDIS_PASS=**None**", "redis.statusCommand":"/usr/bin/docker ps" } }} Hortonworks Inc.Commands: create, start, stop, destroy$ slider create dockerapp1 --resources resources.json -template appConfig.json -metainfo metainfo.json

$ slider list

$ slider stop dockerapp1

$ slider destroy dockerapp1 Hortonworks IncHow to get startedUse Pre-requisite Hadoop 2.6 or above [ or HDP 2.2 ]Build Slider [ link ] Author slider configuration files for your dockerized app or use examples

ContributeExplore Slider @ ideas Aligning Docker security model with YARNsDocker images with multiple initialization stepsActive resource monitoring docker containers by YARN

Hortonworks Inc.Q&A Hortonworks Inc. 201514


View more >