Slider: Applications on YARNSteve Loughran– Hortonworks

stevel at hortonworks.com

@steveloughran

ApacheCon EU, November 2014

log

Modern applications

front end

web

phones

devices

feeds log

stream

processing

front end

front end

log

database

analytics

Fun operational problems

Placement: where to run?

Installation

Configuration & Binding

Client configuration

Lifecycle

Failure handing and recovery

Logging

Upgrading

Metrics & Monitoring

– Start/Stop

– Reconfigure

– Scale up/down

– Rolling-restart

– Decommission/Recommission

Getting your code to work across a set of fairly reliable machines

Apache Slider

Deploying and managingapplications on Apache Hadoop YARN

http://slider.incubator.apache.org/

HDFS

YARN Node Manager

HDFS

YARN Node Manager

HDFS

YARN Resource Manager

“The RM”

HDFS

YARN Node Manager

• Servers run YARN Node Managers (NM)

• NM's heartbeat to Resource Manager (RM)

• RM schedules work over cluster

• RM allocates containers to apps

• NMs start containers

• NMs report container health

Background: YARN

Client creates App Master

HDFS

YARN Node Manager

HDFS

YARN Node Manager

HDFS

YARN Resource Manager

“The RM”

HDFS

YARN Node Manager

ClientSlider Application Master

Slider AM requests containers

HDFS

YARN Node Manager

HDFS

YARN Node Manager

HDFS

YARN Resource Manager

HDFS

YARN Node Manager

Slider Application Master

HBase Master

HBase Region Server

HBase Region Server

Slider: architecture

1. AppMaster

2. AgentProvider

3. Agent

4. AppPackage

5. CLI

Slider

App

Package

HDFS

YARN Resource Manager

“The RM”

HDFS

YARN Node Manager

Agent Component

HDFS

YARN Node Manager

Agent Component

Slider Application Master

Slider CLI

Desired State, agents & scripts

Slider

App

PackageCLI

HDFS

YARN Resource Manager

“The RM”

HDFS

YARN Node Manager

Agent Component

HDFS

YARN Node Manager

Agent Component

1. CLI starts an instance of the AM

2. AM requests containers

3. Containers activate with an Agent

4. Agent gets application definition

5. Agent registers with Slider AM

6. AM issues commands

7. Agent reports back, status,

configuration, etc.

8. AM publishes endpoints,

configurations

Slider App Master

Registry

© Hortonworks Inc. 2014:

App Package: zip XML .py

App package Metainfo XML

Page 12

<service>

<name>HBASE</name>

<version>0.96.0.2.1.1</version>

<exportGroups><exportGroup>

<name>QuickLinks</name>

<exports><export>

<name>org.apache.slider.jmx</name>

<value>http://${HBASE_MASTER_HOST}:${site.hbase-site.hbase.master.info.port}/jmx</value>

</export></exports></exportGroup></exportGroups>

<commandOrders><commandOrder>

<command>HBASE_REGIONSERVER-START</command>

<requires>HBASE_MASTER-STARTED</requires>

</commandOrder></commandOrders>

<components><component>

<name>HBASE_MASTER</name>

<minInstanceCount>1</minInstanceCount>

<commandScript>

<script>scripts/hbase_master.py</script>

</commandScript></component></components>

</service>

Application Info

Dependencies

Publish an URI

Contains

components

scripts

deploy: resource.json{

"schema": "http://example.org/specification/v2.0.0",

"global": {

"yarn.memory": "512"

},

"components": {

"HBASE_MASTER": {

"yarn.role.priority": "1",

"yarn.component.instances": "1",

"yarn.memory": "1024",

"yarn.vcores": "1",

},

"HBASE_REGIONSERVER": {

"yarn.role.priority": "2",

"yarn.component.instances": "1"

}

} }

configure: app_config.json

{

"application.def": "/slider/hbase_v096.zip",

"site.global.app_log_dir": "${AGENT_LOG_ROOT}/app/log",

"site.global.app_pid_dir": "${AGENT_WORK_ROOT}/app/run",

"site.global.hbase_master_heapsize": "1024m",

"site.global.ganglia_server_host": "${NN_HOST}",

"site.global.ganglia_server_port": "8667",

"site.global.ganglia_server_id": "Application1",

"site.hbase-site.hbase.tmp.dir": "${AGENT_WORK_ROOT}/work/app/tmp",

"site.hbase-site.hbase.master.info.port": "${HBASE_MASTER.ALLOCATED_PORT}",

"site.hbase-site.hbase.regionserver.port": "0",

"site.hbase-site.hbase.zookeeper.quorum": "${ZK_HOST}",

"site.core-site.fs.defaultFS": "${NN_URI}",

}

Configurations needed by

Slider

Named variables

Site variables for

application

Named variables

for cluster details

Allocate and

advertise

Variables for the

application scripts

create, start, stop, destroy

$ slider create hbase1 --resources resources.json --template appConfig.json

$ slider list

$ slider status hbase1

$ slider stop hbase1

$ slider start hbase1

$ slider stop hbase1--force

$ slider destroy hbase1

Dynamic application resize

slider flex hbase1--component HBASE_REGIONSERVER 2

Registry lookup

$ slider resolve --path ~/services/org-apache-slider/storm1

{ "type" : "JSONServiceRecord","external" : [ {

"api" : "http://","addressType" : "uri","protocolType" : "webui","addresses" : [ {

"uri" : "http://nn.example.com:46132"} ]

}, {"api" : "classpath:org.apache.slider.publisher.configurations","addressType" : "uri","protocolType" : "REST","addresses" : [ {

"uri" : "http://nn.example.com:46132/ws/v1/slider/publisher/slider"} ]

} } ] }

Failures

HDFS

YARN Node Manager

HDFS

YARN Node Manager

HDFS

YARN Resource Manager

HDFS

YARN Node Manager

Slider App Master

HBase Master

HBase Region Server

HBase Region Server

YARN notifies AM

HDFS

YARN Node Manager

HDFS

YARN Node Manager

HDFS

YARN Resource Manager

HDFS

YARN Node Manager

Slider App Master

HBase Master

HBase Region Server

request on old node (lax)

HDFS

YARN Node Manager

HDFS

YARN Node Manager

HDFS

YARN Resource Manager

HDFS

YARN Node Manager

Slider App Master

HBase Master

HBase Region Server

HBase Region Server

Summary

• XML + python defined app deployment

• Declarative YARN resource specification

• Declarative application configuration

• Slider AM keeps app in desired state

• Lifecycle commands

• Binding and configuration via registry

Slider View for Apache Ambari

http://ambari.apache.org/

Managed Slider

Ambari

Slider

App

Package

HDFS

YARN Resource Manager

“The RM”

HDFS

YARN Node Manager

Agent Component

HDFS

YARN Node Manager

Agent Component

Slider App Master

Registry

• Ambari imports app packages

• Starts the AM

• Interacts with AM to start

containers

• Agents register with Ambari

• Ambari sends commands/

receives results

• YARN manages containers

• Ambari interacts with Registry Registry

Project Status

• In ASF incubation

• 0.60-incubating release this week(!)

• Need more developers!

• Need more application packages!

If you want to run existing appsin a YARN cluster:

Get involved!

http://slider.incubator.apache.org/