Unleash your cluster with YARN

Unleash your cluster with YARN

Ferran Galí i Reniu@ferrangali

About me

@ferrangali

Data

Sensors

Smartphones

User behavior

Social Networks

Text

Numbers

Images

Videos

Source: Intel

http://www.intel.es/content/www/es/es/communications/internet-minute-infographic.html

http://www.intel.es/content/www/es/es/communications/internet-minute-infographic.html

Big Data

100 MB/s

2 TB = 3.5 hours

The Big Data problem

100 MB/s

2 TB = 30 min

The Big Data problem

Source: Google data centers

http://www.google.com/about/datacenters/

http://www.google.com/about/datacenters/

HDFS

Node Node Node Node Node Node Node NodeHardware

HDFS

Node Node Node Node Node Node Node Node

HDFS - Hadoop Distributed File System

Hardware

Storage

HDFS



Hardware

Storage

$> hadoop fs -ls

HDFS

$> hadoop fs -lsFound 2 itemsdrwxr-xr-x - hadoop supergroup 0 2015-06-11 11:27 dir-rw-r--r-- 1 hadoop supergroup 2198927 2015-06-10 17:22 file1.txt

$>

HDFS


$> hadoop fs -ls dir

HDFS


$> hadoop fs -ls dirFound 2 items-rw-r--r-- 1 hadoop supergroup 2198927 2015-06-10 17:22 dir/file2.txt-rw-r--r-- 1 hadoop supergroup 2198927 2015-06-10 17:22 dir/file3.txt

$>

HDFS



$> hadoop fs -cat dir/file3.txt

HDFS



$> hadoop fs -cat dir/file3.txtline1line2line3line4line5

HDFS

MapReduce



Hardware

Storage

MapReduce



Hardware

Storage

Processing

MapReduce Job

Data Pipeline

Application

MapReduce Job

MapReduce Job

MapReduce Job

Data Pipeline

Application

MapReduce Job

Map

Map

Map

Map

MapReduce Job

Split

Split

Split

Split

Map

Map

Map

Map

MapReduce Job

Split

Split

Split

Split

map(){ // Your code here}

Map

Map

Map

Map

Reduce

Reduce

Reduce

MapReduce Job

Split

Split

Split

Split


Map

Map

Map

Map

Reduce

Reduce

Reduce

MapReduce Job

Split

Split

Split

Split


reduce(){ // Your code here}

Map

Map

Map

Map

Reduce

Reduce

Reduce

MapReduce Job

Split

Write

Write

Write

Split

Split

Split


reduce(){ // Your code here}



Hardware

Storage

Processing Job Job

The Big Data problemData Pipeline

Application



Hardware

Storage

Processing Job Job


Application



Hardware

Storage

Processing Job Job


Application



Hardware

Storage

Processing Job Job


Application



Hardware

Storage

Processing Job Job


Application

NodeJobTracker

NodeTaskTracker

MapReduce 1.0 Architecture

NodeTaskTracker

NodeTaskTracker

NodeTaskTracker


NodeTaskTracker

Map Map Map Map

Map Map Map Reduce

Reduce Reduce Reduce Reduce

NodeJobTracker

NodeTaskTracker


NodeTaskTracker

NodeTaskTracker

NodeTaskTracker

Application

NodeJobTracker

NodeTaskTracker


NodeTaskTracker

NodeTaskTracker

NodeTaskTracker

Application

Map

NodeJobTracker

NodeTaskTracker


NodeTaskTracker

NodeTaskTracker

NodeTaskTracker

Application

Reduce

Limitations

Limitations

Limitations

Limitations

Map

Reduce

Map

Map

Reduce

Map

Map

Reduce

Map

MapReduce Job

Limitations

Iterative

MapReduce Job

Limitations

Iterative

Graph Algorithms

MapReduce Job

Limitations

Iterative

Graph Algorithms

YARN



YARN

Hardware

Storage

Resource Manager

The Big Data problemYARN - Yet Another Resource Negotiator

Processing

Cores

Memory

NodeManagerResourceManager

YARN Architecture

NodeManager

x8 x8

x8 x8


YARN Architecture

NodeManager

Applicationx61 core1024MB

x8 x8

x8 x8


YARN Architecture

ApplicationMaster

NodeManager


x8 x8

x8 x8


YARN Architecture

ApplicationMaster

Container

Container

Container

NodeManager

Container

Container


Container

x8 x8

x8 x8


YARN Architecture

ApplicationMaster

Map

Map

Map

NodeManager

Map

Map


Map

x8 x8

x8 x8


YARN Architecture

ApplicationMaster

Reduce

Reduce

Reduce

NodeManager

Reduce

Reduce


Reduce

x8 x8

x8 x8


YARN Architecture

ApplicationMaster

Container

Container

Container

NodeManager

Container

Container

Container


Application 2x42 cores2048MB

x8 x8

x8 x8


YARN Architecture

ApplicationMaster

Container

Container

Container

NodeManager

Container

Container

ApplicationMaster

Container



x8 x8

x8 x8

NodeManager

Container

ResourceManager

YARN Architecture

Container

ApplicationMaster

Container

Container

Container

NodeManager

Container

Container

Container

ApplicationMaster

Container

Container



x8 x8

x8 x8

NodeManager

Container

ResourceManager

YARN Architecture

Container

ApplicationMaster

Container

Container

Container

NodeManager

Container

Container

Container

ApplicationMaster

Container

Container

Applicationx61 core1024MBetl

Application 2x42 cores2048MBquery

x8 x8

x8 x8

scheduleretl: weight 1

query: weight 2



YARN

Hardware

Storage

Resource Manager

The Big Data problemNew Paradigms

Processing



YARN

Hardware

Storage

Resource Manager


Processing

Application

Batch



YARN

Hardware

Storage

Resource Manager


Processing

Application

In Memory / Streaming



YARN

Hardware

Storage

Resource Manager


Processing

Application

Interactive SQL



YARN

Hardware

Storage

Resource Manager


Processing ...

Application

Improved Data Pipelines

Map

Reduce

Map

Map

Reduce

Map

Map

Reduce

Map


Map

Reduce

Map

Map

Reduce

Map

Reduce

MapReduce Job

MapReduce Job

MapReduce Job


Application


MapReduce Job

Spark JobMapReduce

Job

Application


MapReduce Job

Spark Job

Application

Trovit

+70 MapReduce Jobs adding business value

Multi-tenant cluster executing +7000 jobs per day

Trovit



Search engine

Trovit



Business Intelligence

Search engine

Trovit




Search engine

Mailing

Trovit




Search engine

Mailing Push Notifications

Trovit




Search engine

Mailing Push Notifications

Online Media Buying

Challenges

Trovit

Maintain



Challenges

Trovit

Maintain Try new paradigms



Challenges

Trovit

Maintain Try new paradigms

Fine tune



Trovit

Data Analysis with SQL on Hadoop



Trovit




HiveSQL M/R

Trovit




HiveSQL M/R

Sqoop onMySQL

Challenges

Trovit




ImpalaInteractive

Challenges

Trovit




ImpalaInteractive

Machine Learning

Trovit


Near Real Time on a Storm cluster



Separated Cluster

Challenges

Trovit


Near Real Time on a Storm cluster



Storm on YARN

Questions?

Thank YouFerran Galí i Reniu

@ferrangali

Icons made by Freepik from Flaticon is licensed by CC BY 3.0

http://www.flaticon.com/authors/freepik

http://www.flaticon.com

http://creativecommons.org/licenses/by/3.0/

Data & Analytics

Unleash your cluster with YARN