Hado"ops" or Had"oops"

Proprietary amp Confidential Copyright copy 2014

HadorsquoopsrsquoorHadrsquooopsrsquo 1

Wersquore Hiringrocketfuelcomcareers

Kishore Kumar YellamrajuAbhijit Pol

The Web Is Monetized By Advertising

Delivery Methods

raquoDisplayraquoVideoraquoMobileraquoSocial

6 Ad Served

User Segments

3 Bid Reques

Overview

Publishers

2 Ad Request

1 Page Request

4 Bid amp Ad

User Engagemen

Data Partners

Advertisers

Browser

Some Exchange Partners

Ad Exchange

Optimize

Rocket Fuel Platform

Real-time BidderAutomated Decisions

Models

Refresh learning

Data Store

Ads ampBudget

ModelScores

Events

5 RocketfuelWinning Ad

$238965$06782$17234

$009$178964$16782$17234$0809$242125

$211$126

$2178$2056$0809$242125

$211$126$278$156

$1809$242125

$211$126$278$056$242125

$211$126$278

$0756$0809$242125

$211$126$278

$1256$1809$242125

$211$126$278

$0586$2009

125$211$126$278$156

[ + ][ + ]

SitePageGeoWeatherTime of DayBrand AffinityUser

Always buying the best impressions amp serving the best ad

Real Time Bidding and Serving

GoalLeadsamp sales

GoalCoupondownloads

GoalBrandawareness

SitePageGeoWeatherTime of DayBrand AffinityDemo

Impression ScorecardDemoBrand AffinityTime of DayGeoWeatherSitePageAd PositionIn-marketBehaviorResponse

Impression ScorecardDemoBrand AffinityTime of DayGeoWeatherSitePageAd PositionIn-MarketBehaviorResponse X

Impression ScorecardDemoBrand AffinityTime of DayGeoWeatherSitePageAd PositionIn-MarketBehaviorResponse

+100+40-20+20+15+10+40+35

+40-70-20+10+15-25-40-18

+10-10-20+20+10-35-25+10

6 Ad Served

User Segments

3 Bid Reques

Overview

Publishers

2 Ad Request

1 Page Request

4 Bid amp Ad

User Engagemen

Data Partners

Advertisers

Browser

Ad Exchange

Optimize

Models

Refresh learning

Data Store

Ads ampBudget

ModelScores

Events

Facebook likes

Searches on Google

Bid Requests Considered by Rocketfuel

Requests per day

Throughput

Blink of an eye

SF to Tokyo network round trip

One beat of a hummindbirds wing

Look up in Blackbird

Time (ms)

Latency

Architecture and Scale

raquoDatacentersraquoScaleraquoGrowthraquoArchitecture

Data Center Expansion

raquoabc

Data Center Design

bull Racks custom built at Rocket Fuelbull Leased spacebandwidth in colocation facilities

Hadoop Server20 2U servers (85kW)

Bidders40 2-U Twin 2 servers (17kW)

Rocket Fuel Scale

raquo34474 CPU processor coresndash2655 serversndash1874 Teraflops of computing

raquo188 Terabytes of memoryndash13X the memory of IBM computer Watson that

played Jeopardy

raquo42PB Petabytes of storagendash106X the data volume of the entire Library of

Congress

Hadoop at Rocket Fuel

raquo 1400 servers

raquo 15K Disks

raquo 15K Cores

raquo 90 TB

raquo 30K MR slots

raquo 12K daily MR jobs

200 Servers 1400 Servers

1 Year

41 PB8x

Growth

Data Architecture 30

Hadoop Setup

QJM ZK Quorum

raquo 6x2TB Disksraquo 2x6 coreraquo 196 GB RAMraquo 2x1G NIC

raquo 12x3TB Disksraquo 2x6 coreraquo 64 GB RAMraquo 10G NIC

raquo same as DNrsquosraquo Dedicated disk

to ZK or JN

Standby NN

ZKFCZKFC

Active NN

Operations

raquo Maintenanceraquo Performance Tuningraquo Monitoringraquo BCPraquo YARN

Puppet+

Infradb

Automation is key

Maintenance is Not Easy

Puppet and Infradb

raquo Automate as much as you canraquo Adding a slave node to Hadoop cluster lt 120 secondsraquo Bringing up a new Hadoop cluster lt 500 secondsraquo MR slots are automatically determined based on hardware config

Isnrsquot it cool

Just define once

No issues when cluster is small Problems starts when it grows

Performance Tuning

dfsdatanodehandlercount dfsnamenodehandlercount

dfsdatanodemaxtransferthreads dfsimagetransfertimeout

mapredreduceparallelcopies

mapredjobtrackerhandlercount

iosortmbiosortfactor

maxClientCnxns ZK

IMP MAPREDUCE-2026

-XX+UseConcMarkSweepGC

-XXCMSFullGCsBeforeCompaction=1

-XXCMSInitiatingOccupancyFraction=60

ha-timeoutms

Performance Tuning

mapreducereduceshuffleparallelcopies

MAPREDUCE-5351

MAPREDUCE-5508

keepfailedtaskfiles=true

We Have an Issue

instances of JobInProgressrdquo class = no of users submitted jobs X mapredjobtrackercompleteuserjobsmaximum

mapredjobtrackercompleteuserjobsmaximum mapredjobtrackerretirejobinterval

mapredjobtrackerretiredjobscachesize

JT OOM

Operations

Monitoring

Wall of Ops

Monitoring

hadoopnamenodeCallQueueLength hadoopjobtrackerjvmmemheapusedm

Donrsquot fly blind you will crash

MR Workload Monitoring

Network Monitoring

Donrsquot blame network instead monitor it Network Mesh can be mess

Alerting

Monitoring is not enough need better Alerting

Alerts

httphostnameportjmx

qry=Hadoopservice=NameNodename=NameNodeInfo

gtgt Checking whether NN and JT are up is a no brainer gtgt Reduce alert noise by having summaryaggregate alerts gtgt We heavily rely on custom scripts that query jmx for NN and JT

qry=hadoopservice=JobTrackername=JobTrackerInfo

NameDirStatuses DeadNodes NumberOfMissingBlocks

qry=Hadoopservice=NameNodename=FSNamesystemState

FSState CapacityRemaining NumDeadDataNodes UnderReplicatedBlocks

Blacklisted TTrsquos jobs slots_used ThreadCount

qry=javalangtype=Memory

Used jvm free jvm etc

MR Workload Alerting

raquo Monitoring MR workload and alertndash In-house tool that use ldquohoudahrdquo ruby gem monitorsndash Long running jobs jobs with more map tasks blacklisted TTrsquos

with more failure counts etchellip

raquo Collect details and auto-restart blacklisted TTrsquosraquo Parse the JT logfile for rouge jobsraquo Parse the JT log and collects all Job related inforaquo White-elephant or hraven could helpraquo Parse the scheduler html page or use metrics page httpltJT-hostnamegt50030scheduleradvanced httpltJT-hostnamegt50030metrics

Modeling

Ad-hoc

Multi Tenancy

No Scheduler is perfect unless you understand and tune it properly

Scheduling

Operations

raquo BCP Business Continuity Planraquo Near real time reporting over 15+ TB of daily dataraquo Freshness of models trained over petabytes of data

Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Amazon BackupLSV Data

Cluster

USEUHK Serving Clusters

Research

Ad-hoc Queries

Processed Data

JobTracker

raquo Resource Manager - Global resource scheduler - Hierarchical queues - Application management

raquo Node Manager - Per-machine agent - Manages life cycle of container - Container resource monitoring

raquo Application Master - Per-application - Manages application scheduling and task execution

YARN at Rocket FueI

raquo Yarn is in production raquo 700+ nodesraquo 31TB RAM 8500 disks 8500 cores raquo Primary use case Map-Reduceraquo No more static slotsraquo Tez Spark Storm are in race

Obligatory ldquowe are hiringrdquo slide

httprocketfuelcomcareers

THANKS

kishorerocketfuelcomapolrocketfuelcom

Hadorsquoopsrsquo or Hadrsquooopsrsquo 1

Delivery Methods

Overview

Overview (2)

Throughput

Latency

Data Center Design

Rocket Fuel Scale

Growth

Hadoop Setup

Operations

Puppet and Infradb

Performance Tuning

Performance Tuning (2)

We Have an Issue

JT OOM

Operations (2)

Monitoring

Monitoring (2)

Network Monitoring

Alerting

Alerts

Multi Tenancy

Scheduling

Operations (3)

Data BCP Cluster

YARN at Rocket FueI

Delivery Methods

6 Ad Served

User Segments

3 Bid Reques

Overview

Publishers

2 Ad Request

1 Page Request

4 Bid amp Ad

User Engagemen

Data Partners

Advertisers

Browser

Ad Exchange

Optimize

Models

Refresh learning

Data Store

Ads ampBudget

ModelScores

Events

$238965$06782$17234

$009$178964$16782$17234$0809$242125

$211$126

$2178$2056$0809$242125

$211$126$278$156

$1809$242125

$211$126$278$056$242125

$211$126$278

$0756$0809$242125

$211$126$278

$1256$1809$242125

$211$126$278

$0586$2009

125$211$126$278$156

[ + ][ + ]

GoalLeadsamp sales

GoalCoupondownloads

GoalBrandawareness

+100+40-20+20+15+10+40+35

+40-70-20+10+15-25-40-18

+10-10-20+20+10-35-25+10

6 Ad Served

User Segments

3 Bid Reques

Overview

Publishers

2 Ad Request

1 Page Request

4 Bid amp Ad

User Engagemen

Data Partners

Advertisers

Browser

Ad Exchange

Optimize

Models

Refresh learning

Data Store

Ads ampBudget

ModelScores

Events

Facebook likes

Searches on Google

Requests per day

Throughput

Blink of an eye

Time (ms)

Latency

raquoabc

Data Center Design

Rocket Fuel Scale

played Jeopardy

Congress

raquo 1400 servers

raquo 15K Disks

raquo 15K Cores

raquo 90 TB

raquo 30K MR slots

1 Year

41 PB8x

Growth

Hadoop Setup

QJM ZK Quorum

to ZK or JN

Standby NN

ZKFCZKFC

Active NN

Operations

Puppet+

Infradb

Automation is key

Puppet and Infradb

Isnrsquot it cool

Just define once

Performance Tuning

maxClientCnxns ZK

IMP MAPREDUCE-2026

ha-timeoutms

Performance Tuning

MAPREDUCE-5351

MAPREDUCE-5508

We Have an Issue

JT OOM

Operations

Monitoring

Wall of Ops

Monitoring

Network Monitoring

Alerting

Alerts

httphostnameportjmx

Modeling

Ad-hoc

Multi Tenancy

Scheduling

Operations

Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Cluster

Research

Ad-hoc Queries

Processed Data

JobTracker

YARN at Rocket FueI

THANKS

Delivery Methods

Overview

Overview (2)

Throughput

Latency

Data Center Design

Rocket Fuel Scale

Growth

Hadoop Setup

Operations

Puppet and Infradb

Performance Tuning

We Have an Issue

JT OOM

Operations (2)

Monitoring

Monitoring (2)

Network Monitoring

Alerting

Alerts

Multi Tenancy

Scheduling

Operations (3)

Data BCP Cluster

YARN at Rocket FueI

Delivery Methods

6 Ad Served

User Segments

3 Bid Reques

Overview

Publishers

2 Ad Request

1 Page Request

4 Bid amp Ad

User Engagemen

Data Partners

Advertisers

Browser

Ad Exchange

Optimize

Models

Refresh learning

Data Store

Ads ampBudget

ModelScores

Events

$238965$06782$17234

$009$178964$16782$17234$0809$242125

$211$126

$2178$2056$0809$242125

$211$126$278$156

$1809$242125

$211$126$278$056$242125

$211$126$278

$0756$0809$242125

$211$126$278

$1256$1809$242125

$211$126$278

$0586$2009

125$211$126$278$156

[ + ][ + ]

GoalLeadsamp sales

GoalCoupondownloads

GoalBrandawareness

+100+40-20+20+15+10+40+35

+40-70-20+10+15-25-40-18

+10-10-20+20+10-35-25+10

6 Ad Served

User Segments

3 Bid Reques

Overview

Publishers

2 Ad Request

1 Page Request

4 Bid amp Ad

User Engagemen

Data Partners

Advertisers

Browser

Ad Exchange

Optimize

Models

Refresh learning

Data Store

Ads ampBudget

ModelScores

Events

Facebook likes

Searches on Google

Requests per day

Throughput

Blink of an eye

Time (ms)

Latency

raquoabc

Data Center Design

Rocket Fuel Scale

played Jeopardy

Congress

raquo 1400 servers

raquo 15K Disks

raquo 15K Cores

raquo 90 TB

raquo 30K MR slots

1 Year

41 PB8x

Growth

Hadoop Setup

QJM ZK Quorum

to ZK or JN

Standby NN

ZKFCZKFC

Active NN

Operations

Puppet+

Infradb

Automation is key

Puppet and Infradb

Isnrsquot it cool

Just define once

Performance Tuning

maxClientCnxns ZK

IMP MAPREDUCE-2026

ha-timeoutms

Performance Tuning

MAPREDUCE-5351

MAPREDUCE-5508

We Have an Issue

JT OOM

Operations

Monitoring

Wall of Ops

Monitoring

Network Monitoring

Alerting

Alerts

httphostnameportjmx

Modeling

Ad-hoc

Multi Tenancy

Scheduling

Operations

Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Cluster

Research

Ad-hoc Queries

Processed Data

JobTracker

YARN at Rocket FueI

THANKS

Delivery Methods

Overview

Overview (2)

Throughput

Latency

Data Center Design

Rocket Fuel Scale

Growth

Hadoop Setup

Operations

Puppet and Infradb

Performance Tuning

We Have an Issue

JT OOM

Operations (2)

Monitoring

Monitoring (2)

Network Monitoring

Alerting

Alerts

Multi Tenancy

Scheduling

Operations (3)

Data BCP Cluster

YARN at Rocket FueI

6 Ad Served

User Segments

3 Bid Reques

Overview

Publishers

2 Ad Request

1 Page Request

4 Bid amp Ad

User Engagemen

Data Partners

Advertisers

Browser

Ad Exchange

Optimize

Models

Refresh learning

Data Store

Ads ampBudget

ModelScores

Events

$238965$06782$17234

$009$178964$16782$17234$0809$242125

$211$126

$2178$2056$0809$242125

$211$126$278$156

$1809$242125

$211$126$278$056$242125

$211$126$278

$0756$0809$242125

$211$126$278

$1256$1809$242125

$211$126$278

$0586$2009

125$211$126$278$156

[ + ][ + ]

GoalLeadsamp sales

GoalCoupondownloads

GoalBrandawareness

+100+40-20+20+15+10+40+35

+40-70-20+10+15-25-40-18

+10-10-20+20+10-35-25+10

6 Ad Served

User Segments

3 Bid Reques

Overview

Publishers

2 Ad Request

1 Page Request

4 Bid amp Ad

User Engagemen

Data Partners

Advertisers

Browser

Ad Exchange

Optimize

Models

Refresh learning

Data Store

Ads ampBudget

ModelScores

Events

Facebook likes

Searches on Google

Requests per day

Throughput

Blink of an eye

Time (ms)

Latency

raquoabc

Data Center Design

Rocket Fuel Scale

played Jeopardy

Congress

raquo 1400 servers

raquo 15K Disks

raquo 15K Cores

raquo 90 TB

raquo 30K MR slots

1 Year

41 PB8x

Growth

Hadoop Setup

QJM ZK Quorum

to ZK or JN

Standby NN

ZKFCZKFC

Active NN

Operations

Puppet+

Infradb

Automation is key

Puppet and Infradb

Isnrsquot it cool

Just define once

Performance Tuning

maxClientCnxns ZK

IMP MAPREDUCE-2026

ha-timeoutms

Performance Tuning

MAPREDUCE-5351

MAPREDUCE-5508

We Have an Issue

JT OOM

Operations

Monitoring

Wall of Ops

Monitoring

Network Monitoring

Alerting

Alerts

httphostnameportjmx

Modeling

Ad-hoc

Multi Tenancy

Scheduling

Operations

Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Cluster

Research

Ad-hoc Queries

Processed Data

JobTracker

YARN at Rocket FueI

THANKS

Delivery Methods

Overview

Overview (2)

Throughput

Latency

Data Center Design

Rocket Fuel Scale

Growth

Hadoop Setup

Operations

Puppet and Infradb

Performance Tuning

We Have an Issue

JT OOM

Operations (2)

Monitoring

Monitoring (2)

Network Monitoring

Alerting

Alerts

Multi Tenancy

Scheduling

Operations (3)

Data BCP Cluster

YARN at Rocket FueI

$238965$06782$17234

$009$178964$16782$17234$0809$242125

$211$126

$2178$2056$0809$242125

$211$126$278$156

$1809$242125

$211$126$278$056$242125

$211$126$278

$0756$0809$242125

$211$126$278

$1256$1809$242125

$211$126$278

$0586$2009

125$211$126$278$156

[ + ][ + ]

GoalLeadsamp sales

GoalCoupondownloads

GoalBrandawareness

+100+40-20+20+15+10+40+35

+40-70-20+10+15-25-40-18

+10-10-20+20+10-35-25+10

6 Ad Served

User Segments

3 Bid Reques

Overview

Publishers

2 Ad Request

1 Page Request

4 Bid amp Ad

User Engagemen

Data Partners

Advertisers

Browser

Ad Exchange

Optimize

Models

Refresh learning

Data Store

Ads ampBudget

ModelScores

Events

Facebook likes

Searches on Google

Requests per day

Throughput

Blink of an eye

Time (ms)

Latency

raquoabc

Data Center Design

Rocket Fuel Scale

played Jeopardy

Congress

raquo 1400 servers

raquo 15K Disks

raquo 15K Cores

raquo 90 TB

raquo 30K MR slots

1 Year

41 PB8x

Growth

Hadoop Setup

QJM ZK Quorum

to ZK or JN

Standby NN

ZKFCZKFC

Active NN

Operations

Puppet+

Infradb

Automation is key

Puppet and Infradb

Isnrsquot it cool

Just define once

Performance Tuning

maxClientCnxns ZK

IMP MAPREDUCE-2026

ha-timeoutms

Performance Tuning

MAPREDUCE-5351

MAPREDUCE-5508

We Have an Issue

JT OOM

Operations

Monitoring

Wall of Ops

Monitoring

Network Monitoring

Alerting

Alerts

httphostnameportjmx

Modeling

Ad-hoc

Multi Tenancy

Scheduling

Operations

Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Cluster

Research

Ad-hoc Queries

Processed Data

JobTracker

YARN at Rocket FueI

THANKS

Delivery Methods

Overview

Overview (2)

Throughput

Latency

Data Center Design

Rocket Fuel Scale

Growth

Hadoop Setup

Operations

Puppet and Infradb

Performance Tuning

We Have an Issue

JT OOM

Operations (2)

Monitoring

Monitoring (2)

Network Monitoring

Alerting

Alerts

Multi Tenancy

Scheduling

Operations (3)

Data BCP Cluster

YARN at Rocket FueI

GoalLeadsamp sales

GoalCoupondownloads

GoalBrandawareness

+100+40-20+20+15+10+40+35

+40-70-20+10+15-25-40-18

+10-10-20+20+10-35-25+10

6 Ad Served

User Segments

3 Bid Reques

Overview

Publishers

2 Ad Request

1 Page Request

4 Bid amp Ad

User Engagemen

Data Partners

Advertisers

Browser

Ad Exchange

Optimize

Models

Refresh learning

Data Store

Ads ampBudget

ModelScores

Events

Facebook likes

Searches on Google

Requests per day

Throughput

Blink of an eye

Time (ms)

Latency

raquoabc

Data Center Design

Rocket Fuel Scale

played Jeopardy

Congress

raquo 1400 servers

raquo 15K Disks

raquo 15K Cores

raquo 90 TB

raquo 30K MR slots

1 Year

41 PB8x

Growth

Hadoop Setup

QJM ZK Quorum

to ZK or JN

Standby NN

ZKFCZKFC

Active NN

Operations

Puppet+

Infradb

Automation is key

Puppet and Infradb

Isnrsquot it cool

Just define once

Performance Tuning

maxClientCnxns ZK

IMP MAPREDUCE-2026

ha-timeoutms

Performance Tuning

MAPREDUCE-5351

MAPREDUCE-5508

We Have an Issue

JT OOM

Operations

Monitoring

Wall of Ops

Monitoring

Network Monitoring

Alerting

Alerts

httphostnameportjmx

Modeling

Ad-hoc

Multi Tenancy

Scheduling

Operations

Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Cluster

Research

Ad-hoc Queries

Processed Data

JobTracker

YARN at Rocket FueI

THANKS

Delivery Methods

Overview

Overview (2)

Throughput

Latency

Data Center Design

Rocket Fuel Scale

Growth

Hadoop Setup

Operations

Puppet and Infradb

Performance Tuning

We Have an Issue

JT OOM

Operations (2)

Monitoring

Monitoring (2)

Network Monitoring

Alerting

Alerts

Multi Tenancy

Scheduling

Operations (3)

Data BCP Cluster

YARN at Rocket FueI

6 Ad Served

User Segments

3 Bid Reques

Overview

Publishers

2 Ad Request

1 Page Request

4 Bid amp Ad

User Engagemen

Data Partners

Advertisers

Browser

Ad Exchange

Optimize

Models

Refresh learning

Data Store

Ads ampBudget

ModelScores

Events

Facebook likes

Searches on Google

Requests per day

Throughput

Blink of an eye

Time (ms)

Latency

raquoabc

Data Center Design

Rocket Fuel Scale

played Jeopardy

Congress

raquo 1400 servers

raquo 15K Disks

raquo 15K Cores

raquo 90 TB

raquo 30K MR slots

1 Year

41 PB8x

Growth

Hadoop Setup

QJM ZK Quorum

to ZK or JN

Standby NN

ZKFCZKFC

Active NN

Operations

Puppet+

Infradb

Automation is key

Puppet and Infradb

Isnrsquot it cool

Just define once

Performance Tuning

maxClientCnxns ZK

IMP MAPREDUCE-2026

ha-timeoutms

Performance Tuning

MAPREDUCE-5351

MAPREDUCE-5508

We Have an Issue

JT OOM

Operations

Monitoring

Wall of Ops

Monitoring

Network Monitoring

Alerting

Alerts

httphostnameportjmx

Modeling

Ad-hoc

Multi Tenancy

Scheduling

Operations

Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Cluster

Research

Ad-hoc Queries

Processed Data

JobTracker

YARN at Rocket FueI

THANKS

Delivery Methods

Overview

Overview (2)

Throughput

Latency

Data Center Design

Rocket Fuel Scale

Growth

Hadoop Setup

Operations

Puppet and Infradb

Performance Tuning

We Have an Issue

JT OOM

Operations (2)

Monitoring

Monitoring (2)

Network Monitoring

Alerting

Alerts

Multi Tenancy

Scheduling

Operations (3)

Data BCP Cluster

YARN at Rocket FueI

Facebook likes

Searches on Google

Requests per day

Throughput

Blink of an eye

Time (ms)

Latency

raquoabc

Data Center Design

Rocket Fuel Scale

played Jeopardy

Congress

raquo 1400 servers

raquo 15K Disks

raquo 15K Cores

raquo 90 TB

raquo 30K MR slots

1 Year

41 PB8x

Growth

Hadoop Setup

QJM ZK Quorum

to ZK or JN

Standby NN

ZKFCZKFC

Active NN

Operations

Puppet+

Infradb

Automation is key

Puppet and Infradb

Isnrsquot it cool

Just define once

Performance Tuning

maxClientCnxns ZK

IMP MAPREDUCE-2026

ha-timeoutms

Performance Tuning

MAPREDUCE-5351

MAPREDUCE-5508

We Have an Issue

JT OOM

Operations

Monitoring

Wall of Ops

Monitoring

Network Monitoring

Alerting

Alerts

httphostnameportjmx

Modeling

Ad-hoc

Multi Tenancy

Scheduling

Operations

Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Cluster

Research

Ad-hoc Queries

Processed Data

JobTracker

YARN at Rocket FueI

THANKS

Delivery Methods

Overview

Overview (2)

Throughput

Latency

Data Center Design

Rocket Fuel Scale

Growth

Hadoop Setup

Operations

Puppet and Infradb

Performance Tuning

We Have an Issue

JT OOM

Operations (2)

Monitoring

Monitoring (2)

Network Monitoring

Alerting

Alerts

Multi Tenancy

Scheduling

Operations (3)

Data BCP Cluster

YARN at Rocket FueI

Blink of an eye

Time (ms)

Latency

raquoabc

Data Center Design

Rocket Fuel Scale

played Jeopardy

Congress

raquo 1400 servers

raquo 15K Disks

raquo 15K Cores

raquo 90 TB

raquo 30K MR slots

1 Year

41 PB8x

Growth

Hadoop Setup

QJM ZK Quorum

to ZK or JN

Standby NN

ZKFCZKFC

Active NN

Operations

Puppet+

Infradb

Automation is key

Puppet and Infradb

Isnrsquot it cool

Just define once

Performance Tuning

maxClientCnxns ZK

IMP MAPREDUCE-2026

ha-timeoutms

Performance Tuning

MAPREDUCE-5351

MAPREDUCE-5508

We Have an Issue

JT OOM

Operations

Monitoring

Wall of Ops

Monitoring

Network Monitoring

Alerting

Alerts

httphostnameportjmx

Modeling

Ad-hoc

Multi Tenancy

Scheduling

Operations

Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Cluster

Research

Ad-hoc Queries

Processed Data

JobTracker

YARN at Rocket FueI

THANKS

Delivery Methods

Overview

Overview (2)

Throughput

Latency

Data Center Design

Rocket Fuel Scale

Growth

Hadoop Setup

Operations

Puppet and Infradb

Performance Tuning

We Have an Issue

JT OOM

Operations (2)

Monitoring

Monitoring (2)

Network Monitoring

Alerting

Alerts

Multi Tenancy

Scheduling

Operations (3)

Data BCP Cluster

YARN at Rocket FueI

raquoabc

Data Center Design

Rocket Fuel Scale

played Jeopardy

Congress

raquo 1400 servers

raquo 15K Disks

raquo 15K Cores

raquo 90 TB

raquo 30K MR slots

1 Year

41 PB8x

Growth

Hadoop Setup

QJM ZK Quorum

to ZK or JN

Standby NN

ZKFCZKFC

Active NN

Operations

Puppet+

Infradb

Automation is key

Puppet and Infradb

Isnrsquot it cool

Just define once

Performance Tuning

maxClientCnxns ZK

IMP MAPREDUCE-2026

ha-timeoutms

Performance Tuning

MAPREDUCE-5351

MAPREDUCE-5508

We Have an Issue

JT OOM

Operations

Monitoring

Wall of Ops

Monitoring

Network Monitoring

Alerting

Alerts

httphostnameportjmx

Modeling

Ad-hoc

Multi Tenancy

Scheduling

Operations

Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Cluster

Research

Ad-hoc Queries

Processed Data

JobTracker

YARN at Rocket FueI

THANKS

Delivery Methods

Overview

Overview (2)

Throughput

Latency

Data Center Design

Rocket Fuel Scale

Growth

Hadoop Setup

Operations

Puppet and Infradb

Performance Tuning

We Have an Issue

JT OOM

Operations (2)

Monitoring

Monitoring (2)

Network Monitoring

Alerting

Alerts

Multi Tenancy

Scheduling

Operations (3)

Data BCP Cluster

YARN at Rocket FueI

raquoabc

Data Center Design

Rocket Fuel Scale

played Jeopardy

Congress

raquo 1400 servers

raquo 15K Disks

raquo 15K Cores

raquo 90 TB

raquo 30K MR slots

1 Year

41 PB8x

Growth

Hadoop Setup

QJM ZK Quorum

to ZK or JN

Standby NN

ZKFCZKFC

Active NN

Operations

Puppet+

Infradb

Automation is key

Puppet and Infradb

Isnrsquot it cool

Just define once

Performance Tuning

maxClientCnxns ZK

IMP MAPREDUCE-2026

ha-timeoutms

Performance Tuning

MAPREDUCE-5351

MAPREDUCE-5508

We Have an Issue

JT OOM

Operations

Monitoring

Wall of Ops

Monitoring

Network Monitoring

Alerting

Alerts

httphostnameportjmx

Modeling

Ad-hoc

Multi Tenancy

Scheduling

Operations

Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Cluster

Research

Ad-hoc Queries

Processed Data

JobTracker

YARN at Rocket FueI

THANKS

Delivery Methods

Overview

Overview (2)

Throughput

Latency

Data Center Design

Rocket Fuel Scale

Growth

Hadoop Setup

Operations

Puppet and Infradb

Performance Tuning

We Have an Issue

JT OOM

Operations (2)

Monitoring

Monitoring (2)

Network Monitoring

Alerting

Alerts

Multi Tenancy

Scheduling

Operations (3)

Data BCP Cluster

YARN at Rocket FueI

Data Center Design

Rocket Fuel Scale

played Jeopardy

Congress

raquo 1400 servers

raquo 15K Disks

raquo 15K Cores

raquo 90 TB

raquo 30K MR slots

1 Year

41 PB8x

Growth

Hadoop Setup

QJM ZK Quorum

to ZK or JN

Standby NN

ZKFCZKFC

Active NN

Operations

Puppet+

Infradb

Automation is key

Puppet and Infradb

Isnrsquot it cool

Just define once

Performance Tuning

maxClientCnxns ZK

IMP MAPREDUCE-2026

ha-timeoutms

Performance Tuning

MAPREDUCE-5351

MAPREDUCE-5508

We Have an Issue

JT OOM

Operations

Monitoring

Wall of Ops

Monitoring

Network Monitoring

Alerting

Alerts

httphostnameportjmx

Modeling

Ad-hoc

Multi Tenancy

Scheduling

Operations

Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Cluster

Research

Ad-hoc Queries

Processed Data

JobTracker

YARN at Rocket FueI

THANKS

Delivery Methods

Overview

Overview (2)

Throughput

Latency

Data Center Design

Rocket Fuel Scale

Growth

Hadoop Setup

Operations

Puppet and Infradb

Performance Tuning

We Have an Issue

JT OOM

Operations (2)

Monitoring

Monitoring (2)

Network Monitoring

Alerting

Alerts

Multi Tenancy

Scheduling

Operations (3)

Data BCP Cluster

YARN at Rocket FueI

Rocket Fuel Scale

played Jeopardy

Congress

raquo 1400 servers

raquo 15K Disks

raquo 15K Cores

raquo 90 TB

raquo 30K MR slots

1 Year

41 PB8x

Growth

Hadoop Setup

QJM ZK Quorum

to ZK or JN

Standby NN

ZKFCZKFC

Active NN

Operations

Puppet+

Infradb

Automation is key

Puppet and Infradb

Isnrsquot it cool

Just define once

Performance Tuning

maxClientCnxns ZK

IMP MAPREDUCE-2026

ha-timeoutms

Performance Tuning

MAPREDUCE-5351

MAPREDUCE-5508

We Have an Issue

JT OOM

Operations

Monitoring

Wall of Ops

Monitoring

Network Monitoring

Alerting

Alerts

httphostnameportjmx

Modeling

Ad-hoc

Multi Tenancy

Scheduling

Operations

Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Cluster

Research

Ad-hoc Queries

Processed Data

JobTracker

YARN at Rocket FueI

THANKS

Delivery Methods

Overview

Overview (2)

Throughput

Latency

Data Center Design

Rocket Fuel Scale

Growth

Hadoop Setup

Operations

Puppet and Infradb

Performance Tuning

We Have an Issue

JT OOM

Operations (2)

Monitoring

Monitoring (2)

Network Monitoring

Alerting

Alerts

Multi Tenancy

Scheduling

Operations (3)

Data BCP Cluster

YARN at Rocket FueI

raquo 1400 servers

raquo 15K Disks

raquo 15K Cores

raquo 90 TB

raquo 30K MR slots

1 Year

41 PB8x

Growth

Hadoop Setup

QJM ZK Quorum

to ZK or JN

Standby NN

ZKFCZKFC

Active NN

Operations

Puppet+

Infradb

Automation is key

Puppet and Infradb

Isnrsquot it cool

Just define once

Performance Tuning

maxClientCnxns ZK

IMP MAPREDUCE-2026

ha-timeoutms

Performance Tuning

MAPREDUCE-5351

MAPREDUCE-5508

We Have an Issue

JT OOM

Operations

Monitoring

Wall of Ops

Monitoring

Network Monitoring

Alerting

Alerts

httphostnameportjmx

Modeling

Ad-hoc

Multi Tenancy

Scheduling

Operations

Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Cluster

Research

Ad-hoc Queries

Processed Data

JobTracker

YARN at Rocket FueI

THANKS

Delivery Methods

Overview

Overview (2)

Throughput

Latency

Data Center Design

Rocket Fuel Scale

Growth

Hadoop Setup

Operations

Puppet and Infradb

Performance Tuning

We Have an Issue

JT OOM

Operations (2)

Monitoring

Monitoring (2)

Network Monitoring

Alerting

Alerts

Multi Tenancy

Scheduling

Operations (3)

Data BCP Cluster

YARN at Rocket FueI

1 Year

41 PB8x

Growth

Hadoop Setup

QJM ZK Quorum

to ZK or JN

Standby NN

ZKFCZKFC

Active NN

Operations

Puppet+

Infradb

Automation is key

Puppet and Infradb

Isnrsquot it cool

Just define once

Performance Tuning

maxClientCnxns ZK

IMP MAPREDUCE-2026

ha-timeoutms

Performance Tuning

MAPREDUCE-5351

MAPREDUCE-5508

We Have an Issue

JT OOM

Operations

Monitoring

Wall of Ops

Monitoring

Network Monitoring

Alerting

Alerts

httphostnameportjmx

Modeling

Ad-hoc

Multi Tenancy

Scheduling

Operations

Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Cluster

Research

Ad-hoc Queries

Processed Data

JobTracker

YARN at Rocket FueI

THANKS

Delivery Methods

Overview

Overview (2)

Throughput

Latency

Data Center Design

Rocket Fuel Scale

Growth

Hadoop Setup

Operations

Puppet and Infradb

Performance Tuning

We Have an Issue

JT OOM

Operations (2)

Monitoring

Monitoring (2)

Network Monitoring

Alerting

Alerts

Multi Tenancy

Scheduling

Operations (3)

Data BCP Cluster

YARN at Rocket FueI

Hadoop Setup

QJM ZK Quorum

to ZK or JN

Standby NN

ZKFCZKFC

Active NN

Operations

Puppet+

Infradb

Automation is key

Puppet and Infradb

Isnrsquot it cool

Just define once

Performance Tuning

maxClientCnxns ZK

IMP MAPREDUCE-2026

ha-timeoutms

Performance Tuning

MAPREDUCE-5351

MAPREDUCE-5508

We Have an Issue

JT OOM

Operations

Monitoring

Wall of Ops

Monitoring

Network Monitoring

Alerting

Alerts

httphostnameportjmx

Modeling

Ad-hoc

Multi Tenancy

Scheduling

Operations

Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Cluster

Research

Ad-hoc Queries

Processed Data

JobTracker

YARN at Rocket FueI

THANKS

Delivery Methods

Overview

Overview (2)

Throughput

Latency

Data Center Design

Rocket Fuel Scale

Growth

Hadoop Setup

Operations

Puppet and Infradb

Performance Tuning

We Have an Issue

JT OOM

Operations (2)

Monitoring

Monitoring (2)

Network Monitoring

Alerting

Alerts

Multi Tenancy

Scheduling

Operations (3)

Data BCP Cluster

YARN at Rocket FueI

Hadoop Setup

QJM ZK Quorum

to ZK or JN

Standby NN

ZKFCZKFC

Active NN

Operations

Puppet+

Infradb

Automation is key

Puppet and Infradb

Isnrsquot it cool

Just define once

Performance Tuning

maxClientCnxns ZK

IMP MAPREDUCE-2026

ha-timeoutms

Performance Tuning

MAPREDUCE-5351

MAPREDUCE-5508

We Have an Issue

JT OOM

Operations

Monitoring

Wall of Ops

Monitoring

Network Monitoring

Alerting

Alerts

httphostnameportjmx

Modeling

Ad-hoc

Multi Tenancy

Scheduling

Operations

Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Cluster

Research

Ad-hoc Queries

Processed Data

JobTracker

YARN at Rocket FueI

THANKS

Delivery Methods

Overview

Overview (2)

Throughput

Latency

Data Center Design

Rocket Fuel Scale

Growth

Hadoop Setup

Operations

Puppet and Infradb

Performance Tuning

We Have an Issue

JT OOM

Operations (2)

Monitoring

Monitoring (2)

Network Monitoring

Alerting

Alerts

Multi Tenancy

Scheduling

Operations (3)

Data BCP Cluster

YARN at Rocket FueI

Operations

Puppet+

Infradb

Automation is key

Puppet and Infradb

Isnrsquot it cool

Just define once

Performance Tuning

maxClientCnxns ZK

IMP MAPREDUCE-2026

ha-timeoutms

Performance Tuning

MAPREDUCE-5351

MAPREDUCE-5508

We Have an Issue

JT OOM

Operations

Monitoring

Wall of Ops

Monitoring

Network Monitoring

Alerting

Alerts

httphostnameportjmx

Modeling

Ad-hoc

Multi Tenancy

Scheduling

Operations

Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Cluster

Research

Ad-hoc Queries

Processed Data

JobTracker

YARN at Rocket FueI

THANKS

Delivery Methods

Overview

Overview (2)

Throughput

Latency

Data Center Design

Rocket Fuel Scale

Growth

Hadoop Setup

Operations

Puppet and Infradb

Performance Tuning

We Have an Issue

JT OOM

Operations (2)

Monitoring

Monitoring (2)

Network Monitoring

Alerting

Alerts

Multi Tenancy

Scheduling

Operations (3)

Data BCP Cluster

YARN at Rocket FueI

Puppet+

Infradb

Automation is key

Puppet and Infradb

Isnrsquot it cool

Just define once

Performance Tuning

maxClientCnxns ZK

IMP MAPREDUCE-2026

ha-timeoutms

Performance Tuning

MAPREDUCE-5351

MAPREDUCE-5508

We Have an Issue

JT OOM

Operations

Monitoring

Wall of Ops

Monitoring

Network Monitoring

Alerting

Alerts

httphostnameportjmx

Modeling

Ad-hoc

Multi Tenancy

Scheduling

Operations

Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Cluster

Research

Ad-hoc Queries

Processed Data

JobTracker

YARN at Rocket FueI

THANKS

Delivery Methods

Overview

Overview (2)

Throughput

Latency

Data Center Design

Rocket Fuel Scale

Growth

Hadoop Setup

Operations

Puppet and Infradb

Performance Tuning

We Have an Issue

JT OOM

Operations (2)

Monitoring

Monitoring (2)

Network Monitoring

Alerting

Alerts

Multi Tenancy

Scheduling

Operations (3)

Data BCP Cluster

YARN at Rocket FueI

Puppet and Infradb

Isnrsquot it cool

Just define once

Performance Tuning

maxClientCnxns ZK

IMP MAPREDUCE-2026

ha-timeoutms

Performance Tuning

MAPREDUCE-5351

MAPREDUCE-5508

We Have an Issue

JT OOM

Operations

Monitoring

Wall of Ops

Monitoring

Network Monitoring

Alerting

Alerts

httphostnameportjmx

Modeling

Ad-hoc

Multi Tenancy

Scheduling

Operations

Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Cluster

Research

Ad-hoc Queries

Processed Data

JobTracker

YARN at Rocket FueI

THANKS

Delivery Methods

Overview

Overview (2)

Throughput

Latency

Data Center Design

Rocket Fuel Scale

Growth

Hadoop Setup

Operations

Puppet and Infradb

Performance Tuning

We Have an Issue

JT OOM

Operations (2)

Monitoring

Monitoring (2)

Network Monitoring

Alerting

Alerts

Multi Tenancy

Scheduling

Operations (3)

Data BCP Cluster

YARN at Rocket FueI

Performance Tuning

maxClientCnxns ZK

IMP MAPREDUCE-2026

ha-timeoutms

Performance Tuning

MAPREDUCE-5351

MAPREDUCE-5508

We Have an Issue

JT OOM

Operations

Monitoring

Wall of Ops

Monitoring

Network Monitoring

Alerting

Alerts

httphostnameportjmx

Modeling

Ad-hoc

Multi Tenancy

Scheduling

Operations

Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Cluster

Research

Ad-hoc Queries

Processed Data

JobTracker

YARN at Rocket FueI

THANKS

Delivery Methods

Overview

Overview (2)

Throughput

Latency

Data Center Design

Rocket Fuel Scale

Growth

Hadoop Setup

Operations

Puppet and Infradb

Performance Tuning

We Have an Issue

JT OOM

Operations (2)

Monitoring

Monitoring (2)

Network Monitoring

Alerting

Alerts

Multi Tenancy

Scheduling

Operations (3)

Data BCP Cluster

YARN at Rocket FueI

maxClientCnxns ZK

IMP MAPREDUCE-2026

ha-timeoutms

Performance Tuning

MAPREDUCE-5351

MAPREDUCE-5508

We Have an Issue

JT OOM

Operations

Monitoring

Wall of Ops

Monitoring

Network Monitoring

Alerting

Alerts

httphostnameportjmx

Modeling

Ad-hoc

Multi Tenancy

Scheduling

Operations

Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Cluster

Research

Ad-hoc Queries

Processed Data

JobTracker

YARN at Rocket FueI

THANKS

Delivery Methods

Overview

Overview (2)

Throughput

Latency

Data Center Design

Rocket Fuel Scale

Growth

Hadoop Setup

Operations

Puppet and Infradb

Performance Tuning

We Have an Issue

JT OOM

Operations (2)

Monitoring

Monitoring (2)

Network Monitoring

Alerting

Alerts

Multi Tenancy

Scheduling

Operations (3)

Data BCP Cluster

YARN at Rocket FueI

MAPREDUCE-5351

MAPREDUCE-5508

We Have an Issue

JT OOM

Operations

Monitoring

Wall of Ops

Monitoring

Network Monitoring

Alerting

Alerts

httphostnameportjmx

Modeling

Ad-hoc

Multi Tenancy

Scheduling

Operations

Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Cluster

Research

Ad-hoc Queries

Processed Data

JobTracker

YARN at Rocket FueI

THANKS

Delivery Methods

Overview

Overview (2)

Throughput

Latency

Data Center Design

Rocket Fuel Scale

Growth

Hadoop Setup

Operations

Puppet and Infradb

Performance Tuning

We Have an Issue

JT OOM

Operations (2)

Monitoring

Monitoring (2)

Network Monitoring

Alerting

Alerts

Multi Tenancy

Scheduling

Operations (3)

Data BCP Cluster

YARN at Rocket FueI

JT OOM

Operations

Monitoring

Wall of Ops

Monitoring

Network Monitoring

Alerting

Alerts

httphostnameportjmx

Modeling

Ad-hoc

Multi Tenancy

Scheduling

Operations

Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Cluster

Research

Ad-hoc Queries

Processed Data

JobTracker

YARN at Rocket FueI

THANKS

Delivery Methods

Overview

Overview (2)

Throughput

Latency

Data Center Design

Rocket Fuel Scale

Growth

Hadoop Setup

Operations

Puppet and Infradb

Performance Tuning

We Have an Issue

JT OOM

Operations (2)

Monitoring

Monitoring (2)

Network Monitoring

Alerting

Alerts

Multi Tenancy

Scheduling

Operations (3)

Data BCP Cluster

YARN at Rocket FueI

Operations

Monitoring

Wall of Ops

Monitoring

Network Monitoring

Alerting

Alerts

httphostnameportjmx

Modeling

Ad-hoc

Multi Tenancy

Scheduling

Operations

Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Cluster

Research

Ad-hoc Queries

Processed Data

JobTracker

YARN at Rocket FueI

THANKS

Delivery Methods

Overview

Overview (2)

Throughput

Latency

Data Center Design

Rocket Fuel Scale

Growth

Hadoop Setup

Operations

Puppet and Infradb

Performance Tuning

We Have an Issue

JT OOM

Operations (2)

Monitoring

Monitoring (2)

Network Monitoring

Alerting

Alerts

Multi Tenancy

Scheduling

Operations (3)

Data BCP Cluster

YARN at Rocket FueI

Monitoring

Wall of Ops

Monitoring

Network Monitoring

Alerting

Alerts

httphostnameportjmx

Modeling

Ad-hoc

Multi Tenancy

Scheduling

Operations

Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Cluster

Research

Ad-hoc Queries

Processed Data

JobTracker

YARN at Rocket FueI

THANKS

Delivery Methods

Overview

Overview (2)

Throughput

Latency

Data Center Design

Rocket Fuel Scale

Growth

Hadoop Setup

Operations

Puppet and Infradb

Performance Tuning

We Have an Issue

JT OOM

Operations (2)

Monitoring

Monitoring (2)

Network Monitoring

Alerting

Alerts

Multi Tenancy

Scheduling

Operations (3)

Data BCP Cluster

YARN at Rocket FueI

Monitoring

Network Monitoring

Alerting

Alerts

httphostnameportjmx

Modeling

Ad-hoc

Multi Tenancy

Scheduling

Operations

Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Cluster

Research

Ad-hoc Queries

Processed Data

JobTracker

YARN at Rocket FueI

THANKS

Delivery Methods

Overview

Overview (2)

Throughput

Latency

Data Center Design

Rocket Fuel Scale

Growth

Hadoop Setup

Operations

Puppet and Infradb

Performance Tuning

We Have an Issue

JT OOM

Operations (2)

Monitoring

Monitoring (2)

Network Monitoring

Alerting

Alerts

Multi Tenancy

Scheduling

Operations (3)

Data BCP Cluster

YARN at Rocket FueI

Network Monitoring

Alerting

Alerts

httphostnameportjmx

Modeling

Ad-hoc

Multi Tenancy

Scheduling

Operations

Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Cluster

Research

Ad-hoc Queries

Processed Data

JobTracker

YARN at Rocket FueI

THANKS

Delivery Methods

Overview

Overview (2)

Throughput

Latency

Data Center Design

Rocket Fuel Scale

Growth

Hadoop Setup

Operations

Puppet and Infradb

Performance Tuning

We Have an Issue

JT OOM

Operations (2)

Monitoring

Monitoring (2)

Network Monitoring

Alerting

Alerts

Multi Tenancy

Scheduling

Operations (3)

Data BCP Cluster

YARN at Rocket FueI

Network Monitoring

Alerting

Alerts

httphostnameportjmx

Modeling

Ad-hoc

Multi Tenancy

Scheduling

Operations

Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Cluster

Research

Ad-hoc Queries

Processed Data

JobTracker

YARN at Rocket FueI

THANKS

Delivery Methods

Overview

Overview (2)

Throughput

Latency

Data Center Design

Rocket Fuel Scale

Growth

Hadoop Setup

Operations

Puppet and Infradb

Performance Tuning

We Have an Issue

JT OOM

Operations (2)

Monitoring

Monitoring (2)

Network Monitoring

Alerting

Alerts

Multi Tenancy

Scheduling

Operations (3)

Data BCP Cluster

YARN at Rocket FueI

Alerting

Alerts

httphostnameportjmx

Modeling

Ad-hoc

Multi Tenancy

Scheduling

Operations

Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Cluster

Research

Ad-hoc Queries

Processed Data

JobTracker

YARN at Rocket FueI

THANKS

Delivery Methods

Overview

Overview (2)

Throughput

Latency

Data Center Design

Rocket Fuel Scale

Growth

Hadoop Setup

Operations

Puppet and Infradb

Performance Tuning

We Have an Issue

JT OOM

Operations (2)

Monitoring

Monitoring (2)

Network Monitoring

Alerting

Alerts

Multi Tenancy

Scheduling

Operations (3)

Data BCP Cluster

YARN at Rocket FueI

Alerts

httphostnameportjmx

Modeling

Ad-hoc

Multi Tenancy

Scheduling

Operations

Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Cluster

Research

Ad-hoc Queries

Processed Data

JobTracker

YARN at Rocket FueI

THANKS

Delivery Methods

Overview

Overview (2)

Throughput

Latency

Data Center Design

Rocket Fuel Scale

Growth

Hadoop Setup

Operations

Puppet and Infradb

Performance Tuning

We Have an Issue

JT OOM

Operations (2)

Monitoring

Monitoring (2)

Network Monitoring

Alerting

Alerts

Multi Tenancy

Scheduling

Operations (3)

Data BCP Cluster

YARN at Rocket FueI

Modeling

Ad-hoc

Multi Tenancy

Scheduling

Operations

Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Cluster

Research

Ad-hoc Queries

Processed Data

JobTracker

YARN at Rocket FueI

THANKS

Delivery Methods

Overview

Overview (2)

Throughput

Latency

Data Center Design

Rocket Fuel Scale

Growth

Hadoop Setup

Operations

Puppet and Infradb

Performance Tuning

We Have an Issue

JT OOM

Operations (2)

Monitoring

Monitoring (2)

Network Monitoring

Alerting

Alerts

Multi Tenancy

Scheduling

Operations (3)

Data BCP Cluster

YARN at Rocket FueI

Modeling

Ad-hoc

Multi Tenancy

Scheduling

Operations

Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Cluster

Research

Ad-hoc Queries

Processed Data

JobTracker

YARN at Rocket FueI

THANKS

Delivery Methods

Overview

Overview (2)

Throughput

Latency

Data Center Design

Rocket Fuel Scale

Growth

Hadoop Setup

Operations

Puppet and Infradb

Performance Tuning

We Have an Issue

JT OOM

Operations (2)

Monitoring

Monitoring (2)

Network Monitoring

Alerting

Alerts

Multi Tenancy

Scheduling

Operations (3)

Data BCP Cluster

YARN at Rocket FueI

Scheduling

Operations

Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Cluster

Research

Ad-hoc Queries

Processed Data

JobTracker

YARN at Rocket FueI

THANKS

Delivery Methods

Overview

Overview (2)

Throughput

Latency

Data Center Design

Rocket Fuel Scale

Growth

Hadoop Setup

Operations

Puppet and Infradb

Performance Tuning

We Have an Issue

JT OOM

Operations (2)

Monitoring

Monitoring (2)

Network Monitoring

Alerting

Alerts

Multi Tenancy

Scheduling

Operations (3)

Data BCP Cluster

YARN at Rocket FueI

Operations

Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Cluster

Research

Ad-hoc Queries

Processed Data

JobTracker

YARN at Rocket FueI

THANKS

Delivery Methods

Overview

Overview (2)

Throughput

Latency

Data Center Design

Rocket Fuel Scale

Growth

Hadoop Setup

Operations

Puppet and Infradb

Performance Tuning

We Have an Issue

JT OOM

Operations (2)

Monitoring

Monitoring (2)

Network Monitoring

Alerting

Alerts

Multi Tenancy

Scheduling

Operations (3)

Data BCP Cluster

YARN at Rocket FueI

Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Cluster

Research

Ad-hoc Queries

Processed Data

JobTracker

YARN at Rocket FueI

THANKS

Delivery Methods

Overview

Overview (2)

Throughput

Latency

Data Center Design

Rocket Fuel Scale

Growth

Hadoop Setup

Operations

Puppet and Infradb

Performance Tuning

We Have an Issue

JT OOM

Operations (2)

Monitoring

Monitoring (2)

Network Monitoring

Alerting

Alerts

Multi Tenancy

Scheduling

Operations (3)

Data BCP Cluster

YARN at Rocket FueI

Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Cluster

Research

Ad-hoc Queries

Processed Data

JobTracker

YARN at Rocket FueI

THANKS

Delivery Methods

Overview

Overview (2)

Throughput

Latency

Data Center Design

Rocket Fuel Scale

Growth

Hadoop Setup

Operations

Puppet and Infradb

Performance Tuning

We Have an Issue

JT OOM

Operations (2)

Monitoring

Monitoring (2)

Network Monitoring

Alerting

Alerts

Multi Tenancy

Scheduling

Operations (3)

Data BCP Cluster

YARN at Rocket FueI

JobTracker

YARN at Rocket FueI

THANKS

Delivery Methods

Overview

Overview (2)

Throughput

Latency

Data Center Design

Rocket Fuel Scale

Growth

Hadoop Setup

Operations

Puppet and Infradb

Performance Tuning

We Have an Issue

JT OOM

Operations (2)

Monitoring

Monitoring (2)

Network Monitoring

Alerting

Alerts

Multi Tenancy

Scheduling

Operations (3)

Data BCP Cluster

YARN at Rocket FueI

THANKS

Delivery Methods

Overview

Overview (2)

Throughput

Latency

Data Center Design

Rocket Fuel Scale

Growth

Hadoop Setup

Operations

Puppet and Infradb

Performance Tuning

We Have an Issue

JT OOM

Operations (2)

Monitoring

Monitoring (2)

Network Monitoring

Alerting

Alerts

Multi Tenancy

Scheduling

Operations (3)

Data BCP Cluster

YARN at Rocket FueI

THANKS

Delivery Methods

Overview

Overview (2)

Throughput

Latency

Data Center Design

Rocket Fuel Scale

Growth

Hadoop Setup

Operations

Puppet and Infradb

Performance Tuning

We Have an Issue

JT OOM

Operations (2)

Monitoring

Monitoring (2)

Network Monitoring

Alerting

Alerts

Multi Tenancy

Scheduling

Operations (3)

Data BCP Cluster

YARN at Rocket FueI

THANKS

Delivery Methods

Overview

Overview (2)

Throughput

Latency

Data Center Design

Rocket Fuel Scale

Growth

Hadoop Setup

Operations

Puppet and Infradb

Performance Tuning

We Have an Issue

JT OOM

Operations (2)

Monitoring

Monitoring (2)

Network Monitoring

Alerting

Alerts

Multi Tenancy

Scheduling

Operations (3)

Data BCP Cluster

YARN at Rocket FueI

Engineering

Hado"ops" or Had"oops"