Apache Hadoop YARN: Present and Future

Vinod Kumar VavilapalliHortonworks

Apache Hadoop YARNPresent and Future

Vinod Kumar Vavilapalli

vinodkv [at] apache.org

@tshooter

A quick show of hands..

• Hadoop 2

Architecting the Future of Big Data

Real life Hadoop Logo

Who am I?

• 6.75 Hadoop-years old• Last thing at School – a two node Tomcat cluster. Three months later,

first thing at job, brought down a 800 node cluster ;)• Previously @Yahoo!• Now @Hortonworks• Two hats

– Hortonworks: Hadoop MapReduce and YARN Development lead– Apache: Apache Hadoop YARN lead. Apache Hadoop PMC, Apache Member

• Worked/working on– YARN, Hadoop MapReduce, HadoopOnDemand, CapacityScheduler, Hadoop

security– Apache Ambari: Kickstarted the project and its first release– Stinger: High performance data processing with Hadoop/Hive

• Lots of trouble shooting on clusters• 99% + code in Apache, Hadoop

Agenda

• Apache Hadoop 2 : Overview• Past• Present• Future

Apache Hadoop 2Next Generation Architecture

Architecting the Future of Big DataPage 6

What is YARN?

• Resource Management Platform– MapReduce v2– Beyond MapReduce with Tez, Storm, Spark; in Hadoop!– Did I mention Services like HBase, Accumulo on YARN with HoYA/Slider?

• How is it different from Hadoop 1? ..

Hadoop 1 vs Hadoop 2

HADOOP 1.0

HDFS(redundant, reliable storage)

MapReduce(cluster resource management

& data processing)

HDFS2(redundant, highly-available & reliable storage)

YARN(cluster resource management)

MapReduce(data processing)

Others

HADOOP 2.0

Single Use SystemBatch Apps

Multi Purpose PlatformBatch, Interactive, Online, Streaming, …

Key Benefits of YARN

• Scale

• New Programming Models & Services

• Improved cluster utilization

• Agility

• To infinity and beyond ..

Why Migrate?

• 2.0 >= 2 * 1.0– HDFS: Lots of ground-breaking features– YARN: Next generation architecture

• Return on Investment: 2x throughput on same hardware!• Ready for improvements in hardware• Not convinced? Let’s see what others are saying!

Yahoo!

• Leader/Visionary on all things Hadoop!• On YARN (0.23.x)• Moving fast to 2.x

http://developer.yahoo.com/blogs/ydn/hadoop-yahoo-more-ever-54421.html

Twitter

• Has one of the largest Hadoop clusters in the industry with many petabytes of data

• Migrated production clusters to Hadoop-2• Go to Mayank’s talk

– “Hadoop-2 @ ebay”!– Thursday, April 3– Track : Deployment and Operations

• Should be convinced by now .. . No?

YARN: the Data Operating System

Present

Apache Hadoop releases

• 15 October, 2013• The 1st GA release of Apache Hadoop 2.x• YARN

– First stable and supported release of YARN– Binary Compatibility for MapReduce applications built on hadoop-1.x– YARN level APIs solidified for the future– Performance– Scale!

• HDFS– High Availability for HDFS– HDFS Federation– HDFS Snapshots– NFSv3 access to data in HDFS

• Support for running Hadoop on Microsoft Windows• Substantial amount of integration testing with rest of projects in the

ecosystem

Apache Hadoop 2.2

Apache Hadoop releases (contd)

• 24 February, 2014• First post GA release for the year 2014

• Alpha features in YARN– ResourceManager HA– Application History– Will cover in the 2.4 content

• HDFS– Details follow..

• Number of bug-fixes, enhancements

Apache Hadoop 2.3

HDFS: Heterogeneous Storage

HDFS: DataNode caching

Apache Hadoop releases (contd)

• Very soon!

• YARN– Details follow..– ResourceManager restart fail-over for high availability– Preemption– Application History and timeline

• HDFS– FileSystem ACLs– Rolling upgrades

Apache Hadoop 2.4

ResourceManager Restart and fail-over

ZooKeeper

Capacity Scheduler Preemption

Application History and Timeline

• Few MR specific implementations: History and web-UI• Not just MR anymore!• History

– MapReduce specific Job History Server– Beyond ResourceManager Restart

• Timeline– Framework specific event collection and UIs

• Run analytics on historical apps!

Future

Future: Operational enhancements

• Rolling upgrades– No/minimal impact to users– Ideal: Always rolling!

• HDFS in• YARN

Future: Enabling more apps

• Beyond MR• Discussing next

– Long running services– Isolation– Multi-dimensional resource

scheduling

Future: Long running services

• You can run them already!• Few enhancements needed

– Logs– Security– Management/monitoring

• Resource sharing across workload types

• Project Slider

Fine-grain isolation for multi-tenancy

• Custom memory-monitoring• Cgroups• Linux Containers• VMs

Multi-resource scheduling

• Today – memory & cpu– Physical memory / virtual memory– Cpu Cores – Virtual cores

• CPU stuff: More bake in• Disks

– Space– IOPS

• Network

Other features

• Application SLAs• Node labels• Node affinity/anti-affinity• Better online queue-management

YARN EcosystemBeyond the core YARN project: Briefly

Eco-system

Applications Powered by YARN

Apache Giraph – Graph Processing

Apache Hama – BSP

Apache Hadoop MapReduce – Batch

Apache Tez – Batch/Interactive

Apache S4 – Stream Processing

Apache Samza – Stream Processing

Apache Storm – Stream Processing

Apache Spark – Iterative applications

HOYA – HBase on YARNYARN FrameworksApache Twill

REEF by Microsoft

Spring support for Hadoop 2

There's an app for that...

YARN App Marketplace!

Apache TEZ

• Moving beyond MR• A data processing framework that can execute a complex DAG

of tasks.

• “Apache Tez - A New Chapter in Hadoop Data Processing”– By Siddharth Seth: YARN & Tez Committer/PMC Member– Thursday, April 3 (4:20-5:00pm)

• Apache Hadoop 2 is, at least, twice as good!

• Exciting journey with Hadoop for this decade…– Hadoop is no longer a one-trick pony, err elephant– Beyond just HDFS & MapReduce

• Architecture for the future– Centralized data– Exciting spectrum of application types, workloads and usecases

Couple more things..

The Book is out!

http://yarn-book.com/

Thank you!

Download Sandbox: Experience Apache Hadoop

Both 2.x and 1.x Versions Available!

http://hortonworks.com/products/hortonworks-sandbox/

Questions Time!

Apache Hadoop YARN: Present and Future

Technology

Hadoop Summit San Jose 2014 - Apache Hadoop YARN: Best Practices

Apache Yarn - Hadoop Cluster Management

Hadoop 2.6 Conﬁguration and More Examplestorlone.dia.uniroma3.it/bigdata/E1-AWS.pdfApache Hadoop & YARN Apache Hadoop (1.X) De facto Big Data open source platform Running for about

Review The practice of Hadoop and Spark to Analyze Big ... · Apache Hadoop), and YARN. Hadoop common, Apache Foundation, has a number of pre-defined utilities and libraries that

Apache Hadoop YARN - pearsoncmg.com · Contents Foreword by Raymie Stata xiii Foreword by Paul Dix xv Preface xvii Acknowledgments xxi About the Authors xxv 1 Apache Hadoop YARN:

An example Apache Hadoop Yarn upgrade

YARN: Future of Data Processing with Apache Hadoop

PaaSon$Hadoop$YARN$€¦ · PaaSon$Hadoop$YARN$! IdeaandPrototype! ABSTRACT$ This!document!describes!aprototype!implementationof!asimple!PAAS built!on!the!Hadoop!YARN!framework.!

Apache Hadoop YARN: Yet Another Resource Negotiatorweiwa/teaching/Fall15-COMP6611B/reading... · Apache Hadoop YARN: Yet Another Resource Negotiator ... running YARN on production

Apache hadoop yarn 勉強会 8. capacity scheduler in yarn

Apache Spark Performance Troubleshooting at Scale ......Apache Spark @ • Spark is a popular component for data processing – Deployed on four production Hadoop/YARN clusters •

Apache Hadoop YARN: Past, Present and Future

Hadoop Summit Europe Talk 2014: Apache Hadoop YARN: Present and Future

Apache Hadoop YARN Enabling next generation data applications

The future of Apache Hadoop YARN

Discover HDP 2.1: Apache Hadoop 2.4.0, YARN & HDFS

Hadoop YARN overview

Dimensions Computation With Apache Apex - The Linux …events.linuxfoundation.org/sites/events/files/slides/Apex... · Dimensions Computation With Apache Apex. ... Uses Hadoop YARN

Introduction to Hadoop 2.0 & YARN | Hadoop 2.0 & YARN Fundamentals | Hadoop 2.0 & YARN Architecture

Apache Hadoop YARN Reading - Chapter7: NodeManager