Topic 7: Shortcomings in the MapReduce Paradigm

7: Shortcomings in the MapReduce Paradigm

Zubair Nabi

[email protected]

April 19, 2013

Zubair Nabi 7: Shortcomings in the MapReduce Paradigm April 19, 2013 1 / 31

Outline

1 Hadoop everywhere!

2 Skew

3 Heterogeneous Environment

4 Low-level Programming Interface

5 Strictly Batch-processing

6 Single-input/single output and Two-phase

7 Iterative and Recursive Applications

8 Incremental Computation


Outline


2 Skew








Users1

Adobe: Several areas from social services to unstructured data storageand processing

eBay: 532 nodes cluster storing 5.3PB of data

Facebook: Used for reporting/analytics; one cluster with 1100 nodes(12PB) and another with 300 nodes (3PB)

LinkedIn: 3 clusters with collectively 4000 nodes

Twitter: To store and process Tweets and log files

Yahoo!: Multiple clusters with collectively 40000 nodes; largest clusterhas 4500 nodes!

1http://wiki.apache.org/hadoop/PoweredByZubair Nabi 7: Shortcomings in the MapReduce Paradigm April 19, 2013 4 / 31

http://wiki.apache.org/hadoop/PoweredBy

Users1









Users1









Users1









Users1









Users1









But all is not well

Over the years, Hadoop has become a one-size-fits-all solution to dataintensive computing

As early as 2008, David DeWitt and Michael Stonebraker asserted thatMapReduce was a “major step backwards” for data intensivecomputingThey opined:

I MapReduce is a major step backwards in database access because itnegates schema and is too low-level

I It has a sub-optimal implementation as it, makes use of brute forceinstead of indexing, does not handle skew, and uses data pull instead ofpush

I It is just rehashing old database conceptsI It is missing most DBMS functionalities, such as updates, transactions,

etc.I It is incompatible with DBMS tools, such as human visualization, data

replication from one DBMS to another, etc.


But all is not well


As early as 2008, David DeWitt and Michael Stonebraker asserted thatMapReduce was a “major step backwards” for data intensivecomputing

They opined:I MapReduce is a major step backwards in database access because it

negates schema and is too low-levelI It has a sub-optimal implementation as it, makes use of brute force

instead of indexing, does not handle skew, and uses data pull instead ofpush





But all is not well









But all is not well









But all is not well





I It is just rehashing old database concepts

I It is missing most DBMS functionalities, such as updates, transactions,etc.

I It is incompatible with DBMS tools, such as human visualization, datareplication from one DBMS to another, etc.


But all is not well






etc.

I It is incompatible with DBMS tools, such as human visualization, datareplication from one DBMS to another, etc.


But all is not well









Outline


2 Skew








Introduction

Due to the uneven distribution of intermediate key/value pairs somereduce workers end up doing more work

Such reducers become “stragglers”

A large number of real-world applications follow long-tailed distributions(Zipf-like)


Introduction





Introduction





Wordcount and skew

Text corpora have a Zipfian skew, i.e. a very small number of wordsaccount for most occurrences

For instance, of 242,758 words in the dataset used to generate thefigure, the 10, 100, and 1000 most frequent words account for 22%,43%, and 64% of the entire set

Such skewed intermediate results lead to uneven distribution ofworkload across reduce workers


Wordcount and skew





Wordcount and skew





Page rank and skew

Even Google’s implementation of its core PageRank algorithm isplagued by the skew problem

Google uses PageRank to calculate a webpage’s relevance for a givensearch query

I Map: Emit the outlinks for each pageI Reduce: Calculate rank per page

The skew in intermediate data exists due to the huge disparity in thenumber of incoming links across pages on the Internet

The scale of the problem is evident when we consider the fact thatGoogle currently indexes more than 25 billion webpages with skewedlinks

For instance, Facebook has 49,376,609 incoming links (at the time ofwriting) while the personal webpage of the presenter only has 4 (=))


Page rank and skew

Even Google’s implementation of its core PageRank algorithm isplagued by the skew problemGoogle uses PageRank to calculate a webpage’s relevance for a givensearch query






Page rank and skew


I Map: Emit the outlinks for each page

I Reduce: Calculate rank per page





Page rank and skew







Page rank and skew







Page rank and skew







Page rank and skew







Zipf distributions are everywhere

Followed by Inverted Indexing, Publish/Subscribe systems, frauddetection, and various clustering algorithms

P2P systems have Zipf distributions too both in terms of users andcontent

Web caching schemes as well as email and social networks












Outline


2 Skew








Introduction

In the MapReduce model, tasks which take exceptionally long arelabelled “stragglers”

The framework launches a speculative copy of each straggler onanother machine expecting it to finish quickly

Without this, the overall job completion time is dictated by the sloweststraggler

On Google clusters, speculative execution can reduce job completionby 44%


Introduction






Introduction






Introduction






Hadoop’s assumptions regarding speculation

1 All nodes are equal, i.e. they can perform work at more or less thesame rate

2 Tasks make progress at a constant rate throughout their lifetime

3 There is no cost of launching a speculative cost on an otherwise idleslot/node

4 The progress score of a task captures the fraction of its total work thatit has done. Specifically, the shuffle, merge, and reduce logic phaseseach take roughly 1/3 of the total time

5 As tasks finish in waves, a task with a low progress score is most likelya straggler

6 Tasks within the same phase, require roughly the same amount of work










































Assumptions 1 and 2



Both breakdown in heterogeneous environments which consist ofmultiple generations of hardware


Assumptions 1 and 2





Assumptions 1 and 2





Assumption 3


Breaks down due to shared resources


Assumption 3


Breaks down due to shared resources


Assumption 4


Breaks down due the fact that in reduce tasks the shuffle phase takesthe longest time as opposed to the other 2


Assumption 4


Breaks down due the fact that in reduce tasks the shuffle phase takesthe longest time as opposed to the other 2


Assumption 5


Breaks down due to the fact that task completion is spread across timedue to uneven workload


Assumption 5


Breaks down due to the fact that task completion is spread across timedue to uneven workload


Assumption 6


Breaks down due to data skew


Assumption 6


Breaks down due to data skew


Outline


2 Skew








Introduction

The one-input, two-stage data flow is extremely rigid for ad-hocanalysis of large datasets

Hacks need to be put into place for different data flow, such as joins ormultiple stages

Custom code has to be written for common DB operations, such asprojection and filtering

The opaque nature of map and reduce functions makes it impossible toperform optimizations, such as operator reordering


Introduction






Introduction






Introduction






Outline


2 Skew








Introduction

In case of MapReduce, the entire output of a map or a reduce taskneeds to be materialized to local storage before the next stage cancommence

Simplifies fault-tolerance

Reducers have to pull their input instead of the mappers pushing it

Negates pipelining, result estimation, and continuous queries (streamprocessing)


Introduction






Introduction






Introduction






Outline


2 Skew








Introduction

1 Not all applications can be broken down into just two-phases, such ascomplex SQL-like queries

2 Tasks take in just one input and produce one output


Introduction

1 Not all applications can be broken down into just two-phases, such ascomplex SQL-like queries

2 Tasks take in just one input and produce one output


Outline


2 Skew








Introduction

1 Hadoop is widely employed for iterative computations

2 For machine learning applications, the Apache Mahout library is usedatop Hadoop

3 Mahout uses an external driver program to submit multiple jobs toHadoop and perform a convergence test

4 No fault-tolerance and overhead of job submission

5 Loop-invariant data is materialized to storage


Introduction







Introduction







Introduction







Introduction







Outline


2 Skew








Introduction

1 Most workloads processed by MapReduce are incremental by nature,i.e. MapReduce jobs often run repeatedly with small changes in theirinput

2 For instance, most iterations of PageRank run with very smallmodifications

3 Unfortunately, even with a small change in input, MapReducere-performs the entire computation


Introduction





Introduction





References

1 MapReduce: A major step backwards:http://homes.cs.washington.edu/~billhowe/mapreduce_a_major_step_backwards.html

2 Matei Zaharia, Andy Konwinski, Anthony D. Joseph, Randy Katz, andIon Stoica. 2008. Improving MapReduce performance inheterogeneous environments. In Proceedings of the 8th USENIXconference on Operating systems design and implementation(OSDI’08). USENIX Association, Berkeley, CA, USA, 29-42.

3 Christopher Olston, Benjamin Reed, Utkarsh Srivastava, Ravi Kumar,and Andrew Tomkins. 2008. Pig latin: a not-so-foreign language fordata processing. In Proceedings of the 2008 ACM SIGMODinternational conference on Management of data (SIGMOD ’08). ACM,New York, NY, USA, 1099-1110.


http://homes.cs.washington.edu/~billhowe/mapreduce_a_major_step_backwards.html

http://homes.cs.washington.edu/~billhowe/mapreduce_a_major_step_backwards.html

M

References (2)

4 Tyson Condie, Neil Conway, Peter Alvaro, Joseph M. Hellerstein,Khaled Elmeleegy, and Russell Sears. 2010. MapReduce online. InProceedings of the 7th USENIX conference on Networked systemsdesign and implementation (NSDI’10). USENIX Association, Berkeley,CA, USA.

5 Michael Isard, Mihai Budiu, Yuan Yu, Andrew Birrell, and DennisFetterly. 2007. Dryad: distributed data-parallel programs fromsequential building blocks. In Proceedings of the 2nd ACMSIGOPS/EuroSys European Conference on Computer Systems 2007(EuroSys ’07). ACM, New York, NY, USA, 59-72.

6 Derek G. Murray, Malte Schwarzkopf, Christopher Smowton, StevenSmith, Anil Madhavapeddy, and Steven Hand. 2011. CIEL: a universalexecution engine for distributed data-flow computing. In Proceedings ofthe 8th USENIX conference on Networked systems design andimplementation (NSDI’11). USENIX Association, Berkeley, CA, USA.


References (3)

7 Pramod Bhatotia, Alexander Wieder, Rodrigo Rodrigues, Umut A.Acar, and Rafael Pasquin. 2011. Incoop: MapReduce for incrementalcomputations. In Proceedings of the 2nd ACM Symposium on CloudComputing (SOCC ’11). ACM, New York, NY, USA.


Technology

Topic 7: Shortcomings in the MapReduce Paradigm