Upload
salil-navgire
View
467
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Citation preview
Map Reduce and Hadoop
- SALIL NAVGIRE
Big Data Explosion• 90% of today's data was created in the
last 2 years
• Moore's law: Data volume doubles every 18 months
• YouTube: 13 million hours and 700 billion views in 2010
• Facebook: 20TB/day (compressed)
• CERN/LHC: 40TB/day (15PB/year)
• Many more examples
Solution: Scalability
How?
Divide and Conquer
Challenges!• How to assign units of work to the workers?
• What if there are more units of work than workers?
• What if the workers need to share intermediate incomplete data?
• How do we aggregate such intermediate data?
• How do we know when all workers have completed their assignments?
• What if some workers failed?
History• 2000: Apache Lucene: batch index updates
and sort/merge with on disk index
• 2002: Apache Nutch: distributed, scalable open source web crawler
• 2004: Google publishes GFS and MapReduce papers
• 2006: Apache Hadoop: open source Java implementation of GFS and MapReduce to solve Nutch’ problem; later becomes standalone project
What is Map Reduce?• A programming model to distribute a task
on multiple nodes
• Used to develop solutions that will process large amounts of data in a parallelized fashion in clusters of computing nodes
• Original MapReduce paper by Google
• Features of MapReduce:• Fault-tolerance
• Status and monitoring tools
• A clean abstraction for programmers
MapReduce Execution Overview
UserProgram
Worker
Worker
Master
Worker
Worker
Worker
fork fork fork
assignmap
assignreduce
readlocalwrite
remoteread,sort
OutputFile 0
OutputFile 1
write
Split 0Split 1Split 2
Input Data
Hadoop Components
HDFS
Storage
Self-healinghigh-bandwidthclustered storage
MapReduce
Processing
Fault-tolerantdistributedprocessing
HDFS Architecture
HDFS Basics• HDFS is a filesystem written in Java
• Sits on top of a native filesystem
• Provides redundant storage for massive amounts of data
• Use Commodity devices
HDFS Data• Data is split into blocks and stored on multiple nodes in the cluster
• Each block is usually 64 MB or 128 MB
• Each block is replicated multiple times
• Replicas stored on different data nodes
2 Types of Nodes
Master Nodes
Slave Nodes
Master Node• NameNode
• only 1 per cluster
• metadata server and database
• SecondaryNameNode helps with some housekeeping
• JobTracker• only 1 per cluster
• job scheduler
Slave Nodes• DataNodes
• 1-4000 per cluster
• block data storage
• TaskTrackers
• 1-4000 per cluster
• task execution
NameNode• A single NameNode stores all metadata, replication of blocks and read/write access to files
• Filenames, locations on DataNodes of each block, owner, group, etc.
• All information maintained in RAM for fast lookup
Secondary NameNode• Does memory-intensive administrative functions for the NameNode
• Should run on a separate machine
Data Node• DataNodes store file contents
• Different blocks of the same file will be stored on different DataNodes
• Same block is stored on three (or more) DataNodes for redundancy
Word Count Example• Input• Text files
• Output• Single file containing (Word <TAB> Count)
• Map Phase• Generates (Word, Count) pairs
• [{a,1}, {b,1}, {a,1}] [{a,2}, {b,3}, {c,5}] [{a,3}, {b,1}, {c,1}]
• Reduce Phase• For each word, calculates aggregate
• [{a,7}, {b,5}, {c,6}]
Typical Cluster• 3-4000 commodity servers
• Each server • 2x quad-core
• 16-24 GB ram
• 4-12 TB disk space
• 20-30 servers per rack
When Should I use it?Good choice for jobs that can be broken into parallelized jobs:
• Indexing/Analysis of log files
• Sorting of large data sets
• Image Processing/Machine Learning
Bad choice for serial or low latency jobs:
• For real-time processing
• For processing intensive task with little data
• Replacing MySQL
Who uses Hadoop?