CSci 5707, Fall 2013
MapReducevs.
Parallel DBMS
Hamid Safizadeh, Otelia Buffington
University of Minnesota
2
MapReduce Idea
Mapping
map (k1, v1) list (k2, v2)
Reducing
reduce (k2, list(v2)) list (v2)
Pseudo-code for counting the number of occurrences of each word in a large collection of documents
Jeffrey Dean and Sanjay Ghemawat, MapReduce: Simplified Data Processing on Large Clustering, OSDI’08
3
MapReduce Example
Calculation of the number of occurrences of each word
http://aimotion.blogspot.com/2010/08/mapreduce-with-mongodb-and-python.html
4
MapReduce Architecture
Execution overview
Jeffrey Dean and Sanjay Ghemawat, MapReduce: Simplified Data Processing on Large Clustering, OSDI’08
5
MapReduce or Parallel DBMS
Pavlo, A., Paulson, E., Rasin, A., Abadi, D.J., DeWitt, D.J., Madden, S., and Stonebraker, M., “A comparison of approaches to large-scale data analysis”, ACM SIGMOD International Conference, 2009 (http://database.cs.brown.edu/projects/mapreduce-vs-dbms)
Dean, J., and Ghemawat, S., “MapReduce: A flexible data processing tool”, Communications of the ACM, Vol. 53, 2010 (DOI: 10.1145/1629175.1629198)
MapReduce Design Properties
6
Heterogeneous Systems Processing and combining data from a wide variety of storage systems
(such as relational databases, file systems, etc.)
Fault Tolerance Providing fine-grain fault tolerance for large jobs (Failure in middle of a
multi-hour execution does not require restarting the job from scratch)
Complex Functions Simple Map and Reduce functions with straightforward SQL equivalents Offering a better framework for some complicated tasks
Jeffrey Dean and Sanjay Ghemawat, MapReduce: A Flexible Data Processing Tool, Communications of the ACM, Vol. 53, 2010
MapReduce Design Properties
7
Performance Loading data: Startup overhead for MapReduce Reading data: Full scan over large data files Merging results: A MapReduce as the next consumer
Jeffrey Dean and Sanjay Ghemawat, MapReduce: A Flexible Data Processing Tool, Communications of the ACM, Vol. 53, 2010
Cost Hardware: Network workstations Software: Open source (Hodoop) Communication: Network system
Companies Using Hodoop
8
Facebook Yahoo! Google Amazon Twitter