introduction to big data frameworks

03/05/2023

1

Machines Learning in Big Data ( MapReduce, Knime, Spark)Présenté par: Sahmoudi Yahia Targhi Amal

Proposé par : Bouchra Frikh

03/05/2023

2

Outlines Introduction

Big Data

Machine Learning

Applications of ML Techniques to Data mining Tasks

Why Machine Learning in Big Data?

Big Data frameworks processing

03/05/2023

3

Introduction• Every day, 2.5 quintillion bytes of data are created and 90% of the data

in the world today were produced within the past two years (IBM 2012).

• On October 4, 2012, the first presidential debate between President Barack Obama and Governor Mitt Romney triggered more than 10 million tweets within two hours (Twitter Blog 2012).

• Another example is Flickr, a public picture sharing site, which received 1.8 million photos per day, on average, from February to March 2012 (Michel F. 2012).

03/05/2023

4

•“ The most fundamental challenge for the Big Data applications is to explore the large volumes of data and extract useful information or knowledge for future actions “ (Rajaraman and Ullman, 2011).

03/05/2023

5

• What is big data?

• What is machine learning?

• Why do machine learning in big data?

• What are they the big data processing frameworks

which integrate machine learning algorithm?

03/05/2023

6

Big Data• Big data is a term that describes the large volume of

data ( structed, semi- structed, unstructed), which can exploited to get informations.

• Big data can be characterized by 3Vs:

• the extreme volume of data.• the wide variety of types of data• velocity at which the data must be must processed.

03/05/2023

7

P.S and we can also add: Variability and Complexity.

03/05/2023

8

Big Data AnalyticsBig data analytics is the process of examining large data

sets containing a variety of data types.“ It’s important to remember that the primary value from big data comes not from the data in its raw form, but from the processing and analysis of it and the insights, products, and services that emerge from analysis. The sweeping changes in big data technologies and management approaches need to be accompanied by similarly dramatic shifts in how data supports decisions and product/service innovation.” Thomas H. Davenport in  Big Data in Big Companies

03/05/2023

9

Machine LearningDesign off implementation methods allowing a machine to evolve by

a systematic process and completing tasks which are difficult

They are difficult for a classic algorithm.

Machine Learning it’s when de data replace the algortihm

Example

Facebook's News Feed changes according to the user's personal interactions with

other users. If a user frequently tags a friend in photos, writes on his wall or likes his

links, the News Feed will show more of that friend's activity in the user's News Feed

due to presumed closeness.

03/05/2023

10

Applications of ML Techniques to Data mining Tasks

• The amount of the data seems to increase rapidly every single day for the

majority domains related to information processing, and the need to find a

way to mine and get knowledge from databases is still crucial.

• Data Mining (DM) defines the automated extraction procedures of hidden

predictive information from databases . Among the tasks of DM are:

• Diagnosis• Pattern Recognition: • Prediction• Classification• Clustering• Optimization: • Control:

03/05/2023

11

The impact of ML techniques used for DM tasks

Artificial Neural Networks for DMGenetic Algorithms for DMInductive Logic Programming for DMRule Induction for DMDecision Trees for DMInstance-based Learning Algorithms for DM

03/05/2023

12

Exemple•Amazon is an interesting example of

application of "Machine Learning" in e-commerce. For example, suppose we perform the search for a product on Amazon today. When we come back another day on the site, it is able to offer us products related to our specific needs (prediction). And thanks to algorithms of "Machine Learning" which provide our changing needs from our previous visits to the site.

03/05/2023

13

Why Machine Learning in big Data? • It delivers on the promise of extracting value from big and

disparate data sources with far less reliance on human direction. It is data driven and runs at machine scale. It is well suited to the complexity of dealing with disparate data sources and the huge variety of variables and amounts of data involved. And unlike traditional analysis, machine learning thrives on growing datasets. The more data fed into a machine learning system, the more it can learn and apply the results to higher quality insights.

03/05/2023

14

Big Data framework s processing

• The Machine Learning is far from being a recent

phenomenon. What is new, however, is the number of

treatments platforms parallelized data for the

management of Big Data.

• in this Presentation we will take a look on some of the most

used algorithms as MapReduce, knime, spark.

03/05/2023

15

MapReduce• MapReduce invented by google, it's the heart

of hadoop and it's is a processing technique and a program model for distributed computing based on java.

• MapReduce allows for massive scalability across hundreds or thousands of servers in a Hadoop cluster., allows to write applications to process huge amounts of data, in parallel, on large clusters of commodity hardware in a reliable manner.

03/05/2023

16

• The MapReduce algorithm refers two important tasks,Map and Reduce:

• Map: Map takes a set of data and converts it into another set of data, where individual elements are broken down into tuples (key/value pairs).

• Reduce: takes the output from a map as an input and combines those data tuples into a smaller set of tuples. As the sequence of the name MapReduce implies, the reduce task is always performed after the map job.

03/05/2023

17

Algorithm• Map stage : The map or mapper’s job is to process the input data.

Generally the input data is in the form of file or directory and is

stored in the Hadoop file system (HDFS). The input file is passed to

the mapper function line by line. The mapper processes the data and

creates several small chunks of data.

• Reduce stage : This stage is the combination of the Shuffle stage and

the Reduce stage. The Reducer’s job is to process the data that

comes from the mapper. After processing, it produces a new set of

output, which will be stored in the HDFS.

03/05/2023

18

03/05/2023

19

Exemple•The Word Count program used in this

section to explain how a map will also be used to explain the operation of a reducer.

•the object of this program is: counting the number of occurrences of the different words which make up a book.

03/05/2023

20

Book Content

book's content

03/05/2023

21

Step 1 : Map

• Line 1: the mapper reads in input a record that comes in the form of a

pair <key, value> with:

• the String value type (a line of the file).

• LongWritable the key type

• Line 2 : for each word in the current line.

• Line 3 : we write on a output file the couple <w,1> , corresponding to

an occurrence of the word in the variable w.

03/05/2023

22

the output file of the mapper

03/05/2023

23

Step 2 ( between map and Reduce: shuffle) and sort

03/05/2023

24

Step3 : Reduce

Line 1: the reducer input a recording as a pair <key, value> with:

Text the key type (one word).

value is a list of values of intWritable type.

Line 2: the reducer resets the counter when the wordcount word is changed.  

Line 3: v for each value in the list is added to inValue2 v Word-Count (in this case v is

always 1).

Line4: When we change the word, we write in a output file: couple <inKey2, wordCount>,

wordCount is the number of occurrences of the word.

03/05/2023

25

03/05/2023

26

Use case in real life

• http://hadoopilluminated.com/hadoop_book/Hadoop_Use_Cases.html#d1575e1290

03/05/2023

27

Spark•Apache Spark is an open source cluster computing 

framework originally developed in the AMPLab at 

University of California, 

•Spark extends the popular MapReduce model to

efficiently support more types of computations.

03/05/2023

28

A Unified Stack

• one of the largest advantages of tight integration is the ability to build

applications that seamlessly combine different processing models. For

example, in Spark you can write one application that uses machine

learning to classify data in real time as it is ingested from streaming

sources. Simultaneously, analysts can query the resulting data, also in real

time, via SQL (e.g., to join the data with unstructured logfiles).

• In addition, more sophisticated data engineers and data scientists can

access the same data via the Python shell for ad hoc analysis. Others

might access the data in standalone batch applications. All the while, the

IT team has to maintain only one system.

03/05/2023

29

Rdd• the resilient distributed dataset (RDD). An RDD is simply a

distributed collection of elements. In Spark all work is

expressed as either creating new RDDs, transforming existing

RDDs, or calling operations on RDDs to compute a result. Under

the hood, Spark automatically distributes the data contained in

RDDs across your cluster and Parallelizes the operations you

perform on them.

• RDDs are the core concept in Spark.

• You can see a RDD as a table in a database.

03/05/2023

30

• Transformations: transformations do not return a single value, they return a new RDD. Nothing is evaluated when one uses a transformation function, this function just takes and RDD and returns a new RDD . The processing functions such as map, filter, flatMap, groupByKey, reduceByKey, aggregateByKey ...

• Actions: shares evaluate and return a new value. When an action function is called on a RDD object, all data processing requests are calculated and the result is returned. The actions are reduce, collect, count, first, take, countByKey and foreach ...

03/05/2023

31

Spark EcoSystem

03/05/2023

32

Spark CoreSpark Core contains the basic functionality of Spark, including components for task

scheduling, memory management, fault recovery, interacting with storage systems,

and more. Spark Core is also home to the API that defines resilient distributed

datasets

(RDDs), which are Spark’s main programming abstraction. RDDs represent a

collection of items distributed across many compute nodes that can be manipulated

in parallel. Spark Core provides many APIs for building and manipulating these

collections.

03/05/2023

33

Example in real Life

Another early Spark adopter is Conviva, one of the largest streaming video companies on the Internet, with about 4 billion video feeds per month (second only to YouTube). As you can imagine, such an operation requires pretty sophisticated behind-the-scenes technology to ensure a high quality of service. As it turns out, it’s using Spark to help deliver that QoS by avoiding dreaded screen buffering.

03/05/2023

34

MapReuce Vs SparkSpark MapReduce

Performance  Spark processes data in-memory

On the Disc

Compatibility Apache Spark can run as standalone or

on top of Hadoop YARN or on the

cloud. ..

With hadoop

Data Processing Data Batch

Failure Tolerance + +

Security is still in its infancy Hadoop MapReduce has more security

features and projects.n it comes

to security

03/05/2023

35

Conclusion