Are you Ready for Big Data?

Dr. Putchong UthayopasDepartment of Computer Engineering, Faculty

of Engineering, Kasetsart University. pu@ku.ac.th

We are living in the world of Data

GeophysicalExploration

Medical Imaging

VideoSurveillance

Mobile Sensors

Gene Sequencing

Smart Grids

Social Media

Big Data

“Big data is data that exceeds the processing capacity of conventional database systems. The data is too big, moves too fast, or doesn’t fit the strictures of your database architectures. To gain value from this data, you must choose an alternative way to process it.”

Reference: “What is big data? An introduction to the big data landscape.”, Edd Dumbill, http://radar.oreilly.com/2012/01/what-is-big-data.html

The Value of Big Data

• Analytical use– Big data analytics can reveal insights hidden previously by

data too costly to process. • peer influence among customers, revealed by analyzing

shoppers’ transactions, social and geographical data.

– Being able to process every item of data in reasonable time removes the troublesome need for sampling and promotes an investigative approach to data.

• Enabling new products.– Facebook has been able to craft a highly personalized user

experience and create a new kind of advertising business

3 Characteristics of Big Data

Big Data Challenge

• Volume– How to process data so big that can not be move, or store.

• Velocity– A lot of data coming very fast so it can not be stored such

as Web usage log , Internet, mobile messages. Stream processing is needed to filter unused data or extract some knowledge real-time.

• Variety– So many type of unstructured data format making

conventional database useless.

How to deal with big data

• Integration of – Storage – Processing– Analysis Algorithm– Visualization

Massive Data

Stream

Stream processing

Processing

Processing

Processing

Visualize

Analysis

Storage

A New Approach For Distributed Big Data

• Disparate Systems• Manual Administration• One Tenant, Many Systems• IT Provisioned Storage

• Single System Across Locations• Automated Policies • Many Tenants One System• Self-Service Access

L.A. BOSTON LONDON L.A. BOSTON LONDON

Storage Islands Single Storage Pool

Hadoop• Hadoop is a platform for distributing computing problems across a

number of servers. First developed and released as open source by Yahoo.– Implements the MapReduce approach pioneered by Google in compiling its

search indexes.– Distributing a dataset among multiple servers and operating on the data: the

“map” stage. The partial results are then recombined: the “reduce” stage.• Hadoop utilizes its own distributed filesystem, HDFS, which makes data

available to multiple computing nodes• Hadoop usage pattern involves three stages:

– loading data into HDFS,– MapReduce operations, and– retrieving results from HDFS.

WHAT FACEBOOK KNOWS

http://www.facebook.com/data

Cameron Marlow calls himself Facebook's "in-house sociologist." He and his team can analyze essentially all the information the site gathers.

Study of Human Society

• Facebook, in collaboration with the University of Milan, conducted experiment that involved – the entire social network as of May 2011– more than 10 percent of the world's population.

• Analyzing the 69 billion friend connections among those 721 million people showed that – four intermediary friends are usually enough to

introduce anyone to a random stranger.

The links of Love• Often young women specify that

they are “in a relationship” with their “best friend forever”.– Roughly 20% of all relationships for

the 15-and-under crowd are between girls.

– This number dips to 15% for 18-year-olds and is just 7% for 25-year-olds.

• Anonymous US users who were over 18 at the start of the relationship– the average of the shortest number

of steps to get from any one U.S. user to any other individual is 16.7.

– This is much higher than the 4.74 steps you’d need to go from any Facebook user to another through friendship, as opposed to romantic, ties.

http://www.facebook.com/notes/facebook-data-team/the-links-of-love/10150572088343859

Graph shown the relationship of anonymous US users who were over 18 at the start of the relationship.

Why?

• Facebook can improve users experience – make useful predictions about users' behavior– make better guesses about which ads you might

be more or less open to at any given time• Right before Valentine's Day this year a

blog post from the Data Science Team listed the songs most popular with people who had recently signaled on Facebook that they had entered or left a relationship

How facebook handle Big Data?

• Facebook built its data storage system using open-source software called Hadoop.– Hadoop spreading them across many machines inside a data center.– Use Hive, open-source that acts as a translation service, making it

possible to query vast Hadoop data stores using relatively simple code.

• Much of Facebook's data resides in one Hadoop store more than 100 petabytes (a million gigabytes) in size, says Sameet Agarwal, a director of engineering at Facebook who works on data infrastructure, and the quantity is growing exponentially. "Over the last few years we have more than doubled in size every year,”

All DataFaster AnswersElastic & Scalable

1The Journey To Big Data

2 Data Science CollaborationSelf-Service

Agile AnalyticsPeople & Productivity Focus

3 Real Time DecisionsNew ApplicationsData Monetization

Analytic Productivity Platform

Agile Process & Tools

Predictive EnterpriseApplication Focus

Big Data Enabled Apps

Big Data InfrastructureTechnology Focus

Analytics Engines

Cloud Infrastructure

Analytic Engines

Data Tsunami

• Data flood is coming, no where to run now!– Data being generated anytime,

anywhere, anyone– Data is moving in fast– Data is too big to move, too

big to store• Better be prepare– Use this to enhance your

business and offer better services to customer

Thank you