Big Data in Real Time

Big Data in REAL TIMERon Zavner

We’re Living in a Real Time World…Homeland Security

Real Time Search

Social

eCommerce

User Tracking & Engagement

Financial Services

® Copyright 2011 Gigaspaces Ltd. All Rights Reserved2

The Flavors of Big Data Analytics

Counting Correlating Research

® Copyright 2011 Gigaspaces Ltd. All Rights Reserved3

It takes a week for users to

send 1 billion tweets

® Copyright 2011 Gigaspaces Ltd. All Rights Reserved4

Twitter in Numbers (March 2011)

Source: http://blog.twitter.com/2011/03/numbers.html

On average,

140 milliontweets get sent every day

® Copyright 2011 Gigaspaces Ltd. All Rights Reserved5

Twitter in Numbers (March 2011)

Source: http://blog.twitter.com/2011/03/numbers.html

The highest throughput to date is

6,939 tweets/sec.® Copyright 2011 Gigaspaces Ltd. All Rights Reserved6

Twitter in Numbers (March 2011)

Source: http://blog.twitter.com/2011/03/numbers.html

460,000 new accounts are created daily

® Copyright 2011 Gigaspaces Ltd. All Rights Reserved7

Twitter in Numbers (March 2011)

Source: http://blog.twitter.com/2011/03/numbers.html

Challenge – Word Count

Word:Count

Tweets

CountCount??® Copyright 2011 Gigaspaces Ltd. All Rights Reserved8

Thousands of tweets per second to process Aggregate counters for each word Latency – less than a second System needs to linearly scale System needs to be fault tolerant Querying & Persisting Data Managing the system

Analyze the Problem

® Copyright 2011 Gigaspaces Ltd. All Rights Reserved9

® Copyright 2011 Gigaspaces Ltd. All Rights Reserved10

Tier Based Architecture?

® Copyright 2011 Gigaspaces Ltd. All Rights Reserved11

Data Grid

® Copyright 2011 Gigaspaces Ltd. All Rights Reserved12

Putting it all together

1. Just2. Found3. Love

® Copyright 2011 Gigaspaces Ltd. All Rights Reserved13

The 3 Most Popular Words on Twitter?

- August 2012

RonZ@gigaspaces.com

® Copyright 2011 Gigaspaces Ltd. All Rights Reserved14

Q&A