Data Insights in NetflixDanny Yuan (@g9yuayon)Jae Bae

1Friday, March 1, 13

Who Am I?

2Friday, March 1, 13

Who Am I?Member of Netflix’s Platform Engineering team, working on very large scale data infrastructure (@g9yuayon)

2Friday, March 1, 13

Who Am I?Member of Netflix’s Platform Engineering team, working on very large scale data infrastructure (@g9yuayon)

Built and operated Netflix’s cloud crypto service

2Friday, March 1, 13

Who Am I?Member of Netflix’s Platform Engineering team, working on very large scale data infrastructure (@g9yuayon)

Built and operated Netflix’s cloud crypto service

Worked with Jae Bae on querying multi-dimensional data in real time

2Friday, March 1, 13

3Friday, March 1, 13

Developers usually think about monitoring metrics when “real-time” data is mentioned. We have powerful monitoring systems that track millions of metrics per second. But I’m not going to talk about it today. Monitoring metric is crucial data. That itself would warrant another multi-hour talk by our monitoring team. :-)

No Monitoring Metrics Today

3Friday, March 1, 13

Developers usually think about monitoring metrics when “real-time” data is mentioned. We have powerful monitoring systems that track millions of metrics per second. But I’m not going to talk about it today. Monitoring metric is crucial data. That itself would warrant another multi-hour talk by our monitoring team. :-)

photo credit: http://www.flickr.com/photos/decade_null/142235888/sizes/o/in/photostream/

4Friday, March 1, 13

Instead, I’m going to talk about logs. Why is it interesting at all?

1,500,000

5Friday, March 1, 13

During peak hours, our data pipeline collects over 1.5 million log events per second

70,000,000,000

6Friday, March 1, 13

Or 70 billions a day on average.

photo credit: http://www.flickr.com/photos/decade_null/142235888/sizes/m/in/photostream/

Server Farm

Server Farm

Server Farm

Log Collectors

Log Filter

Log Filter

Log Filter

Sink Plugin

Sink Plugin

Sink Plugin ElasticSearch

Hadoop

KafkaDruid

7Friday, March 1, 13

We have this tens of thousands of machines, all of which send log data over a robust data pipeline to highly reliable data collectors. The collectors then filter the data, transform the data, and dispatch the data to to different destinations for further processing.

Photo credit: http://www.flickr.com/photos/decade_null/142235888/sizes/m/in/photostream/

Highly Reliable Data Pipeline

photo credit: http://www.flickr.com/photos/decade_null/142235888/sizes/m/in/photostream/

Server Farm

Server Farm

Server Farm

Log Collectors

Log Filter

Log Filter

Log Filter

Sink Plugin

Sink Plugin

Sink Plugin ElasticSearch

Hadoop

KafkaDruid

7Friday, March 1, 13

We have this tens of thousands of machines, all of which send log data over a robust data pipeline to highly reliable data collectors. The collectors then filter the data, transform the data, and dispatch the data to to different destinations for further processing.

Photo credit: http://www.flickr.com/photos/decade_null/142235888/sizes/m/in/photostream/

A Humble Beginning

8Friday, March 1, 13

We didn’t build everything in one night. Actually, we had a humble start. I did a lot of log scraping like these. I also used R to analyze logs. But these are specific tasks, and at some point

A Humble Beginning

8Friday, March 1, 13

We didn’t build everything in one night. Actually, we had a humble start. I did a lot of log scraping like these. I also used R to analyze logs. But these are specific tasks, and at some point

A Humble Beginning

8Friday, March 1, 13

We didn’t build everything in one night. Actually, we had a humble start. I did a lot of log scraping like these. I also used R to analyze logs. But these are specific tasks, and at some point

A Humble Beginning

8Friday, March 1, 13

We didn’t build everything in one night. Actually, we had a humble start. I did a lot of log scraping like these. I also used R to analyze logs. But these are specific tasks, and at some point

9Friday, March 1, 13

Something happened. Our traffic turned into a hockey stick, and the number of applications exploded. So, log traffic also exploded. Simple log scraping wouldn’t cut it any more.

9Friday, March 1, 13

Something happened. Our traffic turned into a hockey stick, and the number of applications exploded. So, log traffic also exploded. Simple log scraping wouldn’t cut it any more.

ApplicationApplication

Application Application

Application

Application

Application

Application

ApplicationApplication

9Friday, March 1, 13

Something happened. Our traffic turned into a hockey stick, and the number of applications exploded. So, log traffic also exploded. Simple log scraping wouldn’t cut it any more.

So We Evolved

10Friday, March 1, 13

So we evolved. One thing we built was a hadoop grep. This tool searches TBs of data. It is much more useful that the one provided by Apache Hadoop Distribution, because it supports many more Grep options like context, sorting by columns, and etc. And DSE’s Hadoop-as-a-service greatly helps each team.

So We Evolved

10Friday, March 1, 13

So we evolved. One thing we built was a hadoop grep. This tool searches TBs of data. It is much more useful that the one provided by Apache Hadoop Distribution, because it supports many more Grep options like context, sorting by columns, and etc. And DSE’s Hadoop-as-a-service greatly helps each team.

hgrep -C 10 -k 5,2,3 'users.*[1-9]{3}' *catalina.out s3//bucket

So We Evolved

10Friday, March 1, 13

So we evolved. One thing we built was a hadoop grep. This tool searches TBs of data. It is much more useful that the one provided by Apache Hadoop Distribution, because it supports many more Grep options like context, sorting by columns, and etc. And DSE’s Hadoop-as-a-service greatly helps each team.

hgrep -C 10 -k 5,2,3 'users.*[1-9]{3}' *catalina.out s3//bucket

So We Evolved

10Friday, March 1, 13

So we evolved. One thing we built was a hadoop grep. This tool searches TBs of data. It is much more useful that the one provided by Apache Hadoop Distribution, because it supports many more Grep options like context, sorting by columns, and etc. And DSE’s Hadoop-as-a-service greatly helps each team.

11Friday, March 1, 13

A search tool that searches live instances’ logs is also developed.

11Friday, March 1, 13

A search tool that searches live instances’ logs is also developed.

11Friday, March 1, 13

A search tool that searches live instances’ logs is also developed.

11Friday, March 1, 13

A search tool that searches live instances’ logs is also developed.

11Friday, March 1, 13

A search tool that searches live instances’ logs is also developed.

11Friday, March 1, 13

A search tool that searches live instances’ logs is also developed.

Field Name Field Value

Client “API”

Server “Cryptex”

StatusCode 200

ResponseTime 73

12Friday, March 1, 13

Hive becomes indispensable.

13Friday, March 1, 13

DSE Sting is a bless.

13Friday, March 1, 13

DSE Sting is a bless.

13Friday, March 1, 13

DSE Sting is a bless.

14Friday, March 1, 13

So we built yet another tool to scratch it with the help of Druid.

Still, We Have a Real-Time Itch

14Friday, March 1, 13

So we built yet another tool to scratch it with the help of Druid.

15Friday, March 1, 13

Error summary in the past 10 seconds. You get to slice and dice through arbitrary combination of different dimension across multiple time series.

Trends over search query of “90210” by Canadians

How many people started streaming any episode of House of Cards in the past hour, grouped by city and sorted by count?

15Friday, March 1, 13

Error summary in the past 10 seconds. You get to slice and dice through arbitrary combination of different dimension across multiple time series.

Trends over search query of “90210” by Canadians

How many people started streaming any episode of House of Cards in the past hour, grouped by city and sorted by count?

15Friday, March 1, 13

Error summary in the past 10 seconds. You get to slice and dice through arbitrary combination of different dimension across multiple time series.

Trends over search query of “90210” by Canadians

How many people started streaming any episode of House of Cards in the past hour, grouped by city and sorted by count?

16Friday, March 1, 13

A query of all the users who started streaming House of Cards in the past three hours, and results came back in seconds.

16Friday, March 1, 13

A query of all the users who started streaming House of Cards in the past three hours, and results came back in seconds.

16Friday, March 1, 13

A query of all the users who started streaming House of Cards in the past three hours, and results came back in seconds.

Interested?

17Friday, March 1, 13

See You Tomorrow

18Friday, March 1, 13

If you’re interested in how we did the real-time interactive queries with the help of Druid, do come to our talk. See you tomorrow