Foursquare Data and Explore Final

7/28/2011 Check-in and Waffles Ben Lee - @benlee

Data and Explore!

Friday, July 29, 2011

Ben Lee - @benlee7/28/2011 Check-in and Waffles

Overview

• Data intro

• Explore recommendation engine

• Technology stack

• More infographics



Data model at a glance

Check-ins

Users

Venues

@

8 PM“Lobster roll!”



Data model at a glance

> 10 million

> 15 million

> 750 million> 3 million / day

Check-ins

Users

Venues

@

8 PM“Lobster roll!”



Our data

• Human mobility patterns

• Explicit geo-spatial/temporal data

• Many ways to interpret, filter, slice, visualize



Growth



Wall Street Journal Study

• Week of Jan 21, 2011

• Firehose of ~10.9 million checkins

• Examined NYC vs SF Bay area



WSJ NYC Heatmp



WSJ SF Heatmap



WSJ factoids• NYC vs SF Bay Area

• Male vs Female

• Women check into sushi, libraries, karaoke more

• Men check into burgers, tech startups, gay bars more

• 50/50 split on mexican, ramen, beaches, desserts

• http://graphicsweb.wsj.com/documents/FOURSQUAREWEEK1104/


http://graphicsweb.wsj.com/documents/FOURSQUAREWEEK1104/

http://graphicsweb.wsj.com/documents/FOURSQUAREWEEK1104/


Explore

• Social recommendation engine

• Leverage check-in data



Coffee?

• Realtime recommendations

• Time of day

• Previous checkins

• Friend history

• User history



Time of day

• Relevant for the time of day, day of week

• Affinities for different categories at different times

• Per-venue check-in distribution over time is useful



Tartine Bakery



DNA Lounge



Blue Bottle



Previous checkins

•We are not unique snowflakes

•Median user checks-in at to a place that their social circle has been to before > 60%

repeat checkin %


Friend history

• Social justification

•Call out “similar” friends

• Surface their tips



User history

• Places I’ve been



User history

•Highlight places “similar” to places I’ve been



Venue similarity

People that go to Blue Bottle, also go to:



Venue similarity

People that go to Tartine, also go to:



Computing similarity

Incredibly SparseMatrixU

sers

VenuesNYC Food sample





sers

Venues vi vj

for all i,jsim(vi, vj)

Venue similarity

Take a flavor of co-occurrence based similarity





sers

Venues

ui

uj

for all i,jsim(ui, uj)

User similarity



Similarity pipeline

• Input: Mongo dumps on S3 or HDFS

• Java mapreduces via Hadoop / Elastic Mapreduce

• Load output into Mongo for serving



Mapreduce overviewvisited venues map

reduce

vi, vj

vi, vj

vi, vj

user

emit “all” pairs of visited venues for each user

Sum up each user’s score contribution to this pair of

venues

score

score

key

key

score scorescore ...

...

final score



Putting it all together

User’s check-in history

Friends’ check-in history, similarity

Nearby relevantvenues

Similar Venues

MOAR Signals

< 200 ms



Explore performance

• Track behavior over time

•Measure improvements

• Run experiments

Model updated



Explore in the future

• We’ve just started!

• New controls

• Beyond here/now



Data Stack

• MongoDB (production)

• Amazon S3, Elastic Mapreduce

• Hadoop

• Hive

• Flume

• R/RStudio


6/28/2011 Big Data Camp Ben Lee - @benlee

Hive interface

Text



Rudest cities



Boca is Nice!



Happiness analysis• Aditya Mukerjee’s awesome work

• Happy

• “woohoo!” SF

• “i love you, bone-in filet. so, so much.” NYC

• Sad

• “stupid at&t and no reception” SF

• “fuck mondays” NYC


Ben Lee - @benlee7/28/2011 Check-in and WafflesFriday, July 29, 2011


*pocalypse• A tradition of hyperbole

• Snowpocalypse 2011

• Heatpocalypse 2011

• NY State senate passes marriage equality 6/24 Marriage Equalitocalypse



Aditya’s Equalitocalypse gephi viz



Love data?

• Join the team!

• http://foursquare.com/jobs


http://foursquare.com/jobs




Documents

Foursquare Data and Explore Final