105
Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Embed Size (px)

Citation preview

Page 1: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Social DataToby Segaran

Author, Programming Collective Intelligence

Data Magnate, Metaweb Technologies

Page 2: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Data mining?

“Sorting through data* to identify patterns and establish

relationships”

* usually a lot of data

Page 3: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Where and why?

Methods and examples

Page 4: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Where and why?•Targeted Advertising

•Recommendations

•Search Results

•Group Discovery

•Filtering of Documents

•Theme Extraction

Page 5: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Google ad

Page 6: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Facebook ad

Page 7: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

This is strange...

•Google just has text

•Facebook knows more about me

•But it’s taking a few cues...

Page 8: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Status: “engaged”

Page 9: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Where and why?•Targeted Advertising

•Recommendations

•Search Results

•Group Discovery

•Filtering of Documents

•Theme Extraction

Page 10: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Real Amazon Products

Page 11: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Netflix Prize

Page 12: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Strands Contest

Page 13: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Custom News

Page 14: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Custom News

Page 15: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Custom News

Page 16: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Where and why?•Targeted Advertising

•Recommendations

•Search Results

•Group Discovery

•Filtering of Documents

•Theme Extraction

Page 17: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Ranking algorithms

The now-incredibly-famous paper

Page 18: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Ranking algorithms

Page 19: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

•Google begins tracking clicks in 2005

•MSN search claims neural network

•AOL Data Scandal

Learning behavior

Page 20: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Where and why?•Targeted Advertising

•Recommendations

•Search Results

•Group Discovery

•Filtering of Documents

•Theme Extraction

Page 21: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

In Biology

Page 22: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Page Grouping

Page 23: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Resumes

“Can resumes be groupedinto career paths?”

Page 24: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Where and why?•Targeted Advertising

•Recommendations

•Search Results

•Group Discovery

•Filtering of Documents

•Theme Extraction

Page 25: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

The obvious: spamSpamBayes

Page 26: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Other email uses

Page 27: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Web documents

“As you add information to Twine, it is automatically tagged so that you and others

can find it more easily”

Page 28: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Where and why?•Targeted Advertising

•Recommendations

•Search Results

•Group Discovery

•Filtering of Documents

•Theme Extraction

Page 29: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

What is the buzz?

Page 30: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Customer Community

Page 31: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Where and why?

Methods and examples

Page 32: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Methods and Examples•Bayesian Filtering

•Distance Metrics

•Clustering

•Decision Trees

•Network Analysis

•Feature Extraction

Page 33: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Bayesian Filtering

Page 34: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Bayesian Filtering

Page 35: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Bayesian Filtering

Page 36: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Bayesian Filtering

Page 37: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Bayesian Filtering

Page 38: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Bayesian Filteringschoolwork

algorithm

Page 39: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Bayesian Filteringschoolwork

algorithm

v1agratrades

associate

Page 40: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Craigslist personals

Page 41: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Analysis

Five Cities

W4M Personal Ads

Page 42: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

ResultsNew YorkMets

Lounges

Offense

Desires

Musical

Submissive

Create

Song

Oral

BostonPink

Sox

Poetry

Intellectually

Punk

Appreciation

Exercise

Winter

Education

ChicagoCubs

Burbs

Bears

Girlie

Insecure

Cheat

Importance

Blunt

Mouth

Page 43: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

ResultsLos AngelesExcellent

Vegas

Meaningful

Star

Lame

Industry

Heat

Fitness

Entertainment

Latino

San FranciscoTee

Employment

Picnic

STD

Tasting

Hikes

French

.com

Kayaking

Cycling

Page 44: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Methods and Examples•Bayesian Filtering

•Distance Metrics

•Clustering

•Decision Trees

•Network Analysis

•Feature Extraction

Page 45: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Preference distance

Sarah Marshall

Leatherheads

3

3

2

3

1

5

2

5

Page 46: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Preference distance5

4

3

2

1

1 2 3 4 5

Page 47: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Preference distance5

4

3

2

1

1 2 3 4 5

1

2.23

Page 48: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

For recommendations5

4

3

2

1

1 2 3 4 5

Prom Night: 5 Prom Night: 2?

1

2.23

Page 49: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

For recommendations5

4

3

2

1

1 2 3 4 5

Prom Night: 5 Prom Night: 24.1

Page 50: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Linguistic distanceThe

Six

Degrees

Hypothesis

Experienced

It

Is

When

You

Travel

Page 51: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Linguistic distanceThe

Six

Degrees

Hypothesis

Experienced

It

Is

When

You

Travel

Six

Degrees

Hypothesis

Experienced

Travel

Six 3

Degrees 3

Hypothesis 1

Experienced 5

Travel 6

Page 52: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Linguistic distance

“china” “kids” “music” “travel” “yahoo”

Gothamist 0 3 3 3 0

GigaOM 6 0 1 4 2

QuickOnlineTips 0 2 2 0 12

O’Reilly Radar 1 0 3 6 4

Page 53: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Linguistic distance“china” “kids” “music” “yahoo”

Gothamist 0 3 3 0

GigaOM 6 0 1 2

Quick Online Tips 0 2 2 12

Euclidean “as the crow flies”

= 12 (approx)

Page 54: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Article/blog similarity

Valleywag - Huffington > Slashdot - Wired

Page 55: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Methods and Examples•Bayesian Filtering

•Distance Metrics

•Clustering

•Decision Trees

•Network Analysis

•Feature Extraction

Page 56: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Hierarchical Clustering5

4

3

2

1

1 2 3 4 5

Page 57: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Hierarchical Clustering5

4

3

2

1

1 2 3 4 5

Page 58: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Hierarchical Clustering5

4

3

2

1

1 2 3 4 5

Page 59: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Hierarchical Clustering5

4

3

2

1

1 2 3 4 5

Page 60: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Hierarchical Clustering

Page 61: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Grouping bloggers

Page 62: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Grouping bloggers

Page 63: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Grouping bloggers

Page 64: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Grouping articles

Page 65: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies
Page 66: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies
Page 67: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies
Page 68: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Methods and Examples•Bayesian Filtering

•Distance Metrics

•Clustering

•Decision Trees

•Network Analysis

•Feature Extraction

Page 69: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Decision Trees

Page 70: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

CART AlgorithmBrand Type

Life (hrs)

Duracell

C 4

Energizer

C 5

Duracell

AA 2

Energizer

AA 2.5From any dataset...

Page 71: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

CART AlgorithmBrand Type

Life (hrs)

Duracell

C 4

Energizer

C 5

Duracell

AA 2

Energizer

AA 2.2... find the best split ...

Type is C?

Avg=4.5Avg=2.1

No Yes

Page 72: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

CART AlgorithmBrand Type

Life (hrs)

Duracell

C 4

Energizer

C 5

Duracell

AA 2

Energizer

AA 2.2... and repeat.

Type is C?No Yes

DuracellNo Yes

DuracellNo Yes

42.2 2 5

Page 73: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Hot or Not

Page 74: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Hot or Not

Page 75: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Methods and Examples•Bayesian Filtering

•Distance Metrics

•Clustering

•Decision Trees

•Network Analysis

•Feature Extraction

Page 76: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

A networkAA

BB

CCDD

EE

FF

Page 77: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

PageRankAA

BB

CCDD

EE

FF

1.0

1.0

1.01.0

1.0

1.0

Page 78: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

PageRankAA

BB

CCDD

EE

FF

1.0

1.0

1.01.0

1.0

1.0D = 0.15 + .85*E/1 + .85 * F/2 + .85*B/1 = 2.275

Page 79: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

PageRankAA

BB

CCDD

EE

FF

0.58

0.58

1.02.275

1.0

0.15

Page 80: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

PageRankAA

BB

CCDD

EE

FF

0.58

0.58

2.081.56

0.3

0.15

Page 81: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

PageRankAA

BB

CCDD

EE

FF

1.03

1.03

1.481.56

0.3

0.15

Page 82: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

PageRankAA

BB

CCDD

EE

FF

0.78

0.78

1.481.34

0.3

0.15

Page 83: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

CI FOO participants

Page 84: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Science papers

The paper attempts to provide an alternative method for measuring the importance of scientific

papers based on the Google's PageRank. The method is a meaningful extension of the common

integer counting of citations and is then experimented for bringing PageRank to the

citation analysis in a large citation network. It offers a more integrated picture of the

publications' influence in a specific field.

Bringing PageRank to the citation analysis

Page 85: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Clustering coefficient

“How many of each persons friendsare friends with each other?”

Page 86: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Clustering coefficient

AA

BBCC

DD

EEFF

Low clustering coefficient

Page 87: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Clustering coefficient

AA

BBCC

DD

EEFF

High clustering coefficient“small world graph”

Page 88: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies
Page 89: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies
Page 90: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies
Page 91: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Twitter!

Page 92: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Twitter!

Page 93: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Methods and Examples

•Bayesian Filtering

•Distance Metrics

•Clustering

•Decision Trees

•Network Analysis

•Feature Extraction

Page 94: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Independent Features

Page 95: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Message boards

Page 96: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Message boards

Page 97: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Matrix Factorization

Msg1 Msg2 Msg3 Msg4 Msg5

Gym 1 3 3 0 1

Calorie 0 2 4 1 3

Weigh 2 3 1 0 1

Carbs 0 1 1 0 2

Treadmill 3 2 0 2 2

Msg1 M2 M3 M4 M5

F1 1 0 2 3 0

F2 0 2 1 1 3

F3 1 0 2 0 0

F1 F2 F3

Gym 0 1 2

Calorie 2 0 1

Weigh 2 2 1

Carbs 1 0 3

Treadmill 0 1 2

Features MatrixWeight Matrix

x

Current Guess

Page 98: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Matrix Factorization

Msg1 M2 M3 M4 M5

F1 1 0 2 3 0

F2 0 2 1 1 3

F3 1 0 2 0 0

F1 F2 F3

Gym 0 1 2

Calorie 2 0 1

Weigh 2 2 1

Carbs 1 0 3

Treadmill 0 1 2

Features MatrixWeight Matrix

x

Msg1 Msg2 Msg3 Msg4 Msg5

Gym 2 0 0 3 0

Calorie 0 2 1 1 3

Weigh 1 0 2 0 0

Carbs 0 3 0 0 2

Treadmill 1 0 0 2 0

Target Result

Msg1 Msg2 Msg3 Msg4 Msg5

Gym 1 3 3 0 1

Calorie 0 2 4 1 3

Weigh 2 3 1 0 1

Carbs 0 1 1 0 2

Treadmill 3 2 0 2 2

Current Guess

Page 99: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Matrix Factorization

Msg1 M2 M3 M4 M5

F1 2 0 0 1 0

F2 0 2 0 1 3

F3 1 0 1 0 0

F1 F2 F3

Gym 1 0 0

Calorie 0 1 1

Weigh 0 0 2

Carbs 0 1 0

Treadmill 1 0 0

Features MatrixWeight Matrix

x

Msg1 Msg2 Msg3 Msg4 Msg5

Gym 2 0 0 3 0

Calorie 0 2 1 1 3

Weigh 1 0 2 0 0

Carbs 0 3 0 0 2

Treadmill 1 0 0 2 0

Target Result

Msg1 Msg2 Msg3 Msg4 Msg5

Gym 2 0 0 3 0

Calorie 0 2 1 1 3

Weigh 1 0 2 0 0

Carbs 0 3 0 0 2

Treadmill 1 0 0 2 0

Current Guess

Page 100: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Interpreting Features

Msg1 M2 M3 M4 M5

F1 2 0 0 1 0

F2 0 2 0 1 3

F3 1 0 1 0 0

F1 F2 F3

Gym 1 0 0

Calorie 0 1 1

Weigh 0 0 2

Carbs 0 1 0

Treadmill 1 0 0

Features Matrix

Weight Matrix

Theme 1 Theme 2 Theme 3

Gym Calorie Weigh

Treadmill Carbs Calorie

Msg1 Msg2 Msg3 etc.

Theme 1 Theme 2 Theme 3

Theme 3

Page 101: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Diet & Body themes

AtkinsInductionSouthBeachCarbs

ChocolateBlackCoffeeOliveBroccoli

GymWeightsExerciseRunningInjured

CookRecipeFriedHome Money

OrganicWantBest

CaloriesWeightFatsProteinCholesterol

Page 102: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Wikipedia peoplesheher

afterwhenfatherwomen

seriestelevision

showwhichradiobbc

leaguemajor

baseballseasonplayedwith

olympicscompeted

wonsummermedal

athelete

universityprofessorreceivedscienceresearch

born

Page 103: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

We’re just getting started...

Page 104: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Homepage http://kiwitobes.com

Freebase http://freebase.com

Page 105: Social Data Toby Segaran Author, Programming Collective Intelligence Data Magnate, Metaweb Technologies

Questions?