44
Keeping It Professional: Relevance, Recommendations, and Reputation. Daniel Tunkelang Principal Data Scientist at LinkedIn Danie l 1

Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

Embed Size (px)

DESCRIPTION

Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn Daniel Tunkelang (LinkedIn) LinkedIn operates the world's largest professional network on the Internet with more than 100 million members in over 200 countries. In order to connect its users to the people, opportunities, and content that best advance their careers, LinkedIn has developed a variety of algorithms that surface relevant content, offer personalized recommendations, and establish topic-sensitive reputation -- all at a massive scale. In this talk, I will discuss some of the most challenging technical problems we face at LinkedIn, and the approaches we are taking to address them. Note: This talk was presented at the Carnegie Mellon University School of Computer Science Intelligence Seminar on September 20, 2011. As of May 2013, LinkedIn has over 225 million members.

Citation preview

Page 1: Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

1Recruiting SolutionsRecruiting SolutionsRecruiting Solutions

Keeping It Professional:Relevance, Recommendations, and Reputation.

Daniel TunkelangPrincipal Data Scientist at LinkedIn

Daniel

Page 2: Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

2

Overview

What is LinkedIn?

Hard problems we’re tackling in:

Relevance

Recommendations

Reputation

Open problems

Page 3: Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

3

IdentityConnect, find and be foundLinkedIn Profile, Address Book, Search

InsightsBe great at what you doHomepage, LinkedIn Today, Groups

Work wherever our members work

EverywhereMobile, APIs, Plug-InsDesktop

Rolodex, Resume, Business Card

Newspapers,

Trade Magazines, Events

What is LinkedIn?

Page 4: Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

4

Identity: Profile of Record

Page 5: Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

5

Identity: Connect with Others

Page 6: Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

6

Identity: Join the Conversation

Page 7: Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

7

Insights: Power of Aggregation

Beforeemployees worked at

Yahoo! (169)Google (96)Oracle (78)Microsoft (72)IBM (43)

Beforeemployees worked at

Google(475)Microsoft (448)LinkedIn (169)Apple, Inc.

(154)ebay (133)

Page 8: Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

8

Insights: Market Research

Page 9: Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

9

Insights: Data Stories

Page 10: Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

10

Everywhere

Page 11: Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

11

Hard Problems: Examples

Relevance

People Search

Recommendations

Job Matching

Reputation

Skills

Page 12: Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

12

People Search

Page 13: Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

13

120M+ members

2B searches in 2010

Based on (cf. http://sna-projects.com/)

People Search: Scale

Page 14: Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

14

People Search: Faceted Search

Page 15: Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

15

People Search: Network Facet

Page 16: Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

16

People Search: Type-Ahead

Page 17: Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

17

Query-Independent Signals Network Rank, Profile Quality

Query-Dependent Signals Field-Based Relevance

Personalized Signals Network Distance

People Search: Relevance

Page 18: Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

18

People Search: Query-Independent Signals

Page 19: Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

19

People Search: Network Rank

Page 20: Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

20

People Search: Profile Quality

Page 21: Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

21

People Search: SEO

Page 22: Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

22

People Search: Query-Dependent Signals

Page 23: Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

23

People Search: Inferring Structure

Page 24: Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

24

vs.

vs.

People Search: Ambiguity

Page 25: Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

25

for i in [1..n] s w1 w2 … wi

if Pc(s) > 0 a new Segment() a.segs {s} a.prob Pc(s) B[i] {a} for j in [1..i-1] for b in B[j] s wj wj+1 … wi

if Pc(s) > 0 a new Segment() a.segs b.segs U {s} a.prob b.prob * Pc(s) B[i] B[i] U {a} sort B[i] by prob truncate B[i] to size k

People Search: HMM + Segmentation

Page 26: Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

26

People Search: Personalized Signals

Page 27: Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

27

QCon 2010 presentation by John Wang on “LinkedIn

Search: Searching the Social Graph in Real Time”

http://www.infoq.com/presentations/LinkedIn-Search

SIGIR 2011 Workshop on Entity-Oriented Search

http://research.microsoft.com/en-us/um/beijing/events/eos2011/

HCIR 2011 paper by Jonathan Koren on “Faceted Search

Query Log Analysis” (forthcoming)

http://hcir.info/hcir-2011/

People Search: Further Reading

Page 28: Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

28

Job Matching

Page 29: Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

29

Job Features Job Description, Location, Similar Jobs, …

Candidate Features Profile Data, Network, Activity, …

Standardization Companies, Job Titles, Education, …

Job Matching: Overview

Page 30: Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

30

Corpus StatsJob

User Base

Filtered

titlegeocompany

industrydescriptionfunctional area

Candidate

Generalexpertisespecialtieseducationheadlinegeoexperience

Current Positiontitlesummarytenure lengthindustryfunctional area…

Similarity (candidate expertise, job description)

0.56Similarity

(candidate specialties, job description)

0.2

Transition probability(candidate industry, job industry)

0.43

Title Similarity

0.8

Similarity (headline, title)

0.7

.

.

.derived

Matching

Binary Exact matches: geo, industry, …

Soft transition probabilities, similarity, …

Text

Job Matching: Algorithm

Transition probabilitiesConnectivityyrs of experience to reach title education needed for this title…

Page 31: Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

31

Most people aren't looking for jobs.

Complicates evaluation, training.

Important not to offend users.

e.g., by offering Peter Norvig a postdoc.

You can’t always get what you want

Every employer wants the hottest candidates.

Job Matching: Challenges

Page 32: Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

32

KDD 2011 paper by Bekkerman & Gavish on “High-

Precision Phrase-based Document Classification”

http://www.stanford.edu/~gavish/documents/phrase_based.pdf

SIGIR 2011 paper by Cetintas et al. on “Identifying Similar

People in Professional Social Networks”

http://dl.acm.org/citation.cfm?id=2010123

Blog post on LinkedIn’s recommendation engine

http://blog.linkedin.com/2011/03/02/linkedin-products-you-may-like/

Job Matching: Further Reading

Page 33: Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

33

Skills

Page 34: Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

34

Skills: What are Skills?

Page 35: Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

35

Skills: Identifying Skills

Page 36: Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

36

Skills: Cluster and Disambiguate

angel

Page 37: Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

37

Skills: Assigning Skills to People

Page 38: Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

38

Skills: Who are the Experts?

Page 39: Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

39

• Relevance

• Combine query-independent, query-dependent,

and personalized features.

• Recommendations

• Match people to jobs, groups, news, …

• Reputation

• Expertise relative to professional skills.

Summary: The 3 Rs

Page 40: Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

40

¿Open Problems?

Page 41: Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

41

Exploratory Search

Fact retrieval

Known item search

Navigation

Transition

Verification

Question answering

Knowledge acquisition

Comprehension/Interpretation

Comparison

Aggregation/Integration

Socialize

Accretion

Analysis

Exclusion/Negation

Synthesis

Evaluation

Discovery

Planning/Forecasting

Transformation

Lookup InvestigateLearn

Exploratory Search

Page 42: Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

42

Explore / Exploit

Page 43: Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

43

Incentives for Online Reputation

Page 44: Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

44

Questions?

Contact:

[email protected]

We’re Hiring!

http://engineering.linkedin.com/

Thank You!