Machine Learning to Grow the World's Knowledge

Xavier Amatriain (@xamat)

8/18/2015

Multithreaded Data

Our Mission“To share and grow the world’s

knowledge”

• Millions of questions & answers

• Millions of users

• Thousands of topics

• ...

Core Product & Quality

Our Product Teams

Distribution

Lookup

Demand

What we care about

Quality

Relevance

Data@Quora

Lots of data relations

Questions Answers

Users Users

TopicsUpvotes/ Downvotes

FOLLOW

ENDORSE

WRITE

HAVE

UPVOTE/DOWNVOTEWANT ANSWERS

FOLLOW

Complex network propagation effects

Importance of topics & semantics

Machine Learning@Quora

Ranking - Answer rankingWhat is a good Quora answer?

• truthful

• reusable

• provides explanation

• well formatted

• ...

Ranking - Answer rankingHow are those dimensions translated

into features?

• Features that relate to the text

quality itself

• Interaction features

(upvotes/downvotes, clicks,

comments…)

• User features (e.g. expertise in topic)

Ranking - Feed• Personalized learning-to-rank

approach

• Goal: Present most interesting stories

for a user at a given time

• Interesting = topical relevance +

social relevance + timeliness

• Stories = questions + answers

Ranking - Feed• Features

• Quality of question/answer

• Topics the user is interested on/

knows about

• Users the user is following

• What is trending/popular

• …

• Different temporal windows

• Multi-stage solution with different

“streams”

Recommendations - TopicsGoal: Recommend new topics for the

user to follow

• Based on

• Other topics followed

• Users followed

• User interactions

• Topic-related features

• ...

Recommendations - UsersGoal: Recommend new users to follow

• Based on:

• Other users followed

• Topics followed

• User interactions

• User-related features

• ...

Related Questions• Given interest in question A (source) what other

questions will be interesting?

• Not only about similarity, but also “interestingness”

• Features such as:

• Textual

• Co-visit

• Topics

• …

• Important for logged-out use case

Duplicate Questions• Important issue for Quora

• Want to make sure we don’t disperse

knowledge to the same question

• Solution: binary classifier trained with

labelled data

• Features

• Textual vector space models

• Usage-based features

• ...

User Trust/Expertise InferenceGoal: Infer user’s trustworthiness in relation

to a given topic

• We take into account:

• Answers written on topic

• Upvotes/downvotes received

• Endorsements

• ...

• Trust/expertise propagates through the network

• Must be taken into account by other algorithms

Trending TopicsGoal: Highlight current events that are

interesting for the user

• We take into account:

• Global “Trendiness”

• Social “Trendiness”

• User’s interest

• ...

• Trending topics are a great discovery mechanism

Spam Detection/Moderation• Very important for Quora to keep quality of

content

• Pure manual approaches do not scale

• Hard to get algorithms 100% right

• ML algorithms detect content/user issues

• Output of the algorithms feed manually

curated moderation queues

Content Creation Prediction• Quora’s algorithms not only optimize for

probability of reading

• Important to predict probability of a user

answering a question

• Parts of our system completely rely on

that prediction

• E.g. A2A (ask to answer) suggestions

Models● Logistic Regression

● Elastic Nets

● Gradient Boosted Decision

Trees

● Random Forests

● Neural Networks

● LambdaMART

● Matrix Factorization

● LDA

● ...

Conclusions

• At Quora we have not only Big, but also “rich” data

• Our algorithms need to understand and optimize complex aspects

such as quality, interestingness, or user expertise

• We believe ML will be one of the keys to our success

• We have many interesting problems, and many unsolved challenges