Recommender Systems: User Experience and …files.grouplens.org/papers/JK-GL-Summer-2005.pdfTitle Microsoft PowerPoint - JK-GL-Summer-2005.ppt [Read-Only] Author gladmin Created Date

Recommender Systems CSCW 2000

Copyright 2000 John Riedl and Joseph A. Konstan December 3, 2000

Summer 2005 1

Recommender Systems:User Experience and System Issues

Joseph A. KonstanUniversity of Minnesota

[email protected]://www.grouplens.org

Summer 2005 2

About me …

Professor of Computer Science & Engineering, Univ. of Minnesota

Ph.D. (1993) from U.C. BerkeleyGUI toolkit architecture

Teaching Interests: HCI, GUI ToolsResearch Interests: General HCI, and ...

Collaborative Information FilteringMultimedia Authoring and SystemsVisualization and Information ManagementMedical/Health Applications and their Delivery



Summer 2005 3

A Quick Introduction

What are recommender systems?Tools to help identify worthwhile stuff

Filtering interfacesE-mail filters, clipping services

Recommendation interfacesSuggestion lists, “top-n,” offers and promotions

Prediction interfacesEvaluate candidates, predicted ratings

Summer 2005 4

Scope of Recommenders

Purely Editorial Recommenders

Content Filtering Recommenders

Collaborative Filtering Recommenders

Hybrid Recommenders



Summer 2005 5

Wide Range of Algorithms

Simple Keyword Vector Matches

Pure Nearest-Neighbor Collaborative Filtering

Machine Learning on Content or Ratings

Summer 2005 6

Classic Collaborative Filtering

• MovieLens*• K-nearest neighbor algorithm• Model-free, memory-based

implementation• Intuitive application, supports typical

interfaces*Note – newest releases use updated architecture/algorithm



Summer 2005 7

CF Classic

C.F. Engine

Ratings Correlations

Summer 2005 8

Submit Ratings

C.F. Engine


ratings



Summer 2005 9

Store Ratings

C.F. Engine


ratings

Summer 2005 10

Compute Correlations

C.F. Engine


pairwise corr.



Summer 2005 11

Request Recommendations

C.F. Engine


request

Summer 2005 12

Identify Neighbors

C.F. Engine


find good …

Neighborhood



Summer 2005 13

Select Items; Predict Ratings

C.F. Engine

Ratings CorrelationsNeighborhood

predictionsrecommendations

Summer 2005 14

Understanding the Computation

Hoop Dreams

Star Wars

Pretty Woman

Titanic Blimp Rocky XV

Joe D A B D ? ? John A F D F Susan A A A A A A Pat D A C Jean A C A C A Ben F A F Nathan D A A



Summer 2005 15

Hoop Dreams

Star Wars

Pretty Woman




Summer 2005 16

Hoop Dreams

Star Wars

Pretty Woman






Summer 2005 17

Hoop Dreams

Star Wars

Pretty Woman




Summer 2005 18

Hoop Dreams

Star Wars

Pretty Woman






Summer 2005 19

Hoop Dreams

Star Wars

Pretty Woman




Summer 2005 20

Hoop Dreams

Star Wars

Pretty Woman






Summer 2005 21

ML-home

Summer 2005 22

ML-comedy



Summer 2005 23

ML-clist

Summer 2005 24

ML-rate



Summer 2005 25

ML-search

Summer 2005 26

ML-slist



Summer 2005 27

ML-buddies

Summer 2005 28

Talk Roadmap

Introduction• Algorithms• Influencing Users• Current Research



Summer 2005 29

Collaborative Filtering Algorithms

Non-Personalized Summary Statistics

K-Nearest Neighboruser-useritem-item

Dimensionality ReductionContent + Collaborative FilteringGraph TechniquesClusteringClassifier Learning

Summer 2005 30

Item-Item Collaborative Filtering

I

II

II I II

I

I

I

I

I

I

I

I

I



Summer 2005 31


I

II

II I II

I

I

I

I

I

I

I

I

I

Summer 2005 32


I

II

II I II

I

I

I

I

I

I

I

I

I



Summer 2005 33

Item Similarities 1 2 3 i n-1 n

12

u

mm-1

j

R-

R -

R R

R R

R R

si,j=?

Used for similarity computation

Summer 2005 34

12

u

m

2nd 1st 3rd 5th4th

5 closest neighbors

R R R R-

u R R R Ri1 2 3 i-1 m-1 m

si,1

si,3

si,i-1

si,m

-

pred

ictio

n

weighted sum regression-based

Raw scoresfor predictiongeneration

Approximationbased on linearregression

Target item

ItemItem--Item Matrix Item Matrix FormulationFormulation



Summer 2005 35

Item-Item Discussion

Good quality, in sparse situationsPromising for incremental model

buildingSmall quality degradationBig performance gain

Summer 2005 36


Non-Personalized Summary StatisticsK-Nearest Neighbor

Dimensionality ReductionSingular Value DecompositionFactor Analysis

Content + Collaborative FilteringGraph TechniquesClusteringClassifier Learning



Summer 2005 37

Dimensionality Reduction

Latent Semantic IndexingUsed by the IR communityWorked well with the vector space modelUsed Singular Value Decomposition (SVD)

Main IdeaTerm-document matching in feature spaceCaptures latent associationReduced space is less-noisy

Summer 2005 38

SVD: Mathematical Background

=R

m X n

U

m X r

S

r X r

V’

r X n

Sk

k X k

Uk

m X k

Vk’

k X n

The reconstructed matrix Rk = Uk.Sk.Vk’ is the closest rank-k matrix to the original matrix R.

Rk



Summer 2005 39

SVD for Collaborative Filtering

. 2. DirectPredictionm x n

1. Low dimensional representation O(m+n) storage requirement

m x k

k x n

Summer 2005 40

Singular Value Decomposition

Reduce dimensionality of problemResults in small, fast modelRicher Neighbor Network

Incremental UpdateFolding inModel Update



Summer 2005 41


Non-Personalized Summary StatisticsK-Nearest NeighborDimensionality Reduction

Content + Collaborative FilteringGraph Techniques

Horting: Navigate Similarity GraphClusteringClassifier Learning

Rule-Induction LearningBayesian Belief Networks

Summer 2005 42

Talk Roadmap

IntroductionAlgorithms

• Influencing UsersCosley et al, CHI 2003

• Current Research



Summer 2005 43

Does Seeing Predictions Affect User Ratings?

RERATE: Ask 212 users to rate 40 movies10 with no shown prediction30 with shown predictions (random order):10 accurate, 10 up a star, 10 down a star

Compare ratings to accurate predictions“Prediction” is user’s original ratingHypothesis: users rate in the direction of the shown prediction

Summer 2005 44

The Study



Summer 2005 45

Seeing Matters

0%

20%

40%

60%

80%

Not show n Show n

Prediction shown?

Rat

ings

%

Below At Above

Summer 2005 46

Accuracy Matters

0%

20%

40%

60%

80%

Down Accurate Up

Prediction manipulation

Rat

ings

%

Below At Above



Summer 2005 47

Domino Effects?

The power to manipulate?

Summer 2005 48

Rated, Unrated, Doesn’t Matter

Recap of RERATE effects:Showing prediction changed 8% of ratingsAltering shown prediction changed 12%

Similar experiment, UNRATED movies137 experimental users, 1599 ratingsShowing prediction changed 8% of ratingsAltering shown prediction changed 14%



Summer 2005 49

But Users Notice!

Users are often insensitive…UNRATED part 2: satisfaction survey

Control group: only accurate predictionsExperimental predictions accurate, useful?ML predictions overall accurate, useful?

Manipulated preds less well likedSurprise: 24 bad = MovieLens worse!

Summer 2005 50

Talk Roadmap

IntroductionAlgorithmsInfluencing Users

• Current Research



Summer 2005 51

Current Research Themes

• Beyond Accuracy: Metrics and Algorithms• New User “Orientation”• Influence and Shilling• Eliciting Participation in On-Line

Communities• Reinventing Conversation• Beyond Entertainment: Recommending

Research Papers

Summer 2005 52

Beyond Accuracy

What affects user satisfaction?NoveltyConfirmabilityDiversityValue

How to measure?Custom tuned algorithmsHerlocker et al., ACM TOIS 1/2004



Summer 2005 53

New User “Orientation”

How do we start new users?Interface (McNee et al., UM’2003)

System-initiatedUser-initiatedMixed

Content (Rashid et al., IUI 2002)PopularEntropyMixPredict Seen

Summer 2005 54

Influence and Shilling

Influence (Rashid et al., SIAM Data Mining 2005)Who has it?How much?Measurements?

Shilling (Lam and Riedl, WWW 2004)AttacksDefensesAlgorithm Differences



Summer 2005 55

Eliciting Participation: An Initial Study

A study of participation in discussions with two factors controlled

Similarity of tastesAwareness of own uniqueness

ResultsDissimilarity increased contributionAwareness of own uniqueness increased contributionActive discussants were not highly-active ratersParticipants rated more than a control group

Summer 2005 56

Motivational Follow-Up

Class projects at CMU used an e-mail campaign to elicit ratings to discover:

Making users aware of their uniqueness increased ratingGiving users specific, achievable goals increased ratingBut…

Reminding users of their self-benefit or benefit to others actually decreased the number of ratings!



Summer 2005 57

Other Projects

Self-Maintaining CommunitiesWhat happens when you let the masses maintain the database?

Social PreferenceUsing economic models to study user behavior

Summer 2005 58

Reinventing Conversation

Goal: More comprehensive engagementFull-factorial design around entities, people, comments/discussion entries



Summer 2005 59

Summer 2005 60



Summer 2005 61

Summer 2005 62



Summer 2005 63

Recommending Research Papers

Using Citation Webs• For a full paper, we can recommend

citationsA paper “rates” the papers it citesEvery paper has ratings in the system

• Other citation web mappings are possible, but many are have problems

Summer 2005 64

Results Thus Far

Off-line studies showed promiseOn-line study showed different algorithms met

different needsGeneral positive attitude from usersTypically one or two useful recommendations in a set of five

That’s enough to be useful

Specific value for hybrid content/collabalgorithsms

Current work: ACM Digital Library



Summer 2005 65

Summer 2005 66



Summer 2005 67

Summer 2005 68



Summer 2005 69

Summer 2005 70

Directions

Application-Focused ResearchAwareness servicePaper and proposal-writing supportFind people (reviewers/experts)Overview of a field



Summer 2005 71

Conclusions

From humble origins …

Substantial algorithmic researchHCI and online community researchImportant applicationsCommercial deployment

Summer 2005 72

Acknowledgements

• This work is being supported by grants from the National Science Foundation, and by grants from Net Perceptions, Inc.

• Many people have contributed ideas, time, and energy to this project.



Summer 2005 73

Recommender Systems:User Experience and System Issues

Joseph A. KonstanUniversity of Minnesota

[email protected]://www.grouplens.org

Documents

Recommender Systems: User Experience and …files.grouplens.org/papers/JK-GL-Summer-2005.pdfTitle Microsoft PowerPoint - JK-GL-Summer-2005.ppt [Read-Only] Author gladmin Created Date