Institut de Recherche en Informatique de Toulouse (IRIT) - UMR 5505

Bridging the gap between users and systems

27/10/11

Laurent CANDILLIER – Max CHEVALIER – Damien DUDOGNON – Josiane MOTHE

Diversity in recommender systems How to recommend documents for a visited one

Maximizing the chances of retrieving at least one relevant document per user [Santos et al., 2010]

Cover a large range of users’ interests

Context

Blog platform

Unknown user => no profile

Diversity of users, diversity of their expectations

27/10/11 2 Candillier L. – Chevalier M. – Dudognon D. – Mothe M.

Diversity in recommender systems How to recommend documents for a visited one

Maximizing the chances of retrieving at least one relevant document per user [Santos et al., 2010]

Cover a large range of users’ interests

Context

Blog platform

Unknown user => no profile

Diversity of users, diversity of their expectations

=> Diversify the recommendations

27/10/11 3 Candillier L. – Chevalier M. – Dudognon D. – Mothe M.

What is diversity? Definitions from the literature

Topicality

Related to a particular topic [Xu and Chen, 2006]

Diversity

Topical diversity

Extrinsic: solve ambiguity [Radlinski et al., 2009]

Intrinsic: avoid redundancy [Clarke et al., 2008]

Serendipity

Attractive and surprising documents [Herlocker et al., 2004]

27/10/11 4 Candillier L. – Chevalier M. – Dudognon D. – Mothe M.

Approaches to diversify IR results Topical diversity

Clustering

Identify aspects

Reorder depending on the aspects covered

Examples

K-Means [Bi et al., 2009]

Hierarchical Clustering [Meij et al., 2010]

27/10/11 5 Candillier L. – Chevalier M. – Dudognon D. – Mothe M.

Approaches to diversify IR results Topical diversity

Sliding Windows

Reorder the retrieved documents

Select documents using metrics

Similarity with the visited document

Similarity with the current recommended document list

Examples

MMR [Carbonell and Goldstein, 1998]

Intra-list similarity [Ziegler et al., 2005]

27/10/11 6 Candillier L. – Chevalier M. – Dudognon D. – Mothe M.

Approaches to diversify IR results Serendipity

Alternative to topical diversity

Similarity not only based on the content

Examples

Organizational similarity [Cabanac et al., 2007]

Temporal diversity [Lathia et al., 2010]

27/10/11 7 Candillier L. – Chevalier M. – Dudognon D. – Mothe M.

Analysis of the TREC Web 2009 results

Hypothesis

Diversity of approaches

No one approach for all users’ needs

Approaches are complementary

Valuable to combine them

Goals

Analyse results obtained with approaches having

Same goal

Similar performances

=> To identify if diversity exists

27/10/11 8 Candillier L. – Chevalier M. – Dudognon D. – Mothe M.

Analysis of the TREC Web 2009 results Experimental framework

Reference IR corpus (TREC Web 2009)

Two IR contexts

Adhoc task

Diversity task

Compare results (runs) of the 4 best approaches of each task

Similar performances according to IR metrics

MAP for adhoc task

NDCG for diversity task

Overlap for each pair of runs underlying diversity

27/10/11 9 Candillier L. – Chevalier M. – Dudognon D. – Mothe M.

Analysis of the TREC Web 2009 results

Adhoc Task

Top 10 documents

Overlap: 22.4%

Precision: 0.384

Overlap max < 30%

27/10/11 10 Candillier L. – Chevalier M. – Dudognon D. – Mothe M.

Analysis of the TREC Web 2009 results

Diversity Task

Top 10 documents

Overlap: 6.3%

Overlap max < 15%

27/10/11 11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M.

Analysis of the TREC Web 2009 results Conclusions

Two distinct approaches are unlikely to return the same (relevant) documents Low average overlap

Diversity of approaches No approach significantly better than others

A combination can be valuable

TREC tasks focused on topicality and topical diversity Can’t be used to evaluate other types of diversity

Users’ study necessary [Hayes et al., 2002]

27/10/11 12 Candillier L. – Chevalier M. – Dudognon D. – Mothe M.

Users’ Study Our intuitions

Most of the time, users want topicality

Get focused information

Sometime, they want diversity

Topical diversity

Enlarge the subject

Serendipity

Discover new information

27/10/11 13 Candillier L. – Chevalier M. – Dudognon D. – Mothe M.

Users’ Study Goals

Verify our intuitions

Prove that diversified recommendations answer a larger range of users’ needs

Context of experimentation

34 students in M. Sc. (Management field)

Blog post recommendations

27/10/11 14 Candillier L. – Chevalier M. – Dudognon D. – Mothe M.

Users’ Study Experimental Framework

Select a document

27/10/11 15 Candillier L. – Chevalier M. – Dudognon D. – Mothe M.

Users’ Study Experimental Framework

Read the selected document

27/10/11 16 Candillier L. – Chevalier M. – Dudognon D. – Mothe M.

Users’ Study Experimental Framework

Compute the recommendation lists

27/10/11 17 Candillier L. – Chevalier M. – Dudognon D. – Mothe M.

Approach 1

Approach 2

Approach 3

Approach 4

Approach 5

List 1 (random)

List 2 (fused)

Users’ Study Experimental Framework

Compute the recommendation lists

27/10/11 18 Candillier L. – Chevalier M. – Dudognon D. – Mothe M.

List 1 (random)

List 2 (fused)

Approach 1

Approach 2

Approach 3

Approach 4

Approach 5

Users’ Study Experimental Framework

Compute the recommendation lists

27/10/11 19 Candillier L. – Chevalier M. – Dudognon D. – Mothe M.

List 1 (random)

List 2 (fused)

Approach 1

Approach 2

Approach 3

Approach 4

Approach 5

Users’ Study Experimental Framework

Compute the recommendation lists

27/10/11 20 Candillier L. – Chevalier M. – Dudognon D. – Mothe M.

List 1 (random)

List 2 (fused)

Approach 1

Approach 2

Approach 3

Approach 4

Approach 5

Users’ Study Experimental Framework

Compute the recommendation lists

27/10/11 21 Candillier L. – Chevalier M. – Dudognon D. – Mothe M.

List 1 (random)

List 2 (fused)

Approach 1

Approach 2

Approach 3

Approach 4

Approach 5

Users’ Study Experimental Framework

Present recommendation lists for assessment

27/10/11 22 Candillier L. – Chevalier M. – Dudognon D. – Mothe M.

Which list best meets your needs?

Users’ Study Experimental Framework

Present recommendation lists for assessment

27/10/11 23 Candillier L. – Chevalier M. – Dudognon D. – Mothe M.

Which list is the most diversified?

Users’ Study Experimental Framework

Assessment of all documents

27/10/11 24 Candillier L. – Chevalier M. – Dudognon D. – Mothe M.

Topicality

Users’ Study Approaches used

searchsim Vector-space model Document title as query

mlt Apache Solr MoreLikeThis module Document content as query

27/10/11 25 Candillier L. – Chevalier M. – Dudognon D. – Mothe M.

Topical diversity

Users’ Study Approaches used

kmeans K-means classification One element per cluster

27/10/11 26 Candillier L. – Chevalier M. – Dudognon D. – Mothe M.

Serendipity

Users’ Study Approaches used

blogart Random selection from the same blog

topcateg Popular documents in the same category

27/10/11 27 Candillier L. – Chevalier M. – Dudognon D. – Mothe M.

Users’ Study Approaches used

Same analysis than TREC experiments

Same results

Overlap is low (< 10%)

=> High diversity

27/10/11 28 Candillier L. – Chevalier M. – Dudognon D. – Mothe M.

Users’ Study Results

Distribution of relevant documents

35% 0%

65%

blogart fused

52.5% 26.2%

21.3%

kmeans fused

54.7% 12.5%

32.8%

mlt fused

52.4% 8.7%

38.9%

searchsim fused

8.8% 0%

91.2%

topcateg fused

27/10/11 29 Candillier L. – Chevalier M. – Dudognon D. – Mothe M.

Users’ Study Results

Distribution of relevant documents

35% 0%

65% 52.5% 26.2%

21.3%

kmeans fused

54.7% 12.5%

32.8%

mlt fused

52.4% 8.7%

38.9%

searchsim fused

8.8% 0%

91.2%

27/10/11 30 Candillier L. – Chevalier M. – Dudognon D. – Mothe M.

Users’ Study Results

Distribution of relevant documents

35% 0%

65%

blogart fused

52.5% 26.2%

21.3%

54.7% 12.5%

32.8%

52.4% 8.7%

38.9% 8.8% 0%

91.2%

topcateg fused

27/10/11 31 Candillier L. – Chevalier M. – Dudognon D. – Mothe M.

Users’ Study Results

Distribution of relevant documents

Relevant mainly retrieved by topical approaches

But at least 20% are retrieved only by fused

Fused matches with a larger range of needs

27/10/11 32 Candillier L. – Chevalier M. – Dudognon D. – Mothe M.

Conclusions and future work Conclusions

Diversity of users’ expectations

No one approach to rule them all

A diversity of approaches

Complementary

Fused

Diversity helps RS to fit more users’ needs

27/10/11 33 Candillier L. – Chevalier M. – Dudognon D. – Mothe M.

Conclusions and future work Future work

Real scale experiment

OverBlog platform

Renew the user survey

More users (international call for participation)

Avoid revealed biases

e.g. More detailed form

=> Deeper analysis

27/10/11 34 Candillier L. – Chevalier M. – Dudognon D. – Mothe M.

Conclusions and future work Future work

Improve the model

Refining the fusing process

Adding a learning process to weight each approach

For every visited document

Find the proportion of documents coming from each approach (log analysis)

Better match with the real users’ needs

27/10/11 35 Candillier L. – Chevalier M. – Dudognon D. – Mothe M.

27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 36

Questions ?

Thank you for your attention

W. Bi, X. Yu, Y. Liu, F. Guan, Z. Peng, H. Xu, and X. Cheng, “ICTNET at Web Track 2009 diversity task”, Text REtrieval Conf., 2009 G. Cabanac, M. Chevalier, C. Chrisment, and C. Julien, “An Original Usage-based Metrics for Building a Unified View of Corporate Documents”, Inter. Conf. on Database and Expert Systems Applications, 2007, LNCS V. 4653, 2007, pp. 202–212 J. Carbonell and J. Goldstein, “The use of MMR, diversity-based reranking for reordering documents and producing summaries”, ACM Conf. on Research and Development in Information Retrieval, 1998, pp. 335-336 C. L. A. Clarke, M. Kolla, G. V. Cormack, O. Vechtomova, A. Ashkan, S. Buttcher, and I.n MacKinnon, “Novelty and Diversity in Information Retrieval Evaluation”, ACM Conf. on Research and Development in Information Retrieval, 2008, pp. 659-666 C. Hayes, P. Massa, P. Avesani, and P. Cunningham, « An online evaluation framework for recommender systems», Workshop on Personalization and Recommendation in E-Commerce, 2002 J. L. Herlocker, J. A. Konstan, L. G. Terveen, and J. T. Riedl, “Evaluating Collaborative Filtering Recommender Systems”, ACM Trans. Information Systems, 22(1), 2004, pp. 5-53 N. Lathia, S. Hailes, L. Capra, and X. Amatriain, “Temporal diversity in recommender systems”, ACM Conf. on Research and Development in Information Retrieval, 2010, pp. 210-217 E. Meij, J. He, W. Weerkamp, and M. de Rijke, “Topical Diversity and Relevance Feedback”, Text REtrieval Conf., 2010 F. Radlinski, P. N. Bennett, B. Carterette, and T. Joachims. “Redundancy, diversity and interdependent document relevance”, SIGIR Forum, 43(2), 2009, pp. 46–52 R. L. T. Santos, C. Macdonald, and I. Ounis, “Selectively Diversifying Web Search Results”, ACM Inter. Conf. on Information and Knowledge Management, 2010 Y. C. Xu and Z. Chen, “Relevance judgment: What do information users consider beyond topicality”, Journal of the American Society for Information Science and Technology, 57(7), 2006, pp. 961–973 C. Ziegler, S. McNee, J. A. Konstan, and G. Lausen, “Improving recommendation lists through topic diversification”, Inter. Conf. on World Wide Web, 2005, pp. 22–32

References

27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 37