Upload
tamas-jambor
View
592
Download
1
Tags:
Embed Size (px)
DESCRIPTION
Citation preview
Goal-driven Collaborative FilteringA Directional Error Based Approach
Tamas Jambor and Jun Wang
University College London
Structure of the talk
• Background/Problem description• Goal-driven design• Experimental results• Conclusions
Collaborative filtering
• Predicting user preference
towards unknown items• Based on previously expressed preferences
Love Actually
Pulp Fiction
Crazy Heart
WhiteRibbon
Up in the Air
A SingleMan
Sophie «««« « ««««¶ «««««
Peter ««««« «««¶¶ ««¶¶¶
Jaden «««¶¶ «««¶¶ «««««
?? ?
? ? ?
? ? ?
Evaluation metrics
• Root Mean Squared Error• Netflix recommendation competition
adopted this metric• The objective function for some of the SVD
implementations is equivalent to the performance measure [Koren et al 2009]
• Criticism– Error criterion is uniform across rating scales– Is it consistent with users’ satisfactions?
))ˆ(( 2rrE
Goal-driven design
• We argue that – Measure does not always reflect user needs– Different user needs require different performance
measures
• The algorithm should be defined based on user needs– Start from the user point of view, define measure and
algorithm accordingly
Rating-prediction error offset (SVD)
Observed 1Predicted 3
Observed 5Predicted 3
Observed 3Predicted 1
Observed 3Predicted 5
5««««««¶¶¶¶ «««¶¶
31
Boundaries and the direction of error
• Taste boundary - interval between liked and disliked items
• Direction – error towards the boundary• Magnitude – whether the error crosses taste
boundary
Directional risk preference of prediction
The two dimensional weighting function
r = 1,2 r = 3 r = 4,5
p <= 2.5 w1 w2 w3
2.5<p<=3.5 w4 w5 w6
P > 3.5 w7 w8 w9
Two-stage Optimization (in General)
Learning the Directional
Errors
Learning the Recom. Model
Testing
Feedback/IR Metrics
Two-stage Optimization (An example)
Directional Errors
Modeling
Rec. Model (SVD)Evaluations
NDCG/MRR
argminq, p
w(rui qiT
ui pu )
2 ( qi2 pu
2)
Genetic algorithmNDCG as fitness function
Plug in the learned Weights in SVD Training
Genetic algorithms
• Search algorithms that work via the
process of natural selection• Start with a sample set of potential solutions (a set
of weights)• Evolve towards a set of more optimal solutions• Poor solutions tend to die out (smaller NDCG)• Better solutions remain in the population (higher
NDCG)
Experiments
• MovieLens 100k dataset• 1862 movies, 943 users• Only using ratings• Five-fold cross validation
Evaluation metrics
• Recommendation as a ranking problem• IR measures
– Normalized discounted cumulative gain (NDCG)– Mean average precision (MAP)– Mean reciprocal rank (MRR)
Results – Experiment I
SVD with weights where w7>w8>w4
r = 1,2 r = 3 r = 4,5
p <= 2.5 0.0759 0.0407 0.0264
2.5<p<=3.5 0.0837 0.1676 0.2381
p > 3.5 0.0125 0.0583 0.2966
Baseline SVD
r = 1,2 r = 3 r = 4,5
p <= 2.5 0.0517 0.0193 0.0106
2.5<p<=3.5 0.0904 0.1461 0.1391
p > 3.5 0.0299 0.1012 0.4115
Results – Experiment II
r = 1,2 r = 3 r = 4,5
p <= 2.5 w1 w2 w3
2.5<p<=3.5 w4 w5 w6
P > 3.5 w7 w8 w9
Results – Experiment II
• Genetic algorithm to find optimal weigh for sector w7,w8 and w4 (statistically significant)
Weighted Baseline
MAP 0.450 0.447
MRR 0.899 0.889
NDCG@10 0.726 0.720
NDCG@5 0.574 0.570
NDCG@3 0.450 0.447
Probability of correct prediction within sectors
Probability of predicting non-relevant items relevant
Improved user experience
• More likely to receive relevant items on their recommendation list
• Less likely that lower rated items receive higher predictions
• But it is more likely that higher rated items receive lower predictions
Conclusion
• Optimize algorithm from the user point of view• Identify directional errors• Assign risk to each direction• Approach can be changed depending on how
items are presented
Future work
• Taste boundaries might be user dependent • Directional error across items or users• Different recommender goals
Thank you.
References
• Deshpande, M., Karypis, G.: Item-based top-N recommendation algorithms. ACM Trans. Inf. Syst. 22(1) (2004)
• Herlocker, J.L., Konstan, J.A., Borchers, A., Riedl, J.: An algorithmic framework for performing collaborative filtering. In: SIGIR '99. (1999)
• Koren, Y., Bell, R., Volinsky, C.: Matrix factorization techniques for recommender systems. Computer 42(8) (2009)
• Wang, J., de Vries, A.P., Reinders, M.J.T.: Unifying user-based and item-based collaborative filtering approaches by similarity fusion. In: SIGIR '06: Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, New York, NY, ACM Press