Upload
manuel-blechschmidt
View
1.709
Download
0
Embed Size (px)
Citation preview
An architecture for evaluating recommender systems in real world scenarios
Master Thesis Manuel Blechschmidt 2011
SupervisorProf. Dr. Christoph MeinelM.Sc. Rehab Alnemr
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
2
Christmas 2009 ...
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
3
Agenda
■ Motivation and Current Research■ Solution□ Use Cases & Requirements□ Wireframes□ Implementation
■ Related Work■ Conclusion
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
4
Motivation and Current Research
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
7
Motivation
■ The choice overload problem is well known in psychology
□ It is necessary to do a preselection for the customer
■ Recommender systems are already very successful to decrease the choice overload problem in some domains
□ Product-to-Product Recommendation Amazon.com→□ Movie Recommendation NetFlix→
■ Algorithms already produce great results
■ Already research in soft factores like: Diversity, Serendepity, Trust, Explanations
not a lot of emprical studies how these influences customers → → no cross domain data sets not a lot of business intereset integration→
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
8
Current Algorithms and Developments
■ Matrix Factorization (best RMSE 0.855 for NetFlix Dataset)
□ SVD
□ SVD++ R.M.Bell, Y. Koren, and C. Volinsky
□ TimeSVD++ R.M.Bell, Y. Koren, and C. Volinsky
■ Collaborative Filtering
□ Item based
□ User based
■ Performance gains
□ ALS1 István Pilászy, Dávid Zibriczky, Domonkos Tikk
■ Some of the algorithms already implemented in a distributed manner Mahout, MyMedia
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
9
Empirical Studies
■ Current empirical studies (RecSys 2010)□ Understanding Choice Overload in Recommender Systems
174 participants□ Eye-Tracking Product Recommendersʼ Usage
18 participants□ Recommender Algorithms in Activity Motivating Games
180 participants□ Group-Based Recipe Recommendations: Analysis of Data Aggregation
Strategies170 participants
□ A User-Centric Evaluation Framework of Recommender Systems807 participants
□ Information Overload and Usage of Recommendations466 participants
□ ...
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
10
Current Problems
■ Not a lot of big empirical studies how recommender quality influence consumer behavior especially
□ Acurarcy
□ Familiarity
□ Serendipity
□ Attractiveness
□ Enjoyability
□ Novelty
□ Diversity
□ Context Compatibility
■ Taken from A User-Centric Evaluation Framework of Recommender Systems
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
11
Evaluating in real world
■ Most of the academia persons do not know enough persons which are willing to test the algorithms. Therefore the following things are difficult:
□ Evaluating User Interfaces
□ Evaluating Maintenance
□ Evaluating Scalibility
□ Evaluating Performance
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
13
Master Thesis
■ Building and maintaining an evaluation platform for recommender systems in real world scenarios
■ Maintenance challenges in running a recommender system
■ Empirical study about user behavior
□ Brand loyalty
□ Pricing
□ Timing
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
14
Solution: Use Cases
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
15
Roles
■ 5 Roles with different point of views and different interests and goals
■ The roles are describeded with description and goals
■ Example:
□ Provider
□ A provider is a legal personality which has as primary goal to optimize a particular objective. In an economic context this is most of the time a business goal like raise profit or optimize conversion rates. …
□ Goals:– optimizing an objective– get forecasts– ensure privacy of his data
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
16
Use Cases and Requirements
■ Use Cases and Requirements are described based on IEEE 830
■ A use case is defined by:
□ Id
□ Name
□ Summary
□ Roles
□ Preconditions
□ Postconditions
□ Wireframes
□ More optional attributes
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
17
Use Case Example C1 Design User Interaction
■ Id: C1 Name: Design User Interaction
■ Summary: When a user interaction should be run like a newsletter or an item-to-item recommendation the consultant has to do the following steps: …
■ Roles: Consultant
■ Preconditions□ User is logged in
□ User has the Consultant role
□ At least one user interaction is implemented
□ At least one provider is associated with the consultant
□ The provider has the necessary data which is needed for the user interaction
■ Postconditions□ Provider received an email for approving the user interaction
□ User interaction is created in the system
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
18
C1 Design User Interaction
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
19
C1 Design User Interaction
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
20
C1 Design User Interaction
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
21
C1 Design User Interaction
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
22
Implemented Architecture
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
23
Logical Modularization
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
24
Survey Module Entities
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
25
Survey Module Services
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
27
Implemented User Interaction chocStore
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
29
Related Work: Competition
■ NetFlix Grand Prize 2006 – 2009
□ 1.000.000 $ to make CineMatch 10% better
□ Lots research of papers
■ KDD Cup 2011 Recommending Music Itemsbased on the Yahoo! Music Dataset
■ ECML/PKDD’2007 DISCOVERY CHALLENGE
□ User 1 User’s behaviour prediction
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
30
Related Work: Platforms
■ GroupLens Research of University of Minnesota
□ MovieLens 1997 http://movielens.umn.edu/
■ RichRelevance RecLab 2011
□ RecLab: A System For eCommerce Recommender Research with Real Data, Context and Feedback
■ Knowledge and Data Engineering Group of Uni Kassel
□ 2006 BibSonomy is a system for sharing bookmarks and lists of literature.
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
31
Further Research
■ Implement more user interactions
□ Item-to-Item recommender
■ Prove that the platform is scalable
■ Run the platform for a long time and evaluate usage
■ Integrate more companies
■ Promote plattform in science and economics
■ Take part at research projects together with companies
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
33
Conclusion
■ An enterprise ready platform was defined and implemented
■ Companies already applied for using
■ One example user interaction was implemented
□ chocStore
■ Statistical test can be applied to the data to give scientific results
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
34
Questions
Questions?
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
35
Backup: What is a recommender?