From a toolkit of recommendation algorithms into a real business:
13.09.2012.
the Gravity R&D experience
The kick-start
13.09.2012.2 From a toolkit of recommendation algorithms into a real business
Facing with real needs
What we had• rating prediction algorithms
• coded in various languages
• blending mechanism
• accuracy oriented
What clients wanted• recommendations that
bring revenue
• robustness
• low response time
• easy integration
• reporting
13.09.2012.3 From a toolkit of recommendation algorithms into a real business
What we do?
13.09.2012.4 From a toolkit of recommendation algorithms into a real business
users
content of service provider
recommender
Explicit vs implicit feedback
No ratings but interactions
sparse vs. dense matrix
requires different learning
13.09.2012.5 From a toolkit of recommendation algorithms into a real business
Increase revenue: A/B tests
against the original solution
internally
13.09.2012.6 From a toolkit of recommendation algorithms into a real business
Robustness
13.09.2012.7 From a toolkit of recommendation algorithms into a real business
IMPRESSApplication Server #1
Reco LAN
IMPRESSApplication Server #2
Reporting Subsystem
Platform OSS/BSS
Database #1 Database #2
Backend LAN
SOAP
CSV over FTP
Firewall
IMPRESS Frontendweb server #1
TV Service LAN
End users
Management LAN
Nagios MonitoringAggregator
IMPRESS Frontendweb server #2
Load Balancer
HP OpenView
SQL SQL
HTTP/ SQL
HTTP/ SQL
HTTP HTTP(S)
SNMP
Time requirements
• Response time: few ms (max 200)
• Training time: maximum few hours
• regular retraining
• incremental training
• Newsletters:
• nightly batch run
13.09.2012.8 From a toolkit of recommendation algorithms into a real business
Productization
13.09.2012.9 From a toolkit of recommendation algorithms into a real business
IMPRESSfor
IPTV, CATV and satellite
Recommends
Personally Relevant
Linear TV, VOD, catch-up TV and more
Gravity personalization platform
AD•APTfor
ad networks and ad server providers
Recommends Personally Relevant
ads
RECOfor
e-commerce
Recommends Personally Relevant
products & services
The 5% question – Importance of UI
Francisco Martin (Strands): „the algorithm is only 5% in the success of the recommender system”
• placement below or above the fold scrolling easy to recognize floating in
• title not misleading explanation like
• widget carrousel static
13.09.2012.10 From a toolkit of recommendation algorithms into a real business
Recommendation scenario
13.09.2012.11 From a toolkit of recommendation algorithms into a real business
Item2Item recommendation
logic: the ad’s profile will be
matched to the profile model of
available ads
Marketing channels
13.09.2012.12 From a toolkit of recommendation algorithms into a real business
Changing the order of two boxes: 25% CTR increase
Cannibalization
• Goal: increase user engagement
• Measurements
• average visit length
• average page views
• Effect of accurate recommendations:
• use of listing page ↓
• use of item page ↑
• Overall page view: remains the same
• Secondary measurements
• Contacting
• CTR increase
13.09.2012.13 From a toolkit of recommendation algorithms into a real business
?
Evolution: increased user engagement
• not a cold start problem
• parameter optimization and user engagement
13.09.2012.14 From a toolkit of recommendation algorithms into a real business
KPIs – may change during testing
13.09.2012.15 From a toolkit of recommendation algorithms into a real business
Complete personalization: coupon-world
• Newsletter (daily + occassionally)
• Ranking all offers on the website
• top1 item
• category preferences
• user metadata (gender, age, …)
• user category preferences (seldom given)
• item metadata
• context
• customer vs. vendor
13.09.2012.16 From a toolkit of recommendation algorithms into a real business
Business rules – driving/overriding ranking
13.09.2012.17 From a toolkit of recommendation algorithms into a real business
Contexts
13.09.2012.18 From a toolkit of recommendation algorithms into a real business
Context at TV program recommendation
• TV (EPG program & video-on-demand) explicit and implicit identification of the user in the household time-dependent recommendation
13.09.2012.19 From a toolkit of recommendation algorithms into a real business
Some results (offline)
13.09.2012.20 From a toolkit of recommendation algorithms into a real business
Recall@20 MAP@20 Recall@20 MAP@20Grocery 64,31% 137,96% 89,99% 199,82%TV1 14,77% 43,80% 28,66% 85,33%TV2 -7,94% 10,69% 7,77% 14,15%LastFM 96,10% 116,54% 40,98% 254,62%
Dataset
Improvement using seasoniTALS iTALSx
Recall@20 MAP@20 Recall@20 MAP@20Grocery 84,48% 104,13% 108,83% 122,24%TV1 36,15% 55,07% 26,14% 29,93%
Dataset
Improvement using SeqiTALS iTALSx
Some results (online)
Anecdotes
• Item2item recommendations – bookstore
• Placebo effect
• buyer vs. seller
13.09.2012.21 From a toolkit of recommendation algorithms into a real business
Conclusion
• Offline and online testing
• From simple to sophisticated
• Many more potential fields of application
13.09.2012.22 From a toolkit of recommendation algorithms into a real business