8 lessons from deploying Content

Discovery solution at Orange (France)

Dr. Ofer Weintraub

VP Innovation – Viaccess-Orca

Who we are ?

Viaccess-Orca

About

Founded – June 13 2012

350 employees

Offices: France, US, Hong-

Kong, Israel

100+ customers worldwide

Fully owned by France

Telecom

Flexible middleware platform delivering a full

array of IPTV & OTT services – including live TV,

VOD and PVR, across multiple screens

Personalized Content

Discovery platform that

recommends

the right content

TV Everywhere

CAS / DRM

Products

Quick primer

Content Discovery

RecommendationsSearch Exploration

I know what

I am looking for

The service knows

what I am looking for

I’d like to explore

with my personal guide

3 main ways to discover content

3 main ways to discover content

Auto-complete

Auto-suggest

Social-aided

Personalized

search

Smart filters

Collaborative

filtering

NLP - semantic

Social

Popular/Trending

Lists (experts,

operator, users)

RecommendationsSearch Exploration

Personal zone

Games

Trends

Deals

Friends

Gossip

Search example

Recommendations example

Explore example

How it generally Works ?

Middleware

(CMS, CRM)

Usage

data

Event Registration

Engines

Discovery

Manager

User

Profiles

Collaborative

filtering

Advanced

semantic

Filtering

Content Discovery for every service….

VOD Linear TV series

… and on any device

measuring the value of

Content Discovery system

75% of what people watch is from

some sort of recommendation.

Netflix blog April 2012

35% of Amazon sales are due to

recommendations

Venturebeat - 2006

Nice numbers but no single way to measure success…

Typical measures

Accuracy CoverageNovelty

SerendipitySatisfaction

Testing methods

Train Predict

Subjective testing

Lessons learned

Lesson 1 –have a dedicated group of real users for tests

15%

15% satisfaction gain in 1 week by

adding bots and tuning thresholds

and filtering in collaborative-filtering

Lesson 2 –avoiding the rotten

apple is more important

than getting the perfect

ones

We’ve seen dramatic jumps in

satisfaction when pruning bad

results

- Time slices

- Thresholds

- Exclude rules

- Adequate “system warming”

- External guides (e.g. popularity)

Research: The Effect of Dual NetworksProf. J. Goldenberg, Dr. G Ostreicher and S Reichman - Feb. 2011

Multiple engines (i.e. engines blend ) help overcoming the

filter- bubble effect

Overall satisfaction

(1 – 10)

Rating

( 1- 5)

Single

Network 6.01 2.72

Dual

network 8.00 3.14

Change 33% ~15%

Semantic

Social

Lesson 3 –

Research: Negotiation agents n=50 x 2 (US, India)Prof. S. Kraus and Dr. A. Hasidim - Apr. 2012 (not yet published)

Collaborative filtering is popularity biased,

users prefer novelty

Lesson 4 –

The highest correlation found in early experiments is

between novelty and purchase decision (0.55)

Which one to recommend if

score is the same ?

Avatar ?

The Pianist ?

OMG it speaks FrenchLesson 5 –

Fearabhorrence, agitation, angst, anxiety, aversi

on,

awe, chickenheartedness, cold feet,cold sw

eat, concern, consternation, cowardice,cree

ps, despair, discomposure, dismay,disquiet

ude, distress, doubt, dread,faintheartednes

s, foreboding, fright, funk, horror,jitters, mis

giving, nightmare, panic, phobia,presentime

nt, qualm, recreancy, reverence,revulsion, s

care, suspicion, terror, timidity,trembling, tre

mor, trepidation, unease, uneasiness, worry

Peur

panique, phobie, frayeur, appréhension, frisson,

épouvante, crainte, alarme, émotion, affolement

Change in words statistics, change of sources

, change in amount of reviews, change of

vocabulary, correlation to English data is not

always clear

Cold start could get really cold….Lesson 6 –

System bootstrap User cold start Content cold start

Add users

Add values

Hybrid methods

Non-personal

Implicit evidence

Questionnaire

Non-personal

Implicit evidence

Aggregated data

Laziness wins….Lesson 7 –

Privacy mattersLesson 8 –

Profile visibility

Explicit / Implicit- Channels

- Genres

- Actors

- Devices

- Subscriptions

- Black / White

lists

Enough value

when not opted-in

Still relevant

- Semantic without

history

- Popularity

- Trends

- Lists

- General CF

Provide reason

Other points to consider

Collect valuable indirect evidence

Handle “time” with care

Recommended because 3 of

your friends liked it

VOD orderVOD content endVOD ratingAdd to wish listShow movie previewChannel zapping

Channel zappingProgram recordingSetting a reminderLaunches of COMPASSExplicit inputExclude contentSearch terms

Mo

rnin

g

No

on

Evenin

g

Nigh

t

It’s an on going process

It’s getting better every month

It’s a lot of fun

Anyone?