Can Trailers Help to Alleviate Popularity Bias in Choice-Based Preference Elicitation?

Can Trailers Help to Alleviate Popularity Bias in Choice-Based Preference

Elicitation?

Mark P. Graus

Martijn C. Willemsen

Human-Technology Interaction Group

Eindhoven University of Technology

Summary

• We wanted to see if we could make people chose less popular items in a choice-based preference elicitation recommender system by showing them trailers.

• We tested this in a between subjects user study.

• We found that after watching trailers people chose less popular items, while user experience was not negatively affected.

9/16/2016Graus, Willemsen: Can Trailers Help to Alleviate ... IntRS @

RecSys '162

Motivation9/16/2016

Graus, Willemsen: Can Trailers Help to Alleviate ... IntRS @ RecSys '16

3

Latent Feature Diversification

• Can we reduce choice overload through diversification based on the latent features of a matrix factorization model?

Willemsen, M. C., Graus, M. P., & Knijnenburg, B. P. (2016). Understanding the role of latent feature diversification on choice difficulty and satisfaction. User Modeling and User-Adapted Interaction, 1–43. http://doi.org/10.1007/s11257-016-9178-6

Latent Feature 1

Late

nt

Feat

ure

2


RecSys '164

Latent Feature Diversification Findings


RecSys '165

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

low mid high

stan

dar

diz

ed

sco

re

diversification

Perceived diversity

-1

-0.8

-0.6

-0.4

-0.2

0

0.2

low mid highst

and

ard

ize

d s

core

diversification

Expected choice difficulty

Late

nt

Feat

ure

2

Latent Feature 1

4) Iteration 2

Choice-Based Preference Elicitation

• Can we improve the user experience during cold start by having people choose between items instead of rating items?

Graus, M. P., & Willemsen, M. C. (2015). Improving the User Experience during Cold Start through Choice-Based Preference Elicitation. In Proceedings of the 9th ACM Conference on Recommender Systems - RecSys’15 (pp. 273–276). New York, New York, USA: ACM Press. http://doi.org/10.1145/2792838.2799681


RecSys '166

How does this work? Step 1

Latent Feature 1Late

nt

Feat

ure

2

Iteration 1a: Diversified choice set is calculated from a matrix factorization model (red items)

Iteration 1b: User vector (blue arrow) is moved towards chosen item (green item), items with lowest predicted rating are discarded (greyedout items)


RecSys '167

How does this work? Step 2

Iteration 2: New diversified choice set (blue items)

End of Iteration 2: with updated vector and more items discarded based on second choice (green item)


RecSys '168

Choice-Based Preference Elicitation Findings

• People are more satisfied with choice-based than rating-based interfaces

• This comes mainly because of increased popularity (items with many ratings)

But we do not want to

recommend popular

items!


RecSys '169

Satisfaction with Chosen

ItemPopularity

Difficulty

Intra List Similarity

-2.407(.381)

p<.001

-.240 (.145)

p<.1

-.479 (.111)

p<.001

-.257 (.045)

p<.001

14.00 (4.51)

p<.01

Choice-Based List

+

+

- -

+

Why do people end up with popular items?

• Our hypothesis• Users don’t know all movies, hard to judge based on

metadata alone

• People choose movies they know

• People know movies that are popular

• Choosing popular movies results in popular recommendations

• Our Solution• Provide trailers as additional information for making choices


RecSys '1610

Rationale

• In the music domain• Implicit Feedback

• Movie domain• Implicit feedback is sparse

• I (can) listen to 100s of tracks in a week, but I can’t watch 100s of movies a week (and sustain my job).

• We can approximate experiencing movies by providing trailers


RecSys '1611

Study9/16/2016


12

Choice-Based Interface with or without Trailers

• N = 71


RecSys '1613

Expected Effects

Trailers

Perceived Diversity

Informativeness

Perceived Novelty

Choice Satisfaction

System Satisfaction

Popularity of Chosen Items

- +

+

+?

-?-

-

+


RecSys '1614

Set Up

• Random assignment

• Choice Based Preference Elicitation – 9 choices of 10 items

[with/without trailers]

• Recommendation List – Top-10 Items [with/without trailers]

• Survey to measure User Experience

• Informativeness

• Perceived Diversity

• Perceived Novelty

• System Satisfaction

• Choice Satisfaction


RecSys '1615

Results9/16/2016


16

Do trailers affect the popularity of chosen items?

• Checked through repeated measures (10 choices)

• Popularity is expressed as the rank ordering by number of ratings in MovieLens dataset

• Trailers do not decrease popularity of choices• The popularity rank of the item chosen in each choice

set• The average popularity rank of all items in each choice

set


RecSys '1617

However: Relative Popularity of Choice

• average popularity rank of choice set – popularity rank of chosen item


RecSys '1618

If we look at people that actually watched trailers

• People that watch trailers are

more likely to pick less popular

movies from the lists


RecSys '1619

User Experience

Choice Satisfaction

Perceived Diversity

System Satisfaction

Informativeness

.570 (.295)p < 0.1

-.604 (.091)p < 0.01

.244 (.162)n.s.

.785 (.115)p < 0.01

-.266 (.122)p < 0.05

Trailers.611 (.256)p < 0.05

-.570 (.259)p < 0.05

-

+

-

-

+

9/16/2016

+


20

Conclusions9/16/2016


21

What we found

• Providing people with trailers does make them choose less popular items.

• No indication that the overall satisfaction is affected negatively or positively

• As opposed to initial study where popularity resulted in increased satisfaction


RecSys '1622

Limitations

• When were trailers watched?• In the preference elicitation task?

• In the decision task?

Future Work

• How do trailers affect a more standard rating interface?


RecSys '1623

Thank You

• Questions/remarks?

Mark Graus – PhD Student

Human-Technology Interaction Group

Eindhoven University of Technology

[email protected]

https://twitter.com/newmarrk

https://linkedin.com/in/markgraus

http://www.marrk.nl


RecSys '1624

mailto:[email protected]

https://twitter.com/newmarrk

https://linkedin.com/in/markgraus

http://www.marrk.nl/

Science

Can Trailers Help to Alleviate Popularity Bias in Choice-Based Preference Elicitation?