Upload
robin-aly
View
60
Download
1
Tags:
Embed Size (px)
Citation preview
LIME workshop - WWW2013
Outline
• The search and hyperlinking task• Dataset (Videos + user input)• Ground truth creation• Evaluation procedure• Results
5/13/13
DGA workshop - July 2013, Paris
ME 2013 Search & Hyperlinking
• ME 2012 S&HL “brave new” task: – Search: retrieve known-item video segment given a
natural language description – Linking: link known-item video segment to similar
segments in the collection (blib.tv)• ME 2013 S&HL “regular” task– Search: retrieve known video segment (known-
item) given a textual query and visual cues– Linking: link user-defined anchor within the known-
item to relevant target video segments7/2/13
DGA workshop - July 2013, Paris
Terminology• Video (e.g, 2 hours)
• Interesting segment (e.g. 10 min)
• Anchor: segment for which a user requests a link (e.g., 1 min)
“I want to know more about this”• Hyperlink
• Target: relevant segment for given anchor (e.g., 5 min)7/2/13
LIME workshop - WWW2013
Dataset: Video collection
• copyright cleared broadcasts from the period of 01.04.2008 – 11.05.2008
• 1667 hours, 2323 videos• ~200 videos were rebroadcast
5/13/13
LIME workshop - WWW2013
Dataset: Query
• 28 Users- Policemen, Hair dresser, Bouncer, Sales manger, Student, Self-employed
• Two hour session:– Browse the archive– Define known-item– Define anchor
5/13/13
DGA workshop - July 2013, Paris7/2/13
Formulate Information
need
Text search Visual search
AXES PROsystem
LIME workshop - WWW2013
Dataset: outcome (1/2)
• 50 known-items<top> <itemId>item_9</itemId> <queryText>animal park, kenya marathon , wildlife reserve</queryText> <visualQueues>animals, kenya wildlife reserve,marathon</visualQueues></top>
5/13/13
LIME workshop - WWW2013
Dataset: outcome (2/2)
• 98 anchors<anchor> <anchorId>anchor_3</anchorId> <startTime>5.29</startTime> <endTime>6.03</endTime> <item> <fileName>v20080506_093000_...</fileName> </item</anchor>
5/13/13
LIME workshop - WWW2013
Ground truth creation
• Search sub-task: done (known-items)• Judgments linking sub-task:– top-10 of one run per participant (7) by users
locally at BBC– top-10 of most runs using Amazon's mechanical
turk– ~70% Agreement– Similar measurements
5/13/13
LIME workshop - WWW2013
Unifying Qrels
5/13/13
Judgment 1
Judgment 2
Result
Case1: Merge relevant Case2: Prioritize relevant over non-relevant
Evaluation: search sub-task
• Depending on jump-in point• Measures: MRR, MGAP, MSAP• Rebroadcasts not considered
5/13/13
Evaluation: linking sub-task
• Measures: P@5,10,20 and map• Resolving conflicts
Judgedsegment
Submittedsegment
Result:
Conclusions
• Task defined by real users• Innovative elements perceived as interesting• Crowdsourcing valid evaluation strategy• Experimental setup works• Provides a promising starting point for further
research