Comparing Retrieval Effectiveness of Alternative Content Segmentation Methods for Internet Video...

Comparing Retrieval Effectiveness of Alternative Content Segmentation Methods for Internet Video Search

Comparing Retrieval Effectiveness ofAlternative Content Segmentation Methods

for Internet Video Search

Maria Eskevich1, Gareth J.F. Jones1, Martha Larson2

Christian Wartena2,3

Robin Aly4, Thijs Verschoor4, Roeland Ordelman4

1 Centre for Digital Video Processing, Centre for Next Generation LocalisationSchool of Computing, Dublin City University, Dublin, Ireland

2 Delft University of Technology, Delft, The Netherlands3 Univ. of Applied Sciences and Arts Hannover

4 University of Twente, The Netherlands

Outline

I MediaEval 2011 Rich Speech Retrieval TaskI 3 participant groups methodsI Results and examplesI ConclusionI Future Work: Brave New Task at MediaEval 2012

ediaEval 2011Rich Speech Retrieval (RSR) Task

I Task Goal:I Information to be found - combination of required

audio and visual content, and speaker’s intention

Transcript 1

6=Meaning 1 6=

Transcript 2

Meaning 2

Conventional retrieval

Transcript 1

Meaning 1

Transcript 2Meaning 2

Transcript 1 6=Meaning 1 6=

Transcript 1 =Meaning 1 6=

Speech act 1 6=

Speech act 2

Extended speech retrieval (find jump-in points)

Transcript 1 =Meaning 1 6=Speech act 1 6=

Transcript 2Meaning 2Speech act 2

Transcript 1 =Meaning 1 6=Speech act 1 6=

Transcript 2Meaning 2Speech act 2

I Data provided to task participants:I Videos from Internet video sharing platform blip.tv

(ME10WWW dataset: testset: 1727 episodes, ca. 300 hs)I Automatic Speech Recognition (ASR) transcript provided

by LIMSI and Vocapia ResearchI Metadata manually added by the uploaderI 50 user-generated short web style queries collected via

crowdsourcing, associated with following speech act types:I ’expressives’: apology (1), opinion (21)I ’assertives’: definition (17)I ’directives’: warning (6)I ’commissives’: promise (5)

I Data available for results assessment:I Time of the relevant item for the labeled speech actI Accurate transcript of the labeled speech act

I Data provided to task participants:

I Videos from Internet video sharing platform blip.tv(ME10WWW dataset: testset: 1727 episodes, ca. 300 hs)

I Automatic Speech Recognition (ASR) transcript providedby LIMSI and Vocapia Research

I Metadata manually added by the uploaderI 50 user-generated short web style queries collected via

(ME10WWW dataset: testset: 1727 episodes, ca. 300 hs)

I Automatic Speech Recognition (ASR) transcript providedby LIMSI and Vocapia Research

by LIMSI and Vocapia Research

by LIMSI and Vocapia ResearchI Metadata manually added by the uploader

I 50 user-generated short web style queries collected viacrowdsourcing, associated with following speech act types:

I ’expressives’: apology (1), opinion (21)I ’assertives’: definition (17)I ’directives’: warning (6)I ’commissives’: promise (5)

crowdsourcing, associated with following speech act types:

I ’expressives’: apology (1), opinion (21)I ’assertives’: definition (17)I ’directives’: warning (6)I ’commissives’: promise (5)

I Data available for results assessment:

I Time of the relevant item for the labeled speech actI Accurate transcript of the labeled speech act

I Data available for results assessment:I Time of the relevant item for the labeled speech act

I Accurate transcript of the labeled speech act

Evaluation Metrics

I Mean Reciprocal Rank (MRR):

RANKI Mean Generalized Average Precision (mGAP):

GAP =1

RANK. PENALTY

Evaluation Metrics

I Mean Reciprocal Rank (MRR):

RANKI Mean Generalized Average Precision (mGAP):

GAP =1

RANK. PENALTY

Approach 1: Sliding Window (SW)

I Tag and lemmatize the words

I Segmentation with sliding window:

I Tag and lemmatize the wordsI Segmentation with sliding window:

I Retrieval using BM25, BM25F - for use of metadataI Post-processing:

I Retrieval using BM25, BM25F - for use of metadata

I Post-processing:

Approach 2: Speech Segments (Sp)

I Segmentation based on silence points and changes ofspeakers

I Search engine used: PFTijah

Approach 3: Lexical cohesion (LC)

I Segmentation:I into lexically coherent segments, using 2 algorithms:

C99 and TextTilingI additional segment boundaries for silences > 0.5 sec

I SMART IR system with language modeling

Approach 3: Lexical cohesion (LC)

I Segmentation:I into lexically coherent segments, using 2 algorithms:

C99 and TextTilingI additional segment boundaries for silences > 0.5 sec

I SMART IR system with language modeling

RSR Results: MRR and mGAP

RunName WindowSize60 30 10

MRR mGAP MRR mGAP MRR mGAPSW asr sh 0.37 0.32 0.32 0.27 0.19 0.19

SW asr meta sh 0.35 0.29 0.30 0.25 0.14 0.14SW meta 0.20 0.15 0.18 0.13 0.06 0.06

Sp asr 0.34 0.27 0.27 0.22 0.16 0.16Sp asr meta 0.34 0.25 0.26 0.21 0.15 0.15

Sp meta 0.18 0.14 0.14 0.11 0.07 0.07LC asr tt 0.36 0.25 0.29 0.18 0.09 0.09

LC asr meta tt 0.39 0.28 0.30 0.20 0.14 0.14LC meta 0.18 0.11 0.09 0.07 0.03 0.03

RSR Results: MRR and mGAP

RunName WindowSize60 30 10

MRR mGAP MRR mGAP MRR mGAPSW asr sh 0.37 0.32 0.32 0.27 0.19 0.19

SW asr meta sh 0.35 0.29 0.30 0.25 0.14 0.14SW meta 0.20 0.15 0.18 0.13 0.06 0.06

Sp asr 0.34 0.27 0.27 0.22 0.16 0.16Sp asr meta 0.34 0.25 0.26 0.21 0.15 0.15

Sp meta 0.18 0.14 0.14 0.11 0.07 0.07LC asr tt 0.36 0.25 0.29 0.18 0.09 0.09

LC asr meta tt 0.39 0.28 0.30 0.20 0.14 0.14LC meta 0.18 0.11 0.09 0.07 0.03 0.03

Relationship BetweenRetrieval Effectiveness and Segmentation Methods

Segment:I 100 % Recall of the relevant contentI High Precision (30, 56 %) of the relevant contentI Topic consistency

Relationship BetweenRetrieval Effectiveness and Segmentation MethodsExample 1

Overlap of query wordswith ASR transcript or Metadata. Example 3

Segments on the same topicare retrieved in top of the list;

Use of metadata for segmentscontaining relevant content:

I decrease the rank, ifsegment > 1 topic

I does not affect the rank, ifsegment = 1 topic

Segment:I High Recall (83, 100 %)I Precision = 23 %I Several topics covered− > Use of metadata increase the rank (Rank = 1)

Segment:I High Recall (83, 100 %)I Precision = 23 %I Several topics covered

− > Use of metadata increase the rank (Rank = 1)

Segment:I High Recall (83, 100 %)I Precision = 23 %I Several topics covered− > Use of metadata increase the rank (Rank = 1)

Conclusions and Future Work

I Segmentation plays significant role in retrieving relevantcontent

I High recall and precision of the relevant content within thesegment lead to good segment ranking.

I Related metadata is useful to improve ranking of thesegment with high recall and non relevant content.

− > Current exploration of segmentation methods to generatesegments that have high recall and precision of the relevantcontent for the query

I Due to small size of the query set no general conclusionson the difference based on the speech act type

ediaEval 2012 Brave New Task:Search and Hyperlinking

I Use Scenario:I A user is searching for a known segment in a video

collection.I Furthermore, because the information in the segment might

not be sufficient for his information need, s/he wants tohave links to other related video segments, which may helpto satisfy information need related to this video.

I Sub-tasks:I Search: finding suitable video segments based on a short

natural language query,I Linking: defining links to other relevant video segments in

the collection.

I Sub-tasks:

I Search: finding suitable video segments based on a shortnatural language query,

I Linking: defining links to other relevant video segments inthe collection.

natural language query,

I Linking: defining links to other relevant video segments inthe collection.

natural language query,I Linking: defining links to other relevant video segments in

the collection.

ediaEval 2012

Thank you for your attention!

Welcome to MediaEval 2012! http://multimediaeval.org

Comparing Retrieval Effectiveness of Alternative Content Segmentation Methods for Internet Video...

Technology

Comparing Offline and Online Statistics Estimation for Text Retrieval from Overlapped Collections MS Thesis Defense Bhaumik Chokshi Committee Members:

Comparing Document Segmentation for Passage Retrieval in Question Answering

Evaluation via Negativa of Chinese Word Segmentation for Information Retrieval @ PACLIC 2011

Media Retrieval Information Retrieval Image Retrieval Video Retrieval Audio Retrieval Information Retrieval Image Retrieval Video Retrieval Audio Retrieval

Comparing Customer Segmentation With CLV Using Data …Aslan, D., Tolon, M. (2018). Comparing Customer Segmentation With CLV Using Data Mining and Statistics: A Case Study, Journal

5 Satellite-retrieval of mass balance: comparing SAR

Spatial color image retrieval without segmentation using ... · Spatial color image retrieval without . segmentation using thumbnails . and the Earth Mover’s Distance. Thomas Hurtut,

Comparing Manual and Automatic Segmentation of Hippocampal ... · Comparing Manual and Automatic Segmentation of Hippocampal Volumes: Reliability and Validity ... (Hc) has been

Comparing semantic technologies - strengths and weaknessesweb3.custompublish.com/getfile.php/692235.177... · - information retrieval (search, navigation) - information integration

Methanol from TES global observations: retrieval algorithm and … · 2020. 7. 31. · TES retrieval globally and comparing the results with corre-sponding GEOS-Chem output, we ﬁnd

Application of Topic Segmentation in Audiovisual Information Retrieval

Comparing Small Graph Retrieval Performance for …danielschlegel.org/pubs/dmah2015.pdfComparing Small Graph Retrieval Performance for Ontology Concepts in Medical Texts Daniel R

Segmentation-Free Keyword Retrieval in Historical Document ...el-sana/publications/pdf/Segmentation-free... · leaves keyword spotting technique as a practical alternative [15]. In

Bayesian segmentation of brainstem structures in MRI · nation with the FreeSurfer dataset, described below); and second, to directly evaluate the segmentation method, by comparing

Color- And Texture-based Image Segmentation Using Em … · Color- and Texture-Based Image Segmentation Using EM and Its Application to Content-Based Image Retrieval ... algorithm

Automated neurosurgical video segmentation and retrieval system

“Inside the Bible”: Segmentation, Annotation and Retrieval for a … · 2020. 9. 23. · image: we will focus only on them in the content-based image retrieval stage. The retrieval

Medical Image Retrieval Using Fuzzy Connectedness Image ... · Medical Image Retrieval Using Fuzzy Connectedness Image Segmentation ... expectation maximization [4]-[6] algorithm

Comparing Boolean and Probabilistic Information Retrieval Systems

Chances and Challenges in Comparing Cross-Language Retrieval Tools