Upload
yandex
View
354
Download
2
Embed Size (px)
DESCRIPTION
Научно-технический семинар Гарета Джонса 12 сентября
Citation preview
Multimedia Information Seeking ThroughSearch and Hyperlinking
Gareth J. F. Jones
Centre for Next Generation LocalisationSchool of Computing, Dublin City University, Dublin, Ireland
Overview
Background and Motivation
MediaEval Multimedia Evaluation Benchmark
Search and HyperlinkingSearch and Hyperlinking at MediaEval 2012Search and Hyperlinking at MediaEval 2013
Conclusions and Research Prospects
Background and Motivation
Introduction
I Search and hyperlink navigation are well establishedapproaches to information seeking on the textual web.
I Increasing amounts of online digital multimedia content iscreating new challenges and opportunities for informationaccess.
I Realizing the potential of this content requires users to beprovided with engaging ways to interact with it, helpingthem to discover, browse, navigate and search.
I Different classes of users, such as media professionals,students, researchers and home users will potentially havedifferent needs and preferred means of access.
Background and Motivation
Introduction
I Search and hyperlink navigation are well establishedapproaches to information seeking on the textual web.
I Increasing amounts of online digital multimedia content iscreating new challenges and opportunities for informationaccess.
I Realizing the potential of this content requires users to beprovided with engaging ways to interact with it, helpingthem to discover, browse, navigate and search.
I Different classes of users, such as media professionals,students, researchers and home users will potentially havedifferent needs and preferred means of access.
Background and Motivation
Introduction
I Search and hyperlink navigation are well establishedapproaches to information seeking on the textual web.
I Increasing amounts of online digital multimedia content iscreating new challenges and opportunities for informationaccess.
I Realizing the potential of this content requires users to beprovided with engaging ways to interact with it, helpingthem to discover, browse, navigate and search.
I Different classes of users, such as media professionals,students, researchers and home users will potentially havedifferent needs and preferred means of access.
Background and Motivation
Introduction
I Search and hyperlink navigation are well establishedapproaches to information seeking on the textual web.
I Increasing amounts of online digital multimedia content iscreating new challenges and opportunities for informationaccess.
I Realizing the potential of this content requires users to beprovided with engaging ways to interact with it, helpingthem to discover, browse, navigate and search.
I Different classes of users, such as media professionals,students, researchers and home users will potentially havedifferent needs and preferred means of access.
Background and Motivation
Introduction
I Existing research on multimedia information retrievalmainly focused on:
I search for individual relevant items - generally focusing onvisual information needs and features; or
I linking of content based on sharing visual features withoutattention to user information needs, e.g. linking videoscontaining the same individuals.
I Linking of multimedia content to text resources, e.g.wikipedia.
I Social remmendation based on user behaviour.
I Our work in search and hyperlinking aims to exploremultimodal access over multimedia content, and linking ofvideo content to support potential user interests or needs.
Background and Motivation
Introduction
I Existing research on multimedia information retrievalmainly focused on:
I search for individual relevant items - generally focusing onvisual information needs and features; or
I linking of content based on sharing visual features withoutattention to user information needs, e.g. linking videoscontaining the same individuals.
I Linking of multimedia content to text resources, e.g.wikipedia.
I Social remmendation based on user behaviour.
I Our work in search and hyperlinking aims to exploremultimodal access over multimedia content, and linking ofvideo content to support potential user interests or needs.
Background and Motivation
Introduction
I Existing research on multimedia information retrievalmainly focused on:
I search for individual relevant items - generally focusing onvisual information needs and features; or
I linking of content based on sharing visual features withoutattention to user information needs, e.g. linking videoscontaining the same individuals.
I Linking of multimedia content to text resources, e.g.wikipedia.
I Social remmendation based on user behaviour.
I Our work in search and hyperlinking aims to exploremultimodal access over multimedia content, and linking ofvideo content to support potential user interests or needs.
Background and Motivation
Introduction
I Existing research on multimedia information retrievalmainly focused on:
I search for individual relevant items - generally focusing onvisual information needs and features; or
I linking of content based on sharing visual features withoutattention to user information needs, e.g. linking videoscontaining the same individuals.
I Linking of multimedia content to text resources, e.g.wikipedia.
I Social remmendation based on user behaviour.
I Our work in search and hyperlinking aims to exploremultimodal access over multimedia content, and linking ofvideo content to support potential user interests or needs.
Background and Motivation
Introduction
I Existing research on multimedia information retrievalmainly focused on:
I search for individual relevant items - generally focusing onvisual information needs and features; or
I linking of content based on sharing visual features withoutattention to user information needs, e.g. linking videoscontaining the same individuals.
I Linking of multimedia content to text resources, e.g.wikipedia.
I Social remmendation based on user behaviour.
I Our work in search and hyperlinking aims to exploremultimodal access over multimedia content, and linking ofvideo content to support potential user interests or needs.
Background and Motivation
Introduction
I Existing research on multimedia information retrievalmainly focused on:
I search for individual relevant items - generally focusing onvisual information needs and features; or
I linking of content based on sharing visual features withoutattention to user information needs, e.g. linking videoscontaining the same individuals.
I Linking of multimedia content to text resources, e.g.wikipedia.
I Social remmendation based on user behaviour.
I Our work in search and hyperlinking aims to exploremultimodal access over multimedia content, and linking ofvideo content to support potential user interests or needs.
Background and Motivation
Current Video Recommendation
Search for “Things to do in London”
Background and Motivation
Vision of Hyperlinked Video
Background and Motivation
Vision of Hyperlinked Video
Background and Motivation
Search and Hyperlinking
I These studies take place within the context of theMediaEval Multimedia Evaluation Benchmark campaign(www.multimediaeval.org).
MediaEval Multimedia Evaluation Benchmark
MediaEval Multimedia Evaluation Benchmark
I An evaluation benchmarking initiative focusing oninformation access and indexing tasks for multimediacontent.
I Established in 2010, following on from earlier VideoCLEFtasks at CLEF 2008 and CLEF 2009.
I Evaluates new algorithms primarily for novel and emergingtasks linked to a use case.
I Emphasis on the “multi” in multimedia: audio, speech,visual content, tags, users, context.
MediaEval Multimedia Evaluation Benchmark
MediaEval Multimedia Evaluation Benchmark
I TREC style annual evaluation cycle:I call for task proposals,I call for participation,I data release,I participants submit results,I organisers release evaluation of results,I participants gather at a workshop to discuss the results of
the task.
I Search and Hyperlinking task at MediaEval 2012 andMediaEval 2013.
MediaEval Multimedia Evaluation Benchmark
MediaEval Multimedia Evaluation Benchmark
I TREC style annual evaluation cycle:I call for task proposals,I call for participation,I data release,I participants submit results,I organisers release evaluation of results,I participants gather at a workshop to discuss the results of
the task.
I Search and Hyperlinking task at MediaEval 2012 andMediaEval 2013.
MediaEval Multimedia Evaluation Benchmark
MediaEval Workshops
I MediaEval 2012 workshop in Pisa, Italy - co-located withECCV 2012.
I MediaEval 2013 workshop in Barcelona, Spain -co-located with ACM Multimedia 2013.
MediaEval Multimedia Evaluation Benchmark
MediaEval Workshops
I MediaEval 2012 workshop in Pisa, Italy - co-located withECCV 2012.
I MediaEval 2013 workshop in Barcelona, Spain -co-located with ACM Multimedia 2013.
Search and Hyperlinking
Search and Hyperlinking
I Audio-visual archive exploration scenario.I Video-to-video linking.I Start with archive search assuming an “information need”.I Assume search becomes serendipitous or exploratory by
following hyperlinks.
Search and Hyperlinking
Search and Hyperlinking
I Video, e.g. 2 hoursI Interesting segment,
e.g. 10 minutesI Anchor: segment for
which a user would liketo link from (e.g. 1minute), e.g. “I want toknow more about ...”
I HyperlinkI Target: relevant
segment for a givenanchor, e.g. 5 minutes
Search and Hyperlinking
Search and Hyperlinking
Search and Hyperlinking
Search and Hyperlinking
I MediaEval 2012 Search and Hyperlinking task (datasetblip.tv):
I Search: retrieve known-item video segment given a naturallanguage description
I Hyperlinking: link known-item video segment to similarsegments in the collection
I MediaEval 2013 Search and Hyperlinking task (datasetBBC archive):
I Search: retrieve known-item video segment given a naturallanguage description
I Hyperlinking: link user-defined anchor within theknown-item to relevant target video segments in thecollection
Search and Hyperlinking
Search and Hyperlinking at MediaEval 2012
I Multimodal search where relevance can be related to bothvisual and/or audio information streams.
I Models scenario where user seeks single relevantknown-item fragment of multimedia content.
I Once the relevant fragment has been located the user canengage in browsing or exploratory search by followingautomatically created hyperlinks to related multimediafragments within the same collection.
Search and Hyperlinking
Blip10000 dataset
The Blip10000 dataset was created by the PetaMedia EUNetwork of Excellence:
I Contains 14,838 Creative Commons videos crawled fromblip.tv, and corresponding user provided metadata.
I Total of ca. 3,260 hours of data: Divided into 5,288development videos, and 9,550 test videos.
I Automatically created visual features:I shot boundaries - ave. shot length 30 secsI single keyframe for each shotI visual concepts from a set of 589
I Two automatic speech recognition transcripts created:LIMSI/Vocapia and LIUM.
I Comparative experiments using only English languagedataset: 4,890 test videos.
Search and Hyperlinking
Test Query Set
I 30 textual queries consisting of natural language typestatements and search engine style queries.
I Textual information collected via crowdsourcing using theAmazon Mechanical Turk (AMT) platform; multimodalfeatures added manually afterwards.
I Query set for video selected at random from the top 10genre categories.
I Turkers required to:I locate an interesting region of the videoI mark the beginning and end points of the interesting regionI transcribe the words spoken exactlyI create natural language and search engine type queries
which they would use to refind this region of the video
Search and Hyperlinking
Evaluation
Average Precision
AP =1n.
N∑r=1
P[r ]
Generalized Average Precision:
GAP =1n.
N∑r=1
P[r ] ·(
1 − DistanceGranularity
· 0.1)
Search and Hyperlinking
Evaluation
Average Precision
AP =1n.
N∑r=1
P[r ]
Generalized Average Precision:
GAP =1n.
N∑r=1
P[r ] ·(
1 − DistanceGranularity
· 0.1)
Search and Hyperlinking
Evaluation
Segment Precision (SP[r ]) at rank r :
Average Segment Precision:
ASP =1n.
N∑r=1
SP[r ] · rel(sr )
SP[r ] = Segment Precision [r ], rel(sr ) = 1, if relevant content ispresent, otherwise rel(sr ) = 0
Search and Hyperlinking
Evaluation
Segment Precision (SP[r ]) at rank r :
Average Segment Precision:
ASP =1n.
N∑r=1
SP[r ] · rel(sr )
SP[r ] = Segment Precision [r ], rel(sr ) = 1, if relevant content ispresent, otherwise rel(sr ) = 0
Search and Hyperlinking
Hyperlinking
I Linking effectiveness evaluated using MAP.I Relevance of each of the top 10 proposed links for each
submitted run evaluated using AMT.I Separate Qrel file created for each run; pooled unified Qrel
created by merging run qrels.I Turkers given the following options:
I Video segments totally unrealtedI Video segments related, same topic or focus, but different
informationI Video segments related, different perspective or view on
the same informationI Video segments are basically the same.
Search and Hyperlinking
Search and Hyperlinking at MediaEval 2013
I Working with 270 hours of BBC archival data:I one complete weekI material not owned by the BBC excluded
I Task designed to explore the information needs and taskactivities of home users.
I Expected to favour exploratory search options.
Search and Hyperlinking
Search and Hyperlinking at MediaEval 2013
I Search queries created by 30 public users with computerand online experience.
I Recruited using external agency.I Profile:
I Nationality: UKI Age Group: 16-30I Computer familiarity: highI Online activity: high
Search and Hyperlinking
Search and Hyperlinking at MediaEval 2013
I Users given a general introduction on video search usingthe AXES Pro system.
I Scope and type of the collection explained, and usersallowed to explore the collection.
I User creates text query expressing information need whichshould be satisfied by the available dataset.
I User searches for a relevant video, reformulating theirquery if necessary.
I Ask the user to identify anchors within their chosen video:specify region, spoken words or whole segmennts.
I Explain reasons why these anchors were selected.
Search and Hyperlinking
Search and Hyperlinking at MediaEval 2013
I Users given a general introduction on video search usingthe AXES Pro system.
I Scope and type of the collection explained, and usersallowed to explore the collection.
I User creates text query expressing information need whichshould be satisfied by the available dataset.
I User searches for a relevant video, reformulating theirquery if necessary.
I Ask the user to identify anchors within their chosen video:specify region, spoken words or whole segmennts.
I Explain reasons why these anchors were selected.
Search and Hyperlinking
Search and Hyperlinking at MediaEval 2013
I Users given a general introduction on video search usingthe AXES Pro system.
I Scope and type of the collection explained, and usersallowed to explore the collection.
I User creates text query expressing information need whichshould be satisfied by the available dataset.
I User searches for a relevant video, reformulating theirquery if necessary.
I Ask the user to identify anchors within their chosen video:specify region, spoken words or whole segmennts.
I Explain reasons why these anchors were selected.
Search and Hyperlinking
Search and Hyperlinking at MediaEval 2013
I Users given a general introduction on video search usingthe AXES Pro system.
I Scope and type of the collection explained, and usersallowed to explore the collection.
I User creates text query expressing information need whichshould be satisfied by the available dataset.
I User searches for a relevant video, reformulating theirquery if necessary.
I Ask the user to identify anchors within their chosen video:specify region, spoken words or whole segmennts.
I Explain reasons why these anchors were selected.
Search and Hyperlinking
Search and Hyperlinking at MediaEval 2013
I Users given a general introduction on video search usingthe AXES Pro system.
I Scope and type of the collection explained, and usersallowed to explore the collection.
I User creates text query expressing information need whichshould be satisfied by the available dataset.
I User searches for a relevant video, reformulating theirquery if necessary.
I Ask the user to identify anchors within their chosen video:specify region, spoken words or whole segmennts.
I Explain reasons why these anchors were selected.
Search and Hyperlinking
Search and Hyperlinking at MediaEval 2013
I Users given a general introduction on video search usingthe AXES Pro system.
I Scope and type of the collection explained, and usersallowed to explore the collection.
I User creates text query expressing information need whichshould be satisfied by the available dataset.
I User searches for a relevant video, reformulating theirquery if necessary.
I Ask the user to identify anchors within their chosen video:specify region, spoken words or whole segmennts.
I Explain reasons why these anchors were selected.
Search and Hyperlinking
Search and Hyperlinking at MediaEval 2013
Search and Hyperlinking
Search and Hyperlinking at MediaEval 2013
Search and Hyperlinking
Search and Hyperlinking at MediaEval 2013
Search and Hyperlinking
Search and Hyperlinking at MediaEval 2013
Search and Hyperlinking
Specification of Anchors
Search and Hyperlinking
Classification of Selected Clips
Search and Hyperlinking
Classification of Anchors
Search and Hyperlinking
Search and Hyperlinking at MediaEval 2013
I Set of 50 known-item experimental queries selectedI Corresponding set of 100 anchors selected
I Run submissions 10th September!I Relevance assessment of links ongoing using returning
test volunteers at BBC and using AMT (to increase amountof relevance assessment in time available)
I MediaEval 2013 workshop 18th-19th October
Search and Hyperlinking
Search and Hyperlinking at MediaEval 2013
I Set of 50 known-item experimental queries selectedI Corresponding set of 100 anchors selectedI Run submissions 10th September!
I Relevance assessment of links ongoing using returningtest volunteers at BBC and using AMT (to increase amountof relevance assessment in time available)
I MediaEval 2013 workshop 18th-19th October
Search and Hyperlinking
Search and Hyperlinking at MediaEval 2013
I Set of 50 known-item experimental queries selectedI Corresponding set of 100 anchors selectedI Run submissions 10th September!I Relevance assessment of links ongoing using returning
test volunteers at BBC and using AMT (to increase amountof relevance assessment in time available)
I MediaEval 2013 workshop 18th-19th October
Search and Hyperlinking
Search and Hyperlinking at MediaEval 2013
I Set of 50 known-item experimental queries selectedI Corresponding set of 100 anchors selectedI Run submissions 10th September!I Relevance assessment of links ongoing using returning
test volunteers at BBC and using AMT (to increase amountof relevance assessment in time available)
I MediaEval 2013 workshop 18th-19th October
Conclusions and Research Prospects
Conclusions and Research Prospects
I Search is a more established task, but little research so faron identifying reliable jump-in points.
I Promising results from Search and Hyperlinking in 2012encouraged us to move to user studies with BBC data.
I Work so far with volunteer subjects suggests that users willmake use of video hyperlinks.
I Results and other findings from Search and Hyperlinking in2013 will be used to help develop focused researchagenda for 2014 and beyond.
Conclusions and Research Prospects
References
I M.Larson, M.Soleymani, M.Eskevich, P.Serdyukov, R.Ordelman and G.J.F.Jones,The Community and the Crowd: Developing large-scale data collections formultimedia benchmarking. IEEE Multimedia. Vol.19, No.3, pp15-23, 2012.
I M.Eskevich, W.Magdy and G.J.F.Jones. New Metrics for Meaningful Evaluationof Informally Structured Speech Retrieval, In Proceedings of the 34th EuropeanConference on Information Retrieval (ECIR 2012), Barcelona, April 2012.
I M.Eskevich, G.J.F.Jones, S.Chen, R.Aly, R.Ordelman and M.Larson, Search andHyperlinking Task at MediaEval 2012, Proceedings of the MediaEval 2012Workshop, Pisa, Italy, October 2012.
I M.Eskevich, G.J.F.Jones., R.Aly, R.J.F. Ordelman, S.Chen, D.Nadeem,C.Guinaudeau, G.Gravier, P.Sebillot, T.de Nies, P.Debevere, R.van de Walle,P.Galuscakova, P.Pecina and M.Larson, Information Seeking through Search andHyperlinking, In Proceedings of ACM International Conference on MultimediaRetrieval (ICMR 2013), Dallas, Texas, USA, April 2013.
I R.Aly, R.Ordelman, M.Eskevitch, G.J.F.Jones and S.Chen, Linking inside a videocollection – what and how to measure?, Proceedings of the First Worldwide WebWorkshop on Linked Media (LiME-2013) at WWW 2013, Rio de Janeiro, Brazil,May 2013.
Conclusions and Research Prospects
Acknowledgements
Coordinators of the MediaEval Search and Hyperlinking task:Robin Aly, Maria Eskevich, Gareth Jones, Roeland Ordelman
Search and Hyperlinking activities supported by: