Upload
harald-sack
View
419
Download
2
Embed Size (px)
DESCRIPTION
Keynote at LinkedUp Competition Workshop 2014, at ESWC 2014, May 26, 2014, Crete, Greece. Abstract: Video Data has become a first class citizen of the Web. Also in the area of eLearning lecture recordings in combination with massive open online courses have become rather popular and their numbers are constantly growing. To find the appropriate video content according to the user’s information needs and to make well suited content based recommendations for videos still is an issue of ongoing research. Semantic technologies combined with state-of-the-art video analysis, as e.g. speech to text, video ocr, and visual concept detection, enable pinpoint access to video data and clear the path for content based recommender systems. Based on time-based semantically annotated video data new ways become feasible of accessing video in an exploratory way to enhance user experience. - See more at: http://linkedup-project.eu/linkedup-eswc-workshop/#sthash.QuCYDtkN.dpuf
Citation preview
LinkedUp Vidi Competition Presentations and Awards 2014, ESWC 2014
Semantic Analysis of Video Data to Enable Exploratory Search
Dr. Harald SackHasso-Plattner-Institut for IT-Systems Engineering
University of Potsdam
Crete, 25/06/2014
Montag, 26. Mai 14
Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011
•How to Search Video Web vs. Archive
•How to Open Up Video Data?Video Analysis Technologies
•How to „Understand“ Video Metadata? Context-Driven Semantic Analysis
•How to Use of Semantic Metadata?Exploratory Search and Intelligent Recommendations
Semantic Analysis of Video Data to Enable Exploratory Search
Montag, 26. Mai 14
Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität Potsdam
3
Searching the WebMontag, 26. Mai 14
Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität Potsdam
4
Searching the WebMontag, 26. Mai 14
Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität Potsdam
5
Montag, 26. Mai 14
Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität Potsdam
6
Google Knowledge Graph
= “search results with semantic- search information gathered from a wide variety of sources“
Montag, 26. Mai 14
Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Workshop ,Corporate Semantic Web‘, XInnovations 2011, Berlin, 19. Sep. 2011Google Multimedia Search
Montag, 26. Mai 14
Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011
‣Google Multimedia Search relies on text-based metadata and link context
How does Google find Multimedia?
Montag, 26. Mai 14
Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011
Seach by Media Content
Montag, 26. Mai 14
Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011
The Ordinary Archive is a Small World...
Neil Armstrong
Montag, 26. Mai 14
Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011
But, wouldn‘t it be nice, if.....
Neil Armstrong
...but maybe you are also interested in
- Buzz Aldrin (1 videos)
- John Glen (1 video)
- Juri Gagarin (2 videos)
- Richard Nixon (3 videos)
- Apollo 11 (1 video)
- NASA (20 videos)
- Moon (14 videos)
- space exploration (34 videos)
- technology (1.205 videos)
Sorry, no results found for ‘Neil Armstrong‘...
Montag, 26. Mai 14
Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011
How to Search in Video Archives?
Montag, 26. Mai 14
vfm - Seminar: Metadatenmanagement in Medienunternehmen, 05. September 2012, Bonn Jörg Waitelonis, Hasso-Plattner-Institut Potsdam
Content-Based Search in Video Archives relies on text-based Metadata
Current Solution: Manual Annotation
Montag, 26. Mai 14
Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011 image
VisualConceptDetection
Text Recognition
Visual Analysis
(Selected) Automated Multimedia Analysis
Face Detection
Face Detection
Logo Detection
audio-visual
text / images
Audio-Mining
structuralanalysis
AutomatedSpeech
Recognitionaudio event detection
audio
Montag, 26. Mai 14
Annotation of Audiovisual Data
Metadata Extraction
Metadata (e.g. MPEG-7) ... <SpatialDecomposition> <TextAnnotation> <KeywordAnnotation> <Keyword>Astronaut</Keyword> </KeywordAnnotation> </TextAnnotation> <SpatialMask> <SubRegion> <Polygon> <Coords> 480 150 620 480 </Coords> </Polygon> </SubRegion> </SpatialMask> ... </SpatialDecomposition> ...
• Video data with spatiotemporal Annotations
Neil Armstrong
Montag, 26. Mai 14
Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität Potsdam
www.yovisto.com
Montag, 26. Mai 14
Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011
• Authoritative Metadata• structured data• semi-structured data
• natural language text • Non-authoritative Metadata
• (free) user tags and comments• restricted vocabularies
• (Media) Analysis Metadata• low level features• high level features
• etc.
How to Determine the Meaning of Metadata?
SemanticAnalysis
reliability
context
pragmatics
location dependency
accuracy
timedependency
level ofabstraction
Montag, 26. Mai 14
Neil Armstrong
Astronaut
is a
Person
is a
Science Occupation
subClassOf
Employment
subClassOf
Entities
Ontologies
has an
,Neil Armstrong‘ is more than just a character string
Kosmonautsame as
Juri Gagarin
is a
is NOT a
!
Montag, 26. Mai 14
Astronaut Person
Neil Armstrong
Science Occupation
Employment
is a is a
is a
is a has a
Web of Data
Montag, 26. Mai 14
Web of Data = Linked Open DataBut what, if there is no trivial unique identification?
Armstronguser tag
Montag, 26. Mai 14
Semantic Web Technologies , Dr. Harald Sack, Hasso Plattner Institute, University of Potsdam
Armstrong
Montag, 26. Mai 14
Semantic Web Technologies , Dr. Harald Sack, Hasso Plattner Institute, University of Potsdam
ArmstrongArmstrong+Moon
Montag, 26. Mai 14
Web of Data = Linked Open DataUnderstanding requires Context
Armstrong
Moon
EagleSpace
Montag, 26. Mai 14
4242 42 4224424242 42 4242Semantic AnalysisSemantics is determined by Context
Context Item
N.Steinmetz, H.Sack: Semantic Multimedia Information Retrieval Based on Contextual Descriptions, 2013
„Armstrong landed the Eagle on the Moon.“Text
SEMEX Multimedia Context Model
Context Dimensions
TemporalContext
SpatialContext
ProvenanceContext
Relevance
determines
Ambiguity
influences
Accuracy
influences
Contextual Description
ClassDiversity
Level of Structure
SourceReliability
SourceDiversity
Montag, 26. Mai 14
Armstrong
George Armstrong Custer
Neil Armstrong
The Armstrong Twins
Armstrong, Florida
Armstrong, Ontario
Armstrong Automobile
Joe Armstrong
Armstrong County, Texass
Armstrong Gun
Craig Armstrong
Armstrong (Moon Crater)
Louis Armstrong
Armstrong Tunnel
Louis Armstrong International Airport
Armstrong‘s Theorem
Sir Thomas Armstrong
Ian Armstrong
Eagle Moon
Eagle (Bird)
Eagle (heraldry)
USCGC Eagle
The Eagle (2011 film)
Eagle (song)
John H. EagleEagle (typeface)
Eagle Falls (Washington)
Eagle (Moon Crater)
Eagle (comic)
Eagle (lunar module)
Eagle TV
Armstrong Tunnel
The Eagle (Pub)
War Eagle
The Eagle (newspaper)
Eagle (racehorse)
Angela EagleLinda Eagle
James Philipp Eagle
95 entities448 entities
Armstrong (British Columbia)Karen Armstrong
Curtis Armstrong
Gillian Armstrong Hilary Armstrong
William L. Armstrong
156 entities
Man on the Moon (film)
Moon (song)
Moon Son-Ri
C Moon
The Moon (Tarot card)
Edgar Moon
Moon OS
Moon (Band)
Moon
Moon 44
Man on the Moon (soundtrack)
William Moon
Lottie Moon
Mr. Moon (song)
Man on the Moon (musical)
Darvin Moon
Moon 83
Francis MoonGary Moon
Robert Charles Moon
Black Moon
Allan Moon
Ban-Ki Moon
Fly me to the Moon (song)
Semantic AnalysisNamed Entity Mapping
„Armstrong landed the Eagle on the Moon.“
Consider all entities within the same context
Montag, 26. Mai 14
Armstrong
George Armstrong Custer
The Armstrong Twins
Armstrong, Florida
Armstrong, Ontario
Armstrong Automobile
Joe Armstrong
Armstrong County, Texass
Armstrong Gun
Craig Armstrong
Armstrong (Moon Crater)
Armstrong Tunnel
Louis Armstrong International Airport
Armstrong‘s Theorem
Sir Thomas Armstrong
Ian Armstrong
Eagle Moon
Eagle (Bird)
Eagle (heraldry)
USCGC Eagle
The Eagle (2011 film)
Eagle (song)
John H. EagleEagle (typeface)
Eagle Falls (Washington)
Eagle (Moon Crater)
Eagle (comic)
Eagle TV
Armstrong Tunnel
The Eagle (Pub)
War Eagle
The Eagle (newspaper)
Eagle (racehorse)
Angela EagleLinda Eagle
James Philipp Eagle
95 entities448 entities
Armstrong (British Columbia)Karen Armstrong
Curtis Armstrong
Gillian Armstrong Hilary Armstrong
William L. Armstrong
156 entities
Man on the Moon (film)
Moon (song)
Moon Son-Ri
C Moon
The Moon (Tarot card)
Edgar Moon
Moon OS
Moon (Band)
Moon 44
Man on the Moon (soundtrack)
William Moon
Lottie Moon
Mr. Moon (song)
Man on the Moon (musical)
Darvin Moon
Moon 83
Francis MoonGary Moon
Robert Charles Moon
Black Moon
Allan Moon
Ban-Ki Moon
Neil Armstrong
Eagle (lunar module)
Moon
Louis Armstrong
Fly me to the Moon (song)
Semantic AnalysisNamed Entity Mapping
„Armstrong landed the Eagle on the Moon.“
N. Steinmetz, H.Sack: Semantic Multimedia Information Retrieval Based on Contextual Descriptions, 2013
Entity Selection Process(Semantic) Graph Analysis
Montag, 26. Mai 14
4242 42 4224424242 42 4242
vfm - Seminar: Metadatenmanagement in Medienunternehmen, 05. September 2012, Bonn Jörg Waitelonis, Hasso-Plattner-Institut Potsdam
27
Semantically Annotated Video Data
Video Analysis /Metadata Extraction
timemetadata
metadatametadata
metadatametadata
e.g., person xylocation yzevent abc
e.g., bibliographical data,geographical data,encyclopedic data, ..
Entity Recognition/ Mapping
N. Ludwig, H. Sack: Named Entity Recognition for User-Generated Tags. In Proc. of the 8th Int. Workshop on Text-based Information Retrieval, IEEE CS Press, 2011
Montag, 26. Mai 14
Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität Potsdam
28
Entity Based Search
• linguistic ambiguities of traditional keyword based search can be avoided
• enables high precision and high recall retrieval
http://www.yovisto.com/labs/autosuggestion/
• Query string refinement / extension• entity auto-suggestion• interpretation of natural language queries
J. Osterhoff, J. Waitelonis, H. Sack, Widen the Peepholes! Entity-Based Auto-Suggestion as a rich and yet immediate Starting Point for Exploratory Search, IVDW 2012
Montag, 26. Mai 14
Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität Potsdam
29
http://semex.hpi.uni-potsdam.de/mggui-dev2
search facets
C. Hentschel, H. Sack, et al., Open up cultural heritage in video archives with mediaglobe, I2CS 2012
Montag, 26. Mai 14
Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität Potsdam
30
Towards Exploratory Search
dbpedia-owl:mission
dbpedia:Neil_Armstrong
dbpedia:Apollo_11dbpedia-owl:mission
category:Apollo_program
dcterms:subject
dbpedia:Apollo_13
dcterms:subject
yago:Space_accidents_and_incidents
rdf:type
rdf:type
dbpedia:Space_Shuttle_Challenger
dbpedia-owl:mission
J. Waitelonis, H. Sack: Towards exploratory video search using linked data, MTAP Volume 59, Number 2 (2012), 645-672
dbpedia:Buzz_Aldrin
dbpedia:Michael_Collins
Montag, 26. Mai 14
Exploratory Search and Serendipity•Find something that you were not looking for on purpose ...
dbpedia:Buzz_Aldrin
dbpedia:Cookie_Monster
dbpedia:Strictly_Come_Dancing
dbpedia:Transformers
Montag, 26. Mai 14
Explorative Search & Intelligent Recommmendationwith yovisto
J. Waitelonis, H. Sack: Towards exploratory video search using linked data, Multimedia Tools and Applications, Volume 59, Number 2 (2012), pp. 645-672
Montag, 26. Mai 14
Explorative Search & Intelligent Recommmendationwith yovisto
http://mediaglobe.yovisto.com:8080/
Montag, 26. Mai 14
Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011
Contact:Dr. Harald SackHasso-Plattner-Institut für SoftwaresystemtechnikUniversität PotsdamProf.-Dr.-Helmert-Str. 2-3D-14482 Potsdam
Homepage:http://www.hpi.uni-potsdam.de/meinel/team/sack.htmlBlog: http://yovisto.blogspot.com/E-Mail: [email protected] Twitter: lysander07 / biblionomicon / yovisto Slides can be found at http://slideshare.com/lysander07/
Thank you very much
for your attention!
Montag, 26. Mai 14