CONTENTUS—A Semantic Multimodal Search User Interface

More than the Sum of its Parts

CONTENTUS— A Semantic Multimodal Search User Interface

Jörg Waitelonis, Johannes Osterhoff, Harald Sack

Visual Interfaces to the Social and Semantic Web (VISSW 2011) In conjunction with the International Conference on Intelligent User Interfaces (IUI 2011), J. Osterhoff, February 13, 2011

CONTENTUS? A Teaser

Semantic Search Engine

Cultural Heritage Domain

Media types

Articles, Books, Video, Audio, …

People, Locations, …

Part of the HPI research project

Semantic Media Explorer

VISSW 2011 | Johannes Osterhoff | February 13, 2011

2

CONTENTUS? A Teaser


3

Issue #1

How to find the needle

in the haystack?

VISSW 2011 | J. Osterhoff | February 13, 2011

4


5

Image:

iSto

ckphoto


6

needle


7

Google Results

Needless to say…

8

needless


“needle”


9


10

Jimmy Needles American basketball coach

1900—1969

Person Image:

Jay P

. Lee

“needle”


11

Needles, CA


12

Location

Image:

Eddie

Big

gs

“needle”


13

“Needle”


14

Blockbuster by John V. Soto from 2010

Film

Image:

IMD

B

Issue #1

How to find different “needles”

in the haystack?


15

Issue #2

How to present

“needles”

as search results?


16


17

Google Results


18

Flickr Results


19

YouTube Results


20

Wikipedia Results

Issue #2

How to present

different “needles”

as distinguishable

search results

on one page?


21


22

Why?


23

30,000+ Libraries in Germany


24

30,000+ Libraries in Germany


25

Books

Image:

Lib

rary

of

Congre

ss


26

Articles


27

Video

Image:

Lib

rary

of

Congre

ss


28

Audio

Issue #3

How to make the German

cultural heritage available?


29


30

Authors i.e. Bertolt Brecht


31

Years i.e. 1848

People i.e. from Ulm


33

Locations among many others…

Image:

Wik

ipedia

(Soft

eis

)


34

Music Style

Image:

Wik

ipedia


35

Process Chain

HPI Research

Books, Articles, Video, Audio

Authors, Year of Creation,

People, Locations, Music Styles,

Organizations etc.


36

Media Assets

Cultural Categorizations

On the Interface

Issue #3


cultural heritage available

in meaningful

information retrieval?


37


38


39

1. What is CONTENTUS?

2. Interface Design

3. Demonstration of the System

4. Summary

What is CONTENTUS? Introduction

■ a semantic search engine and a multimedia

retrieval system


40



retrieval system

■ web-based graphical user interface


41



retrieval system


■ content-based suggestions and

a multi-modal presentation of

search results to support semantic search


42



retrieval system


■ content-based suggestions and

a multi-modal presentation of

search results to support semantic search

■ developed in the context of the German

research project THESEUS


43


■ Combination of a facetted browsing and

breadcrumb-based navigation

■ A time-line enables time-based filtering

of search results

■ Personalized search (in the next demonstrator)


44


“More than the sum of its parts”


45


More efficient

and more pleasant

user experience.


46

What is CONTENTUS? Context

■ new technologies for managing,

organizing, and searching multimedia

content gets more efficient

■ tremendous diversity of multimedia

data affects the retrieval and search

process

■ image and video search technology is

evolving rapidly


47

What is CONTENTUS? Context


48

■ semantic search technology promises

more accurate and complete search

results

■ new user interfaces

are required to enable query formulation

and intelligent visualization of search

results

What is CONTENTUS? Technologies


49

■ all types of media assets

■ technologies

media optimization

content analysis

semantic metadata generation

speech transcripts

OCR

extraction of semantic entities (i.e. people, places, events)

■ highly heterogeneous collection of semantically

annotated data


50


2. Interface Design


4. Summary

User Interface Paradigms

No research on

a single aspect of

current approaches

in semantic multimedia

retrieval…


51


But:

exploiting synergies

by combining

state-of-the-art

technologies instead.


52


semantic

multimedia

technologies


53


semantic +

multimedia

technologies


54


semantic +

multimedia

technologies +

interface design


55


combination of multimedia

and semantic search

= new challenges in visualizing

search results and facilitating

navigation


56


compare different media at one glance

how to summarize content most

meaningfully in one single search

result item

time-based information (audio, video)


57


multimodal


58


multimodal

= articles, books,

video, audio, people,

locations on one

result-page


59


The route with the

user‟s experience

gets more important

than the original goal

of fact finding.


60


main goal

satisfying search experience

for the user


61

62

63 Search Area

64 Search Field breadcrumbs

65

66

67

Result Area

68

organize and filter

media types

history

collection

69

refine and filter

time-line

faceted suggestions

70 search results

71

article

audio

video

location

article

person video

search results

72

73

74

75

76

less important search results

77

User Interface Detail Page for Books


78

Detail View

User Interface Detail Page for Books


79

Detail View detected entities

full page

other pages

articles on this page

text snippet (OCR)


80


2. Interface Design


4. Summary


81

Demo http://mediaglobe.yovisto.com:8081/contentus/

http://mediaglobe.yovisto.com:8081/contentus/


82


2. Interface Design


4. Summary


83

Issue #1

How to find the needle

in the haystack?

Issue #2

How to present

“needles”

as search results?

Issue #3


cultural heritage available?

Issue #1

How to find different

“needles” in the haystack?

Issue #2

How to present

different “needles” as

distinguishable search

results?

Issue #3


Cultural Heritage available

in meaningful Information

Retrieval?

Summary

Future Work

■ Feedback that lets the user know why exactly the listed

items have been presented

■ Possibility to connect with friends and to share results

with them

■ Suggestions based on friends„ likes

■ State-of-the-art multi-platform web design

■ Next demonstrator to be presented at

IBC 2011 Amsterdam


84


85


86 Person

Location

Book

Video

Audio

Thank you!

Johannes P Osterhoff, M.A.

[email protected] Hasso-Plattner-Institute, Prof.-Dr.-Helmert-Str. 2-3, 14482 Potsdam, Germany


87

mailto:[email protected]



Technology

CONTENTUS—A Semantic Multimodal Search User Interface