Intelligent Semantic Access to Audiovisual

ContentYannis Avrithis, Giorgos Stamou,

Anastasios Delopoulos and Stefanos Kollias

Image, Video and Multimedia Systems Laboratory

Department of Electrical and Computer Engineering

National Technical University of Athens

Conventional A/V ArchivesDelicate or fragile media such as film, videotapes, slides, paper

difficult access, preview & reproductiondegradation / expensive conservation

Not computerized indexes/cataloguesslow & inconvenient search

Lack of standardized documentationLocal nature / isolated archives

multiple searches, physical presence

Digitization of the ArchivesDigital a/v material, digital storage

MPEG digital video & audiopreservation & reproduction is trivialeasy access & media distribution

Digital indexing, cataloguing & documentation

efficient search, visual previewqueries using keywords or "free" text

Beyond Digitization...A/V objects vs. frames (MPEG-4)

direct access to shots, frames or objectsMetadata

info about the a/v materialstandardized annotation & documentation

Visual content based search (MPEG-7)rich information, more efficient queriessearch & retrieval at the level of objects

The Proposed System

FAETHONsystem

a/v archive #1

a/v archive #2

a/v archive #N

end user

FAETHON's 3-tierarchitecture

Major ObjectivesOffer unified semantic views to individual a/v archives

beyond the individual classification schemes & subject indexes of each archive

Personalize these views according to the profile of individual users

semantic interpretation heavily relies on the context which in turn depends on the specific profile.

Unified Semantic ViewsIndividual a/v archives have different

thematic categoriesdetail of content descriptionsupport of a/v content based description

Centralized user access require eitherre-building (common archiving schema)mapping of queries to individual schemas

Mapping possible only at semantic level

System FeaturesAdopts standard MPEG-7 features & descriptions for content-based accessDynamically extracts high level semantic descriptions of a/v unitsEnables queries at high semantic levelCommunicates with custom a/v archivesGenerates & updates user profilesEmploys user preferences for filtering

System ArchitectureDynamic Thematic

Categorization (DTC)

Personalization

Detection of Eventsand CompositeObjects (DECO)

PresentationFiltering Module

Query AnalysisModule

A/V ClassificationModule

User Profile UpdateModule

Semantic Unification

Search Engine

Individual A/V ArchiveInterface

Query TranslationModule

ResponseAssembly Module

Archive Profile User Profiles

- Usage History- User Preferences

Encyclopaedia

- Semantic Entities- Relations- Thesaurus

Index

- Semantic locators

Faethon DBMS

Encyclopaedia UpdateModule

Searching

User Interface

User CommunicationModule

User PresentationModule

User InteractionModule

Information ModelMultimedia document DS’s

creation, media, usage, structure, access & navigation information

Data storageencyclopaedia, index, user profiles, a/v archive profiles

Query / response between interfacesModule communication

internal query / response, database access

Modes of OperationQuery mode

query analysis & search enginea/v classification & presentation filtering

Update modedynamic thematic categorization (DTC)detection of events & composite objects (DECO)encyclopaedia & user profile update

Query Mode

Personalization

PresentationFiltering Module

Query AnalysisModule

A/V ClassificationModule

User Profiles

- Usage History- User Preferences

Encyclopaedia

- Semantic Entities- Relations- Thesaurus

Index

- Semantic locators

Faethon DBMS

Search Engine

User Query FaethonResponse

FaethonQuery

ArchiveResponse

To / from end users

Individual A/V ArchiveInterface

Query TranslationModule

ResponseAssembly Module

Archive Profile

To /

from

indi

vidu

al A

/V a

rchi

ves

User Interface

User CommunicationModule

User PresentationModule

User InteractionModule

Searching

Query ModeUser query: semantic, metadata & visual specificationsQuery analysis: interpretation, expansion & profilingSearch engine: index & archive queryA/v classification: user interestsPresentation filtering: user preferences

Update ModeDynamic Thematic

Categorization (DTC)

Personalization

Detection of Eventsand CompositeObjects (DECO)

User Profile UpdateModule

User Profiles

- Usage History- User Preferences

Encyclopaedia

- Semantic Entities- Relations- Thesaurus

Index

- Semantic locators

Faethon DBMS

Semantic Unification

FaethonQuery

ArchiveResponse

Individual A/V ArchiveInterface

Query TranslationModule

ResponseAssembly Module

Archive Profile

To /

from

indi

vidu

al A

/V a

rchi

ves

Encyclopaedia UpdateModule

Update ModeDTC: associate thematic categories to a/v units in indexDECO: detect & associate simple & composite semantic entities (objects, events, concepts etc.) to a/v unitsEncyclopaedia update: entity relations thesaurusUser profile update: usage history user preferences

ConclusionsUnified semantic views to individual a/v archivesDynamic extraction of high level semantic descriptionsQueries at high semantic levelPersonalized response according to user preferencesConformance to MPEG-7