IST DIVAS Presentation 1 Advanced search technologies for digital audio-visual content

IST DIVAS Presentation 1

Advanced search technologies for digital audio-visual content

Divas represents the combined efforts of eight companies and institutions to:

› design and develop a multimedia search engine › based on advanced direct video and audio

search algorithms applied directly on encoded (compressed) content.

IST DIVAS (FP6 IST-2-04582) was officially launched at the 1st of January 2007, with a duration of 24 months.


Availability of huge and ever expanding distributed repositories of media in various formats

how can a system efficiently and reliably identify

content fragments captured from various streams?

Only techniques for indexing and searching raw (uncompressed) content are available today (and text-based techniques)


› provides the capability to the user to locate captured video and audio feeds with missing additional context like title, filename, origin, location, service provider

etc.

› or in situation where metadata based queries are inapplicable


› Search of multimedia libraries using the techniques for uncompressed content is a heavy duty/ costly solution

because for the search each item has to be decompressed.


› Metadata annotation may be a heavy duty/costly solution to content owners.

› A complementary solution should therefore also be made available


Audio-visual signature/ fingerprint extraction directly from compressed resources

Extend Search TechniquesBy supporting content queries, DIVAS extends the state of the

art beyond nowadays pursued search techniques based on metadata.

Improve the reliability of audio-visual content detection

By its multimodal (video & audio content) approach, and by combining the query results obtained from both modalities.



DIVAS proposes characterization, feature extraction and direct search of compressed video › as opposed to cognitive-level metadata annotation from

uncompressed video streams.

“Video fingerprinting”, › as envisaged (but not extensively exploited) in the MPEG-7

standard is the term approximately fitting to our approach.

DIVAS will pursue:› Mpeg-2 compliant implementation

› a H.264 compliant implementation.


Already a relatively mature technology on uncompressed audio

Based on the extraction of fingerprints, which capture the characteristic features of an audio clip. › These fingerprints are then compared to the

fingerprint of a query (an audio clip to search for).


DIVAS system search techniques incorporate in parallel both audio and video based searching.

In terms of functional decomposition the system will address audio and video in a different way.

DIVAS system utilizes two different engines:

› a/generate unique indexes from each clip › b/search among the aforementioned identifiers,

providing a match/no match answer to the user.


Open architecture Future-proof design Scalability Interoperability Expandability Modularity



Functional OverviewFunctional Overview


Compressed Audio Signal

Direct conversion into the suitable time/frequency

domain

Feature Extraction

Speech Recognition Music Information Retrieval

Decoding

Conversion to suitable time/frequency domain

DIVASConventional

Tool “A”Content uploading

Content index

Tool “C”Administration Updating

Tool “B”Content search

Result of content search

Indexes Indexes (fingerprints)(fingerprints)

DB DB Writing

Reading

DIVASDIVASENGINEENGINE

Multiplexed

Content

Multiplexed

indexes

Contentdemultiplexer

Indexmultiplexer

Video features

extractionengine

Audio features

extractionengine

Engine of text/meta features

extraction

Video/audio/textcontent

Video/audio/textindexes

Plug-ins Plug-ins

See next slide

Video

content

Video Decoder

(Transcoder)

Featuresextractor

Video

index

Plug-ins Plug-insPlug-ins supporting video fingerprints

Scene change plug-in

Brightness change plug-in

Frame content plug-in

Plug-ins supporting video formats

MPEG2

MPEG4

AVC/H.264

VC1

etc.etc.

Query

content

CONTENT FEATURES

EXTRACTION ENGINE

COMPARISON ENGINE

Query

index

Search

result

Query

indexSearchresult

Index reader

Indexcomparer

Plug-ins

Plug-insIndexes Indexes (fingerprints)(fingerprints)

DBDB

Search

result

Sea

rche

d

inde

x

Read

Query

index

Plug-ins for statistics comparison

Plug-in for scene change statistics comparison

Plug-in for brightness statistics comparison

Plug-in for time characteristics statistics comparison

Plug-in for average brightness and contrast statistics

comparison

Query

index

Monitoring resultIndexcomparer

CONTENT FEATURES

EXTRACTION ENGINE

Content stream reader

Plug-insContent Content streamstream

Monitored

content

Inde

x

Read

Query

index

Plug-ins of content stream reading

Plug-ins of read stream from capture devices

Plug-ins of read stream from remote source

Documents

IST DIVAS Presentation 1 Advanced search technologies for digital audio-visual content