A web system for ontology-based multimedia annotation, browsing and search M. Bertini, G. Becchi, A. Del Bimbo, A. Ferracani, D. Pezzatini University of Florence - MICC System architecture The system provides a service-oriented architecture (SOA) that allows for multiple viewpoints of multimedia data inside repositories, providing better ways to reuse, repurpose and share rich media. The analysis layer is responsible for extracting low level features and semantic annotations from media files, through a series of processing pipelines that can be executed in a cloud of servers, orchestrated by apposite services. Annotation of visual content is performed using the Bag-of-Visual-Words paradigm, based on a fusion of MSER, SURF and SIFT features and the Pyramid Matching Kernel. Audio annotation is based on a fusion of features like ZCR, MFCCs, chroma and spectral features and SVM classifiers. CBIR retrieval is performed using rhythm and pitch features for audio and MPEG-7 features for visual data. Semantic search / browse Syntactic search / browse Database Media storage Video/Image analysis pipeline Audio analysis pipeline user-defined pipeline Local SOA architecture System SOA System repository Corporate CMS End-user interfaces Authoring environment File storage Analysis Interface layer Authoring layer SOA Architecture layer Analysis layer Semantinc annotation, browsing and search tools All the web applications of the system have been developed according to the RIA paradigm. In particular the applications of the Interface and Authoring layers are developed in AJAX and Flash/Flex, while data is exchanged using SOAP, RSS and JSON for metadata and RTMP for video streaming. Interfaces allow to perform manual annotation (to check automatic annotations, add metadata or create ground truth annotations to train new automatic concept detectors), browse , search (semantic and content based) and tagging. Usability evaluation The goal of the field trials was to assess the usability of the system, in particular letting the users to interact with the search engine and its interfaces, to pose semantic and syntactic-level queries, but also to annotate, automatically and manually, some videos. http://www.micc.unifi.it/ This work was partially supported by the EU IST IM3I project (contract FP7-222267). http://www.im3i.eu

Icme2011 industrial poster

Download PDF Report

Upload
media-integration-and-communication-center
View
273
Download
0

Embed Size (px)

Citation preview

A web system for ontology-based multimedia annotation,browsing and search

M. Bertini, G. Becchi, A. Del Bimbo, A. Ferracani, D. Pezzatini

University of Florence - MICC

System architecture

The system provides a service-oriented architecture(SOA) that allows for multiple viewpoints of multimedia datainside repositories, providing better ways to reuse, repurposeand share rich media.

The analysis layer is responsible for extracting low levelfeatures and semantic annotations from media files,through a series of processing pipelines that can be executedin a cloud of servers, orchestrated by apposite services.

Annotation of visual content is performed using theBag-of-Visual-Words paradigm, based on a fusion of MSER,SURF and SIFT features and the Pyramid Matching Kernel.Audio annotation is based on a fusion of features like ZCR,MFCCs, chroma and spectral features and SVM classifiers.CBIR retrieval is performed using rhythm and pitch featuresfor audio and MPEG-7 features for visual data.

Semantic search / browse

Syntactic search / browse

Database

Media storage

Video/Image analysis pipeline

Audio analysis pipeline

user-defined pipeline

Local SOA architecture

System SOA

System repository

Corporate CMS

End-user interfaces

Authoring environment

File storage

Analysis

Interface layer Authoring layer

SOA Architecture layer

Analysis layer

Semantinc annotation, browsing and search tools

All the web applications of the system havebeen developed according to the RIAparadigm. In particular the applications of theInterface and Authoring layers are developedin AJAX and Flash/Flex, while data isexchanged using SOAP, RSS and JSON formetadata and RTMP for video streaming.

Interfaces allow to perform manualannotation (to check automaticannotations, add metadata or create groundtruth annotations to train new automaticconcept detectors), browse , search(semantic and content based) and tagging.

Usability evaluation

The goal of the field trials was to assess the usability of the system, in particular letting the users to interact withthe search engine and its interfaces, to pose semantic and syntactic-level queries, but also to annotate, automaticallyand manually, some videos.

http://www.micc.unifi.it/ This work was partially supported by the EU IST IM3I project (contract FP7-222267). http://www.im3i.eu