Analyzing and Ranking Multimedia Ontologies for their Reuse

Date: 11/04/23Speaker: Ghislain Auguste Atemezing

ANALYZING AND RANKING MULTIMEDIA ONTOLOGIES FOR THEIR REUSE

Master ThesisMáster de investigación en inteligencia artificial

Author: Ghislain Auguste Atemezing

Supervisor: Dr. María del Carmen Suárez de Figueroa Baonza

2Analyzing and Ranking Multimedia Ontologies for their Reuse

Outline

• Introduction

• State of the Art on MultiMedia Ontologies

• Searching MM Ontologies

• Assessing MM Ontologies

• Selecting MM Ontologies

• Conclusions

Introduction (I)

3

• Multimedia content is ubiquitous (Web, TV news, Film, Phone, etc.), and store huge collection of data (Library, Museum, Archives, etc.)

• Multimedia includes a combination of text, audio, still images, animation, video, and interactivity content forms [Chapman 09]

• Many of these contents are available online

Jenny Chapman and Nigel Chapman. Digital Multimedia . John Niley & Sons Ltd, 2009.

Analyzing and Ranking Multimedia Ontologies for their Reuse

Maria

Falta el titulo

Introduction (II)


• Continuously consuming multimedia contents of different formats and from different sources in web environment (e.g., Google, Flickr, Picassa, Youtube).

• How to efficiently retrieve multimedia objects for web developers and ordinary users?

Introduction (III)


• Descriptors based on the automatic analysis of audiovisual content are far from what users require.

• Need for correct semantic annotation and representation of multimedia content.

• Recent research focus on the reduction of semantic and conceptual gap between user and machine. That is, based on the content of high-level descriptions. Reusing KNOWLEDGE in ontology engineering.

Mari Carmen

Demasiado texto. intenta reducirlo con alguna figura o similar

Introduction (IV)

• Many standards to describe MM content: MPEG-4, MPEG7, IPTC, etc.

• Standards provide descriptors schemas for low level description.


Mari Carmen

Debes contar tb que significa la interrogacion.

Maria

titulo

Introduction (V)

• “Semantic gap”: mismatch between the information that can be extracted from audio-visual data and the interpretation that each user makes in a given situation for the same data [Smeulders 00].

• Many initiatives in the last decade to bridge the gap: MPEG 7 transformations [Hunter 01, Celma 05]; COMM [Arndt07] by creating ontologies for multimedia.

• Methodologies for ontology engineering: METHONDOLOGY, On-To-Knowledge, DILIGENT, and recently NeOn Methodology.

7

A. Smeulders, M. Worring. Content-based image retrieval at the end of the early years. IEEE Trans. Pattern Anal. Mach. Intell., 22:1349–1380, December 2000.

Jane Hunter. Adding Multimedia to the Semantic Web - Building an MPEG-7 Ontology. In International Semantic Web Working Symposium (SWWS), Stanford, 2001.

R. Arnd R. Troncy. COMM: Designing a Well-Founded Multimedia Ontology for the Web. In 6th International Semantic Web Conference ISWC2007, Busan, Korea. Springer, 2007.


Maria

Falta la referencia de Celma, 05 y no todas tienen el mismo formato. Para añadir la cuarta yo haría más cortas las que tienen y las pondría en dos columnas de dos.

Maria

titulo

Maria

he visto un poco de jaleo con esta referencia, aqui no esta, luego aparece con la referencia a la tesis de MC pero aparece muchas veces y sólo algunas de ella aparece la referencia con los libritos. Mi sugerencia es: ya que aqui no te caben todas las de las metodologías, no ponerla aqui y ponerla a partir de la siguiente aparición, o bien en todas o bien solo en la primera. Pero es solo una opinión.

Introduction (VI)

8

Main objective: To search, find, analyze, rank and select suitable multimedia (MM) ontologies to be reused in the development of a multimedia ontology called M3 (Multimedia-Multidominio-Multilingüe)

Goal 1: To obtain a rank of MM ontologies to select the most appropriate ones that will be reused in the development of the M3 ontology.

Goal 2: To describe in detail and in a pedagogic way an example of how to apply the methodological guidelines for reusing ontologies in the multimedia domain.


Maria

yo no soy muy dada a tener que hacer click para la primera animación, o bien quitaría la animación o la configuraría para que empezara con la dispositiva sin que aparezca esta en blanco y tener que hacer un clik para la primera pastilla de texto.

9

Introduction (VII)

•Apply and extend Neon Methodology guidelines [Suárez-Figueroa, 2010] for reusing domain ontology in MM:

•domain ontology search: look for candidate domain ontologies that could satisfy the needs of the M3 Ontology.

•domain ontology assessment: find out if the set of candidate domain ontologies are useful for the development of the M3 Ontology.

•domain ontology selection: find out which domain ontologies are the most suitable for the development of the M3 Ontology.

•domain ontology integration: integrate the domain ontologies selected in the M3 Ontology.

General

Process

M.C. Suárez-Figueroa. PhD Thesis: NeOn Methodology for Building Ontology Networks: Specification, Scheduling and Reuse. España. Universidad Politécnica de Madrid. Junio 2010.



Outline

• Introduction





• Conclusions


Outline

• Introduction


• MPEG-7

• Ontologies describing MM objects

• Ontologies describing Shapes and Images

• Ontologies describing Visual Resource Object

• Ontologies describing Audio and Music

• Application Ontologies


MPEG 7 Standard: “Multimedia Content Description”

Descriptors Components

Visual Features Color, Texture, Shape, Motion, Localization, Face recognition.

Color Descriptors Color space, ColorQuantization, Dominant Colors, Scalable Color, Color Layout, Color-Structure, GoF/GoP Color.

Texture Descriptors Homogeneous Texture, Edge Histogram, Texture Browsing

Shape Descriptors Region Shape, Contour Shape, Shape 3D

Motion Descriptors Camera Motion, Motion Trajectory, Parametric Motion, Motion Activity

Localization Descriptors Region locator, Spatio-temporal locator

Audio Framework Basic (AudioWaveform, AudioPower), Basic Spectral, Timbral Temporal and Timbral Spectral

Ontologies for describing MM objects

13

•It is composed of multimedia patterns specializing the DOLCE design patterns for Descriptions & Situations and Information Objects, [Arndt et al., 07], OWL DL.•Scope covered: Multimedia, audio/music, image.•Ontological Resource reused: DOLCE, DnS, IO. •Non Ontological Resource reused: MPEG 7

COMM

•It is targeted for rich presentations in the web like SMIL, SVG and Flash.,[C. Scherp , A.Saathoff 10], OWL Full.•Scope covered: Multimedia, audio/music, image, videoOntological Resource reused: DOLCE & DnS Ultralight (DUL). •Non Ontological Resource reused: N/A

M3O

•It aims at integrating data resources related to media, especiallythose used on the Web. W3C initiative, OWL.•Scope covered: Multimedia, audio/music, video.•Ontological Resource reused: N/A. •Non Ontological Resource reused: SKOS

Media Onto

•MPEG-7_Hunter, MPEG-7x , MPEG-7_Tsinakari, MPEG-7_Rhizomik . [2001- 2006]•SWintO (mobile access, 2007): Multimedia, image, video. (RDFS)•Ontological Resource reused: DOLCE, SUMO. •Non Ontological Resource reused: MPEG 7

MPEG7 transformations

+ SWintO



Maria

puntos al final o no puntos al final, esa es la cuestión. por cierto hay algo raro en Flash.,[ muchos simbolos juntos, yo creo que sobra el punto o la coman;)

Ontologies for describing Shapes and Image

14

•Ontology for describing metadata for digital images, [Raphael Troncy + Ughent University, 07], OWL Full.•Scope covered: image.•Ontological Resource reused: N/A. •Non Ontological Resource reused: IIA

DIG 35

•It defines concepts including image, video, video frame, region, as well as relations such as depicts, regionOf, etc.,[Halaschek-Wiener et. al., 2005], OWL Full.•Scope covered: video, image•Non Ontological Resource reused: N/A

MIRO

•It combines high-level domain concepts and low-level multimedia descriptions, enabling for new media content analysis . [Kosmas Petridis et al., 2007] aceMedia Project ,OWL.•Scope covered: Multimedia, video.•Ontological Resource reused: DOLCE, MPEG-7(MDS)

MSO

•Metadata for any kind of shape •Creation and processing digital shapes. AM@SHAPE , 2005•Image, visual. (OWL Full)•Ontological Resource reused: N/A.

CSO, SAPO



Mari Carmen

faltarian algunas ontologias de las incluidas en el documento de la tesis

Ontologies for describing Visual Resource Object

15

•VRA is an Asociation maintining collections of slides, images and works of art. •Two versions of the ontology: SIMILE project (2003,RDFS ) and Assem (2005,OWL)•Scope covered: image, visual•Ontological Resource reused: N/A. •Non Ontological Resource reused: VRA Element Set

Vra Core 3

•Deals with semantic MM content, analysis and reasoning.•Developed within aceMedia Project, 2005•Scope covered: video, image.•Ontological Resource reused: DOLCE extension. •Non Ontological Resource reused: MPEG-7

VDOVDO



Mari Carmen

faltarian algunas ontologias de las incluidas en el documento de la tesis

Ontologies for describing Audio and Music

16

•Vocabulary for linking music-related information (production process, temporal aspect and events in music)•[Frederick Giasson, Yves Raimonf, 2010] in RFS•Scope covered: audio•Ontological Resource reused: Foaf, Time, Event, TimeLine •Non Ontological Resource reused: ABC Data Model.

Music Onto

•It describes classical music and performance.•Difference between musical works (e.g. Ballet) from performance (Ballet_Event), or works (Choral_Music)•[Kanzaki, 2005], OWL DL.•Scope covered: audio•Non Ontological Resource reused: N/A

Kanzaki’s Music Vocab

•It describes artists, music titles and some descriptors from the audio (tonality, rhythm, tempo )•[Oscar Celma, 2006] ,OWL DL•Scope covered: audio•Ontological Resource reused: FOAF•Non Ontological Resource reused: RDF Site Summary (RSS)

Recommendation

OntologyMusic



Application Ontologies

17

•Aims at cross-link of media campaigns over media TV, press and Internet•[MediaCampaign, 2006] in RFS•Scope covered: audiovisual•Ontological Resource reused: PROTON•Non Ontological Resource reused: NewsML, News Codes

MEPCO

•It describes athletics events (e.g. jumping, running, etc.) held in European cities.•[Boemie, 2008], OWL DL.•Scope covered: Multimedia, visual•Ontological Resource reused: GIO•Non Ontological Resource reused: TeleAtlas DB, MPEG-7, IAAF

AEO

•It aims at providing virtual representations of humans•[AM@SHAPE, 2007] ,OWL Full•Scope covered: image, visual•Ontological Resource reused: CSO•Non Ontological Resource reused: RDF Site Summary (RSS)

VHO



Conclusion SoA

• None of existing ontology integrate both low level descriptions (e.g., color, textures, fragments, etc.) and high level descriptions (voice, videoclip, slides presentation, domain content, etc.) of MM resources in all its five aspects (audio, video, image, visual, audiovisual, multimedia)

• None of the existing ontology describes MM resources in different domains and in different natural languages.



Outline

• Introduction





• Conclusions


Semantic Web Engines (SWEs)

Semantic Web Engines are applications for finding ontologies where queries are usually written as natural language keywords and results are ranked.

RDF-based search engines

Ontology-based search engines

Hybrid-based search engine


Selection of the most appropriate SWE

The total number of documents retrieved (T) for a specific keyword search.

The number of OWL documents per each 10 documents (OWL).

A valoration of the retrieval results using the symbols (+) and (-) of the result. We set to (+) if there are more than 2 OWL files per page, and (-) otherwise.

Terms used: Image, Multimedia, Audio, Music Style, Format

Set of criteria

SwoogleSwoogle

Maria

el tema de que aparece en blanco y hay que hacer click para la primera nimación.

22

Searching ontologies based on requirements


Functional requirements

ORSD

Non Functional requirements


Tasks for searching MM ontologies (I)

Terms translated into English

Terms extracted from the ORSD


Tasks for searching MM ontologies (II)


Tasks for searching MM ontologies (III)

But there are missing ontologies from SoA!!

25 ontologies retrieved with Swoogle


Tables of candidate MM ontologies: Unification process

List of 40 ontologies:

SWE + SoA

Mari Carmen

antes de esta faltaria una slide con las ontologias encontradas usando swoogle,y luego diciendo que en ese conjunto no estan algunas de las importantes en el area, y que por ello se ha compeltado el resultado de la busqueda en swoogle con ontologias encontradas en el estado del arte (papers, etc.)


Outline

• Introduction





• Conclusions


Analysis based on requirements (I)1-The competency questions (CQs) and one ontology selected from the searching activity. The result is a set of CQs identifiers that cover the given ontology.

3-Open the ontology to analyze in the Neon Toolkit. Open also the document with the list of CQs.


Analysis based on requirements (II)

4- For each CQs, detect the relevant categories and create a list of "Relevant Categories" (RelevCat). Example: "What are Audio Format", with the answer: "AVI, MP3"; RelevCat={Format, Audio, AVI, MP3}.

5- The matching task consists of finding for each term of the relevant categories, its presence in the ontology as a class or an individual. Update (CQ identifier)

Maria

veo el workflow subido un pelín, se mezcla con la linea roja del título y se nota el movimiento con el cambio de slide.


Assessment table/ ”useful” ontologies

Heuristic

IF (SimilarScope) OR (Similar Purpose) OR (Functional RequirementsCovered) = No

Then

NotUseful (CandidateOntology)EliminateFromSetCandidate(CandidateOntology)

Some wrong situationsSome wrong situations 26 “useful” ontologies:12: SoA

14: SWE

26 “useful” ontologies:12: SoA

14: SWE

[Suárez-Figueroa, 2010]


Outline

• Introduction





• Conclusions

32

Criteria for selecting MM ontologies



M.C. Suárez-Figueroa. PhD Thesis: NeOn Methodology for Building Ontology Networks: Specification, Scheduling and Reuse. Spain. Universidad Politécnica de Madrid. Junio 2010.


Determining the most appropriate MM ontologies. Considerations

1. Easy accessibility of the ontologies

2. Most of the ontologies were developed within a project or institutional initiatives (e.g: Boemie), highest scores in the Quality of the documentation, availability of external knowledge, and code clarity. The rest are made by academic researchers.

3. Some ontologies were developed or transformed by one author reputation and purpose reliability lower than others ontologies.

4. In ”practical support”, very relevant others publications referencing the ontology or the use of the same ontology in a large project (e.g.: COMM, SAPO, Boemie VDO)

5. Difficult to know if the ontologies were tested and/or evaluated after their implementation

Mari Carmen

esta slide tiene demasiado texto; debes abstraer y resumir el texto que qieras realmente mostrar.

34

Determining the most appropriate MM ontologies (I)


35

Determining the most appropriate MM ontologies (II)


)()( ,

jj

j

jTi Weight

WeightValueScore

ji

)(,)(, iii ScoreScoreScore

Value = Unknown ValueT = 0Value = Low ValueT = 1Value = Medium ValueT = 2Value = High ValueT = 3

Formulae to rank ontologies


36

Integrating the MM ontologies reused. General vision


Music Ontology: 1 CQ covered

Media Ontology: 4 CQs coveredMedia Ontology: 4 CQs covered

COMM: 5 CQs coveredCOMM: 5 CQs covered

Boemie VDO: 4 CQs coveredBoemie VDO: 4 CQs covered

They cover 70% of the CQs!!

They cover 70% of the CQs!!

37

Integrating the MM ontologies reused. Overview of the M3 Ontology



Outline

• Introduction





• Conclusions

• We used the NeOn Methodology [Suárez-Figueroa, 2010] to perform a systematic analysis of all the candidate ontologies.

• Methodological guidelines for reusing domain ontologies.

• More specifically, we have been focused on:

• (1) searching for ontological resources in repositories and registries 40 ontologies found.

• (2) assessing the ontological resources in order to find out if such resources satisfy the developers needs 23 ontologies obtained.

• (3) comparing the ontological resources on the basis of a set of criteria and selecting the most appropriate ones based on the requirements: improving and extending them with specific rules to analyze the ontologies Ontologies ranked.

• (4) integrating the ontological resources: selection of 4 suitable ontologies to be reused in the M3 Ontology.

39

Conclusions

What we have done in this master thesis


M.C. Suárez-Figueroa. PhD Thesis: NeOn Methodology for Building Ontology Networks: Specification, Scheduling and Reuse. Spain. Universidad Politécnica de Madrid. June 2010.

• Searching activity:• An overview of MM ontologies in the literature.

• How to select an appropriate SWE to retrieve relevant ontologies in the MM domain.

• Workflow to search relevant ontologies based on set of terms extracted from the CQs.

• Assessing activity:• A comparative framework for MM ontologies.

• Workflow to check if an ontology fits the requirements.

• Selecting activity:• Adaptation of the criteria for selection proposed by [Suárez-Figueroa, 2010]

to the MM domain.

• Inclusion of a new criteria.

• Integration and implementation of the M3 Ontology.

40

Conclusions (II)

Main contributions


Many of the processes described in the ontology reuse activities are described in natural language need to be formalized and automatized.

Searching activity is not exclusive to the used Semantic Search Engines, and must be extended to articles, project web pages, and W3C groups related to the domain.

Semantic Web Engines do not clearly distinguish in the results from keywords queries, RDF data coming from blogs and DBPedia resources to ontologies documents implemented in OWL.

Some criteria proposed in [Suárez-Figueroa, 2010] concerning ranking ontologies need to be adapted to the domain of the ontology being developed.

There is lack of multilingual ontologies in multimedia domain.

41

Conclusions (III)

Lessons learned

M.C. Suárez-Figueroa. PhD tesis: NeOn Methodology for Building Ontology Networks: Specification, Scheduling and Reuse. Spain. Universidad Politécnica de Madrid. Junio 2010.


42

Future Work (I)

How to choose the right SWE that gives better results? What could be the criteria that guide deciding which SWE to use in function of the domain?

CQs Enhancing Ontology Search Tasks:

Many SWE presents their results mixing documents from blogs with ontologies. An API can help the developer to extract efficiently disseminated ontologies in the whole documents retrieved by a SWE.

How to select the right ontology from the results retrieved by search engines

With the continuously growing of the DBPedia resources, analyze how to populate ontologies and their reliability with respect to the one to be built.

Semi-automatic ontology population:


43

End

Thanks!


Date: 11/04/23Speaker: Ghislain Auguste Atemezing


Master ThesisMáster de investigación en inteligencia artificial

Author: Ghislain Auguste Atemezing

Supervisor: Dr. María del Carmen Suárez de Figueroa Baonza

Education

Analyzing and Ranking Multimedia Ontologies for their Reuse