Benchmarking the Interoperability of Ontology Development Tools

Interoperability Benchmarking. October 10th 2005 1 © Raúl García-Castro

Benchmarking the Interoperability of Ontology Development Tools

Raúl García-Castro

<[email protected]>

October 10th, 2005

InteroperabilityWorking DaysOctober 10th-11th, 2005


Ontologies are available in internet

Import

Evaluate

Prune

Merge

Alignment

Identify Differences

Specialize

Extend

Evolution

Export

Conceptualiz

e

Document

Integrate

Anotate

Reasoning

Specify

¿=?

+

O1O2

O3

...


Interoperability problem

• It appears due to ontology reuse.

Tool 1

Tool 2

Tool 3

Tool 4

Tool 5

Ontology development tools

Potential functionalities

Real functionalities


The great diversity in languages and tools originates two problems:– The translation problem:

• How can we translate an ontology between two different languages without losing knowledge?

– The interoperability problem: • How can two tools interchange ontologies or parts of them

without losing knowledge?• How can a tool use ontologies or parts of them from other tool?


Why is it difficult?• Different KR formalisms

frames description logicsconceptual graphsfirst order logic semantic networks

• Different modelling components inside the same KR formalism



• Some results:– It is difficult to preserve the semantics and the

intended meaning of the ontology– Interoperability decisions…

• At many different levels

• Usually hidden in the programming code of ontology exporters/importers

O. Corcho. A Layered Declarative Approach to Ontology Translation with Knowledge PreservationFrontiers in Artificial Intelligence and Applications, Volume 116, January 2005


Do Tools interoperate?

RDF(S)OWL

...

Ontology development tools

Loom

OILOWL

Ontolingua

RDF(S)

OWL

Query systems

Flogic

DAML+OIL

RQLRDQLSeRQL

SPARQL

Loom

Loom Classifier

OntoBrokerFlora, Flora2

FORID

SilRIRIL

TRIPLE

OIL

FACTRACER

FACTRACERTRIPLE

BOR

DAML+ OIL

FACTRACERTRIPLE

JTP

DQL

OWL-QL

Flora, Flora2

Reasoners

DAML+OIL

ICS-FORTH Validating RDF Parser,RDF Validation Service

DAML Validator,DAML+OIL Ontology Checker

OWL Ontology Validator,OWL Validator

Evaluation tools

Flogic

OWL

OWL

RDF(S)

RDF(S)

RDF(S)

Inter

oper

abili

ty

is cr

ucial

in th

e sem

antic

web


Interoperability approaches

Alternative 1. Ontology interchange using an interchange language Tool i

Interchange language

Tool j

Alternative 2. Direct interchange between each two

Interoperability through application program interfaces (APIs)

Common API

T 3T 2T 1

T 6T 5T 4

Alternative 3. Common API

Interoperability through an interchange language

T 3

T 2T 1

T 4

T 5 T 6


Knowledge models comparison

Classes

Template Slots/properties/instance attributes

InstancesData types

RDF(S)

Subclass-of

Subproperty-of

Literals

Containers

Collections

Statements

Concept groups

Disjoint decompositions

Exhaustive decompositions

Partitions

Constants

Relation properties

Synonyms

Abbreviations

Bibliographic references

Metaclasses

Own slots

Class attributes


Translation strategies

Thesis

MSc Thesis PhD Thesis

Disjoint-subclass

Partial loss

Total lossDon’t export

Insert ad-hoc RDF(S)

<rdfsClass rdf:about=“#Thesis”> <a:disjoint rdf:Resource=“#MsC Thesis”> <a:disjoint rdf:Resource=“#PhD Thesis”></rdfs:Class>

Doesn’t import

Thesis

RDF(S) IMPORTEXPORT

Thesis


subclass

Minimize knowledge loss

Thesis


subclass

Thesis


Interoperability using an interchange language

Interoperability is not guaranteed with the existence of importers and exporters

from the tools to the interchange language.

Interoperability using an interchange language depends on the correct working of

the translators…

…and these translators don’t work properly.

RDF(S)OWL

... Loom

OILOWL

Ontolingua

RDF(S)


How can we improve this situation?

Goal 1:• To assess and improve the interoperability of ontology development tools using

RDF(S) for ontology exchange.

Goal 2:• To identify the subset of RDF(S) elements that ontology development

tools can use to correctly interoperate.

Goal 3:• Next step: OWL.

Benchmarking: • Evaluation of the interoperability• Improvement of the interoperability• Collection of best practices


General framework for benchmarkingBENCHMARKING ITERATION

Recalibration task

PLAN PHASE

1. B. goals identification 2. B. subject identification 3. Participant identification 4. B. proposal writing 5. Management involvement 6. B. partner selection 7. B. planning and resource allocation

IMPROVE PHASE

11. B. report writing 12. B. findings communication 13. Improvement planning 14. Improvement 15. Monitor

EXPERIMENT PHASE

8. Experiment definition 9. Experiment execution 10. Experiment results analysis

General evaluation criteria:• Interoperability• Scalability• Robustness

Benchmark suites for:• Interoperability• Scalability• Robustness

Benchmarking supporting tools:• Testing frameworks• Workload generators• Monitoring tools• Statistical packages

García-Castro, Maynard, Wache, Foxvog and González-Cabero. Knowledge Web Deliverable 2.1.4 Specification of a methodology, general criteria, and benchmark suites for benchmarking ontology tools. December 2004.


Plan phase

Benchmarkinggoals

identification

Benchmarkingsubject

identification

Participantidentification

Need for benchmarking

Organisation goals and strategies Benchmarking

proposal writing

Benchmarking goals, benefits, costs

Benchmarking subject, tool functionalities, evaluation criteria

List of involved members, benchmarking team

Management involvement

Benchmarking partner selection

Benchmarking planning and

resource allocation

Benchmarking proposal

Management support

Benchmarking partners, updated benchmarking proposal

Benchmarking planning

Organisation's tools

Tools from outside the organisation

Organisation planning

Improve the interoperability of ontol. development tools

RDF(S) import and export capabilities

Identify ontology components

exported/imported

B.P.


Experimentdefinition

Experimentexecution

Experimentanalysis

Benchmarking planning

Benchmarking proposal Experiment

definition, experimentation planning

Experiment results

Experiment report

Experiment phase

RDF(S) Import benchmark suitesRDF(S) Export benchmark suites

• test 1• test 2• test 3• ...


NOOKOK


OKNOOK

OKOKNO ...



NOOKOK


OKNOOK

OKOKNO

E.R.


Benchmarkingreport writing

Benchmarking findings

communication

Improvementplanning

Updated benchmarking proposal

Experiment report

Improvement

Benchmarking report

Updated benchmarking report

Monitor

Necessary changes, improvement planning, improvement forecast

Organisation support

Improved tool

Monitorisation report

Improve phase

• Comparative analysis• Compliance with standards• Weaknesses• Recommendations on tools• Recommendations on practices


Identification of the evaluation elements and metrics

GQM paradigm: Any software measurement activity should be preceded by:

1.- The identification of a software engineering goal ...

2.- ... which leads to questions ...

3.- ... which in turn lead to actual metrics.

To improve the interoperability of ontology development tools using RDF(S) as an interchange language

Q1: Which are the elements of the knowledge model of a tool that are interchanged with another tool without knowledge loss?

Q2: Which are the elements of the knowledge model of a tool that are interchanged with another tool with knowledge loss?

Q3: Which are the elements of the knowledge model of a tool that are not interchanged with another tool?

M1: Elements of the knowledge model of a tool that are exported to RDF(S)

M5: Elements of the knowledge model of a tool that are not exported to RDF(S)

M2: Knowledge loss when exporting to RDF(S) an element of the knowledge model of a tool

M3: Elements of the knowledge model of a tool that are imported from RDF(S) intro another tool

M6: Elements of the knowledge model of a tool that are not imported from RDF(S) intro another tool

M4: Knowledge loss when importing from RDF(S) into another tool an element of the knowledge model of a tool


Evaluation infrastructure

Ontology Comparer

Ontology Loader

Ontology Development

Tool

Ontology Storer

Import Benchmark

Suite

Export Benchmark

Suite

RDF(S) Comparer

Import Benchmark

Suite Executor

Export Benchmark

Suite Executor

Ontology Repository

RDF(S)File

Repository

Export experiments

Import experiments


Export experiments

Ontology Loader


Tool

Export Benchmark

Suite

RDF(S) Comparer

Ontology Repository

RDF(S)File

Repository

Concept 1

Concept 2 Concept 3 Concept 4

= ?YES

NO

<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"> <rdfs:Class rdf:about="http://www.pru.com/ontology#concept1“ /> <rdfs:Class rdf:about="http://www. pru.com/ontology#concept2"> <rdfs:subClassOf rdf:resource="http://www. pru.com/ontology#concept1"/> </rdfs:Class> <rdfs:Class rdf:about="http://www. pru.com/ontology#concept3"> <rdfs:subClassOf rdf:resource="http://www. pru.com/ontology#concept1"/> </rdfs:Class></rdf:RDF>




Import experimentsOntology Comparer


Tool

Ontology Storer

Import Benchmark

Suite

Ontology Repository

RDF(S)File

Repository


Concept 1

Concept 2 Concept 3 Concept 4

YESConcept 1

Concept 2 Concept 3 Concept 4= ?

Concept 1

Concept 2 Concept 3 Concept 4 NO


Experimentation in the benchmarking

• Agreement phase. The quality of the benchmark suites is essential for the results.

• Evaluation phase 1. The RDF(S) importers and exporters of the ontology development tools are be evaluated.

• Evaluation phase 2. The ontology exchange between ontology development tools is evaluated.

WE ARE HERE


Benchmarking the Interoperability of Ontology Development Tools

Raúl García-Castro

<[email protected]>

October 10th, 2005

InteroperabilityWorking DaysOctober 10th-11th, 2005

Documents

Benchmarking the Interoperability of Ontology Development Tools