Volume xx (200y), Number z, pp. 1–21

Ontology-based Representation and Modelingof Synthetic 3D Content: a State of the Art Review

Jakub Flotynski, Krzysztof Walczak

Poznan University of EconomicsNiepodległosci 10, 61-875 Poznan, Poland

{flotynski, walczak}@kti.ue.poznan.pl

AbstractAn indispensable element of any practical 3D/VR/AR application is synthetic 3D content. Such content is charac-terized by a variety of features - geometry, structure, space, appearance, animation and behavior - which makesthe modeling of 3D content a much more complex, difficult and time-consuming task than in the case of othertypes of content. One of the promising research directions aiming at simplification of modeling 3D content isthe use of the semantic web approach. The formalism provided by semantic web techniques enables declarativeknowledge-based modeling of content based on ontologies. Such modeling can be conducted at different levels ofabstraction, possibly domain-specific, with inherent separation of concerns. The use of semantic web ontologiesenables content representation independent of particular presentation platforms and facilitates indexing, search-ing and analyzing content, thus contributing to increased content re-usability. A range of approaches have beenproposed to permit semantic representation and modeling of synthetic 3D content. These approaches differ in themethodologies and technologies used as well as their scope and application domains. This paper provides a reviewof the current state of the art in representation and modeling of 3D content based on semantic web ontologies,together with a classification, characterization and discussion of the particular approaches.

Categories and Subject Descriptors (according to ACM CCS):Computer graphics [I.3.7]: Three-Dimensional Graphics and Realism—[Virtual reality]; Information interfacesand presentation [H.5.1]: Multimedia information systems—[Artificial, augmented, and virtual realities]; Infor-mation interfaces and presentation [H.5.2]: User Interfaces—[Graphical user interfaces (GUI)]Virtual reality

1. Introduction

Widespread use of interactive 3D technologies, such as vir-tual (VR) and augmented (AR) reality, has been enabled bythe significant progress in hardware performance, the rapidgrowth in the available network bandwidth as well as theavailability of versatile input-output devices. VR/AR sys-tems become increasingly popular in various application do-mains, such as education, medicine, training, tourism, en-tertainment and cultural heritage. In comparison to othertypes of applications, VR/AR applications are equipped withmore advanced user interfaces, which offer the possibility ofpresenting data in the form of animated three-dimensionalmodels with complex behavior, permit flexible interactionof users with the presented models and enable combiningthe presented models with a view of the real world.

The primary element of VR/AR applications, apart fromthe interface technologies, is interactive synthetic three-dimensional (3D) content presented to users. A number ofprogramming libraries (e.g., OpenGL [ope], Direct3D [dir]and Java3D [Ora]) have been developed to enable imperativeprogramming of 3D content with widely-used programminglanguages (e.g., C++ and Java). Imperative programming ofcontent is based on the specification of subsequent steps thatmust be performed to achieve the desirable effects. Further-more, a few declarative languages, such as VRML [W3Cc],X3D [W3Cd] and COLLADA [Col], have been devised forprogramming 3D content. In contrast to imperative program-ming, declarative programming of content is based on thespecification of the desirable results that should be achieved,but not the steps that must be accomplished to achieve the re-sults. The availability of standards for 3D content creation on

submitted to COMPUTER GRAPHICS Forum (1/2017).

Accepted for publication in: Computer Graphics Forum, Wiley, ISSN 0167-7055

2 Jakub Flotynski, Krzysztof Walczak /

the web allows for wide dissemination of content—openingentirely new fields of application.

Synthetic 3D content may be characterized by a varietyof features such as geometry, structure, space, appearance,animation and behavior. This diversity, however, makes themodeling of synthetic 3D content much more complicatedthan the creation of typical multimedia content such as webpages, images, audio and video. Therefore, the potential ofVR/AR applications can be fully exploited only if accompa-nied by efficient methods of modeling 3D content.

A number of visual environments have been developedfor modeling 3D content. Advanced environments, whichare intended for professional users (e.g., Blender [Ble] and3ds Max [Auta]) offer rich capabilities for designing vari-ous 3D content elements, such as geometry, structure, space,appearance, animation as well as behavior described by im-perative scripts. However, their complexity requires author’sexpertise in computer graphics and animation. Modeling ofcontent may be simplified by narrowing the domain of ap-plication and the set of available operations. Environmentsof this type (e.g., Ghost Productions [gp1], Sweet Home 3D[sh1] and AutoCAD Civil 3D [Autb]), designed for domainexperts (e.g., physicians, interior designers and engineers),provide tools enabling relatively fast and efficient modeling,without requiring users’ extensive experience in 3D contentcreation. This approach, however, significantly reduces thegenerality of the method. Regardless of the generality of themodeling method used, graphical modeling of content withthe aforementioned approaches demands users’ knowledgeof issues related to computer graphics, and it requires theexplicit creation of all content elements to be presented.

Further simplification of modeling content requires sep-aration of concerns between users with different expertise,who are equipped with different modeling tools. Previousworks in this area are based on parametrized templates (e.g.,[WC03, Wal12b]) and content composition from reusableelements (e.g., [DHM02, Wal06, Wal08, Wal12a]), with theseparation of concerns between experts in 3D modeling andexperts in the particular application domain. These solutionssignificantly facilitate modeling of 3D content, but they stillhave important limitations, which result from their orienta-tion on the modeling of content instead of the modeling ofconcepts. First, modeling experts do not use methods spe-cific to a particular application domain, but methods specificto 3D content. Second, it is still necessary to explicitly de-scribe all elements of the created 3D content. Third, the con-tent is modeled in its final form, not allowing end-users tohave influence on the final result.

Further progress in modeling of 3D content is possiblethrough the use of semantics [VGLM∗07,ZDLB08], broadlyunderstood as concepts describing different content featuresat different levels of abstraction. Currently, the main ap-proach to describe the semantics of content is the semanticweb. The research on the semantic web was initiated by the

W3C (World-Wide Web Consortium) in 2001 [BLHL∗01].This research aims at evolutionary development of the cur-rent web towards a distributed semantic database linkingstructured content and documents. In semantic web, re-sources of different types (web pages, images, audio, video,3D content) are described using ontologies. Ontologies arespecifications of conceptualization, in which objects, con-cepts, and other entities that are assumed to exist in somearea of interest and the relationships that hold among themare included [Gru95, GN87]. Ontology-based description ofweb content makes it understandable for both humans andcomputers achieving a new quality in building applicationsthat can ’understand’ the meaning of particular elements ofcontent as well as their relationships, leading to much bettermethods of searching, reasoning, combining and presentingthe content.

The research on the semantic web have also influencedcomputer graphics. Ontology-based annotation, extraction,exchange, retrieval and creation of 3D content [DFS07,SF09,LB08] have been regarded as the main issues related tomodern 3D/VR/AR applications [DFS07, SF09, LB08] andan important step towards building the 3D internet [ABK07].Research is also conducted in ontology-based modeling of3D content behavior as well as relations between 3D modelsin 3D scenes [TBSDK08].

The paper addresses two groups of works on the use ofontologies in computer graphics. The first group encom-passes ontologies that enable description of content featuresrelated to content visualization. Such description is referredto as ontology-based representation of 3D content in this pa-per. The second group encompasses methods and tools thatenable creation of 3D content using ontology-based repre-sentations. Such process of content creation is referred toas ontology-based modeling of 3D content in the paper. Incontrast to the previous approaches, semantic web ontolo-gies enable representation and modeling of content in theapplication domain, without loss of the generality of the ap-proach used. The available ontology-based approaches differin the methodologies and technologies used as well as theirscope and application domains. However, an extensive anal-ysis and summary of the current state of the art in this fieldis still missing. In this state-of-the-art report, a taxonomy ofapproaches to ontology-based representation and modelingof 3D content is proposed. The taxonomy has been used toclassify and discuss the available approaches within differ-ent categories.

A number of semantic approaches that are not based onsemantic web ontologies have been developed. In particular,approaches based on MPEG-7 [Hun01], UML [TQDLC10],CityGML [GP12] and several metadata standards, have alsoaddressed 3D content semantics understood as domain-specific description. Nevertheless, they lack formalism nec-essary to permit reasoning on the described content and in-ference of implicit knowledge. This is the most important

submitted to COMPUTER GRAPHICS Forum (1/2017).

Jakub Flotynski, Krzysztof Walczak / 3

difference between the semantic web ontologies and the pre-vious approaches, which classifies those approaches out ofthe scope of this paper. Moreover, various approaches to ex-traction of semantics from 3D models and reasoning havebeen developed before semantic web ontologies have be-come available. Such solutions are not covered by the pro-posed classification, but some of them are referred in thiswork as the precursors of the semantic web.

The paper is structured as follows. In Section 2, the con-cept of the semantic web, including ontologies and the tech-niques for building ontologies is explained. In Section 3, theconcept of ontology-based modeling of 3D content is ex-plained with its motivations and an example application. InSection 4, approaches to ontology-based representation andmodeling of 3D content are classified according to the pro-posed taxonomy. Finally, in Section 5, the review is summa-rized, and the main open challenges in the field are indicated.

2. The Semantic Web Approach

2.1. Ontologies

Ontologies are sets of statements and rules intended to beunderstandable and processable by both humans and com-puters. Ontologies constitute the foundation of the seman-tic web across diverse domains and applications [W3Cb].Every ontology can consist of different types of state-ments that can either define concepts or describe facts re-lated to objects. Terminological statements (T-Box) describeconceptualization—classes and properties [GL96], whichcan be used to specify schemes of content – describe whatare the possible constraints on and relations between differ-ent types of content elements, without specifying any par-ticular elements (Figure 1). Assertion statements (A-Box)describe utilization—content elements (also called individ-uals), which are instances of classes. Content elements aredescribed using properties with particular values assigned.In 3D modeling, ontologies comprised of T-Box statements(in short T-Box ontologies) are schemes of 3D scenes orscene elements. For instance, a T-Box ontology specifies aclass of virtual museum exhibitions with different classes ofartifacts such as statues, stamps and coins as well as pos-sible spatial properties of the artifacts—they are placed onstands [FW16]. In general, such a schema can be fulfilled bymultiple 3D scenes. Ontologies comprised of A-Box state-ments (in short A-Box ontologies) describe particular 3Dscenes or scene elements. For instance, an A-Box ontologydescribes a particular virtual museum exhibition (individual)with artifacts (other individuals) that satisfy the conditionsgiven in the T-Box ontology – belong to the particular classesand are placed on stands, which is described by appropriatevalues of properties.

Ontologies are used in two different ways in com-puter graphics—to describe content metadata and to rep-resent the content. The ontologies used to describe content

metadata (e.g., topic, author and creation date), [ATS∗07,W3Ca, W3Ce], can be considered as advanced metadataschemes (T-Box ontologies) and metadata descriptions (A-Box ontologies), which have been widely used in particularfor 3D content annotation and retrieval [ARC, WWWC04,WMD∗04] [DAD∗11] [WCSS06]. The ontologies used torepresent 3D content (e.g., shapes, textures and transfor-mations) can be considered as advanced encoding formats(T-Box ontologies) and 3D model or scene descriptions(A-Box ontologies). In both cases—metadata descriptionand content representation—the main advantage of ontolo-gies over typical metadata and 3D formats is the formalsemantics, which enables inference of implicit knowledge.However, only ontology-based content representation is thedirect basis for modeling of 3D content. An ontology of 3Dcontent is a T-Box ontology specifying classes and proper-ties that enable 3D content representation at particular levelsof abstraction. A-Box ontologies that are instances of 3Dcontent ontologies are ontology-based 3D content represen-tations of 3D models and 3D scenes. In the remaining partof the paper, the word semantic means specified using on-tologies.

Semantic 3D Modeling

Ontology-based 3D Modeling

3D Content Ontology

T-Box Ontology

Ontology-based 3D Content Representation

Is an

Is an

Is

Uses

3D Content Schema(format or language)

3D Content (3D model or scene)

Equivalent of

Is a

DataModeling technique

Is

3D Modeling Uses

Instance of

Instance of

Ontology

Is a

A-Box Ontology

Is an

Instance of

Figure 1: Relationships between the concepts of representa-tion and modeling of content

Ontologies can be used to represent content internally andexternally. An internal ontology-based content representa-tion is associated with a primary content representation en-coded in a specific 3D format (e.g., VRML, X3D, COL-LADA). Sometimes, it is syntactically built-in into the con-tent encoded in a 3D format [FW13c, FW13d]. Taking intoaccount that the primary representation is sufficient for con-tent visualization (includes all necessary content features),an internal ontology-based representation incorporates re-dundant information (e.g., [FSA∗04], [APP∗05]). Such rep-

submitted to COMPUTER GRAPHICS Forum (1/2017).

4 Jakub Flotynski, Krzysztof Walczak /

resentations enable access to 3D content with semantictools, in particular in semantic annotation, search and re-trieval. For instance, an internal representation specifies suchproperties of content as the number of triangles, verticesand edges to facilitate its search and retrieval [DFHP∗07].An external ontology-based content representation replacesthe primary content representation in a 3D format or ex-tends it by adding new content properties (e.g., [BPKDT04],[VGRP∗10], [AWGH11]). External representations add in-terpretation to elements and properties of 3D content. Forinstance, an ontology links particular meshes in 3D scenesto particular pieces of furniture in a virtual shop to enableintuitive search and presentation to customers [DTKM∗07].

2.2. Techniques

The primary semantic web technique for building ontologiesfor different types of content is the Resource DescriptionFramework (RDF) [W3Cg]. RDF is a data model, which en-ables creation of statements on resources. Each statement isa triple comprised of: a subject (resource described by thestatement), a predicate (property of the subject) and an ob-ject (the value of the property describing the subject). RDFintroduces basic concepts for describing resources, such asdata types, containers and lists. RDF-based descriptions canbe embedded in web pages using RDFa [W3Ch]. The RDFSchema (RDFS) [W3Ci] and the Web Ontology Language(OWL) [W3Cf] are semantic web standards based on RDF.These standards improve the expressiveness of RDF by pro-viding additional concepts, such as hierarchies of classes andproperties, constraints, property restrictions and operationson sets. The Semantic Web Rule Language (SWRL) [W3Cj]and the Rule Markup Language (RuleML) [rul] extend OWLwith rules [Llo87]. Another rule language used in the con-text of 3D modeling is the Access-Limited Logic. Whereas astatement describes a fact, a rule describes an implication be-tween the facts included in the ascendant (body) and the factsincluded in the descendant (head). For instance, if an individ-ual is of the atom class (body), generate a sphere to representit (head) in an ontology for chemical compounds [KCM06].

3. Ontology-based Modeling of 3D Content

Ontology-based modeling of 3D content is performed withrespect to the semantics of 3D content elements, their classesand properties, which are specified in 3D content ontologies.Typically, the result of ontology-based modeling is final 3Dcontent encoded in a 3D format or language (e.g., VRML,X3D, JavaScript) understandable to 3D browsers (e.g., Cor-tona, BS Contact or an internet browser). Ontology-basedmodeling emphasizes the specification of the desirable pre-sentational effects to be achieved instead of the concrete se-quence of instructions that must be performed to achievethese effects. Hence, ontology-based modeling, along withmodeling by constraints (e.g., [XSF02, LRGC04]), can beregarded as a specific kind of declarative modeling [Gai07].

3.1. Relation to Semantic Modeling

Ontology-based modeling of 3D content is a specific typeof broader semantic modeling, which has been introduced in[FS98] as a sequence of transitions between four universes:shape universe and knowledge domain, mathematical uni-verse, representation universe and implementation universe.In the first universe, elements of 3D content are coupled withdomain knowledge to provide a conceptual 3D world that isrepresented at an arbitrarily chosen level of abstraction. Aconceptual world may be directly associated with an appli-cation domain, e.g., virtual museum, building informationmodels and interior design. In the mathematical universe, amathematical model is used to represent the content. For in-stance, terrain may be modeled using bi-dimensional scalarfields. A mathematical model may be transformed into dif-ferent representation models, e.g., solids may be representedusing constructive solid geometry or boundary representa-tion. Finally, a representation model may be transformedinto different implementation models (data structures), e.g.,a mesh is encoded using a structure with fields containingthe coordinates of the vertices. Taking into account the afore-mentioned description, semantic modeling of 3D content canbe seen as an extension of geometric modeling. While in ge-ometric modeling users leverage concepts that are directlyrelated to computer graphics, in semantic modeling usersleverage concepts that are directly related to a selected ap-plication or domain. In semantic modeling, the linkage ofsuch application- or domain-specific concepts to graphicalconcepts (which are necessary for final visualization) is pro-vided in advance or the transformation is performed trans-parently from the users’ point of view.

Incorporating domain or application knowledge into 3Dcontent, which occurs in semantic modeling, is one of themain aspects of building intelligent virtual environments,which may be explored based on the meaning of their ele-ments. It exceeds the capabilities of the available 3D formats(including scene graphs) and requires new specific methodsof mapping conceptual objects onto collections of graphicalprimitives [LA00]. The semantic representation of a virtualenvironment can be created as an additional level of the ap-plication, enabling conceptual processing of 3D content at ahigher level of abstraction [AC01], e.g., actors, actions andfeatures.

3.2. Preliminaries

Ontology-based modeling, in which A-Box ontologies (3Dmodels and scenes) are created, is typically preceded by thedesign of T-Box ontologies (schemes of the content). The de-sign of an ontology used in modeling covers: specification ofa target domain, identification of applications, gathering de-velopers’ requirements for the content to be created with theontology, identification of the key concepts to be includedin the ontology, elicitation of competency questions for theontology, and the initial design of the ontology [PAMR05].

submitted to COMPUTER GRAPHICS Forum (1/2017).

Jakub Flotynski, Krzysztof Walczak / 5

Design of 3D content ontologies is usually performed us-ing ontology editors, such as Protégé [pro]. The modelingprocess is always limited by the concepts included in theused ontology. The more specific to 3D graphics and ani-mation the ontology is, the more potential use cases it has.However, such general-purpose low-level ontologies do notincorporate domain-specific elements, and thereby do not fa-cilitate content creation by domain experts. Therefore, anontology for 3D modeling should keep balance between theexpected level of domain representation and the level of con-tent details that can be manipulated with the ontology. Tosatisfy this requirement, it is crucial to take into account re-quirements of the target users of the ontology. Such require-ments should cover the planned use cases as well as classesand properties of content elements, which will enable theuse cases. For instance, an ontology for modeling virtualmuseum exhibitions by curators should include classes andproperties of artifacts and skip concepts specific to their low-level graphical representation [Flo14].

3.3. Motivations

Ontology-based modeling of 3D content offers several ad-vantages over the other approaches to modeling.

Declarative content representation is comprised of logi-cal statements and rules, which significantly differ from in-structions of imperative programming languages [VRH04].First, the syntax of statements (triples) and rules (implica-tions) is less complex then the syntax of imperative instruc-tions (e.g., loops and conditions in C++ and Java). Second,the order of appearance of statements and rules in ontolo-gies is not important, in contrast to the order of instructionsin imperative content representations. Therefore, declarativecontent representation can be more intuitive, in particularfor content authors who are not programmers. It also signif-icantly facilitates automatic content management (indexing,searching and analyzing) in content repositories in compari-son to imperative representation.

Conceptual content representation covers 3D content atdifferent levels of abstraction by the use of classes, prop-erties and individuals. Such concepts can be either equiv-alents to basic low-level 3D content elements (e.g., shapesand textures) described by low-level 3D content proper-ties (e.g., coordinates and indicies) [Gut05] or high-levelelements (e.g., body parts) described by high-level proper-ties (e.g., joint attributes) [GTV∗05b]. The possible spec-trum of high-level content representations is practically un-limited and encompasses such applications and domains ascultural heritage [PDF06], interior design [Ott05a], simula-tion [KLN∗10] and e-commerce [DTKM∗07]. Although im-perative programming languages permit creation of concep-tual content representations at different levels of abstraction,the representations are expressed imperatively, which is not

convenient for automatic analysis and knowledge inference,thus reducing possible use in content repositories on the web.

Reasoning-based content creation enables inference ofimplicit knowledge including properties of content, whichmay be hidden (not explicitly specified), but are the en-tailment of explicitly specified statements and rules [FS98].Such implicit properties can be automatically inferred on thebasis of the explicit properties, liberating developers fromspecifying all details of the created content [Ple99, PM08].In modeling of 3D content, reasoning is especially usefulwhen the description of desirable results is simpler than thedescription of the steps that must be completed to achievethe results. This usually appears in two cases. First, whenthere are multiple conditions on the content that can be sat-isfied in different manners. For instance, to place artifacts onstands in a VR museum, it is sufficient only to indicate thatone artifact is associated with one stand [WF15b, FW16],without designating explicit assignment. Second, reasoningis especially useful when modeling of content requires com-plex processing of large structures of content elements. Forinstance, to present only the content related to the sovereignsbeing successors to a particular king, it is sufficient onlyto indicate the king and to define the successor relation-ship [WRF14], without explicit verification of all potentialcandidates. In both cases, only the required solution is de-scribed, and computations are completed by a reasoning en-gine, which infers implicit properties. The explicit imple-mentation of algorithms solving both types of problems istypically problematic using imperative programming lan-guages, in which all necessary subsequent steps must be de-scribed. Currently, reasoning is supported neither in impera-tive programming languages (e.g., C++, Java) nor in declar-ative 3D content languages (e.g., VRML, X3D).

Separation of concerns enables completion of substan-tially different tasks related to different features of the con-tent by different modeling users, who have different skillsand are equipped with different modeling tools [BQLC03].Although, available modeling tools (e.g., Blender and 3dsMax) enable allocation of tasks among different users (e.g.,creating different content elements or writing different partsof code), the tasks typically require expertise in computergraphics and animation from all the users involved.

Platform-independent content representation that fol-lows the independence of the semantic web ontologiesfrom particular hardware and software platforms. Therefore,ontology-based 3D content representations can be trans-formed to different content formats and languages specificto different presentation platforms [FW14c]. Although 3Dmodeling tools often support different content formats andlanguages, and some of them (e.g., Blender and 3ds Max)enable introduction of new formats and languages by im-plementing appropriate plug-ins, they do not enable genericcontent transformation, which would be described by rules

submitted to COMPUTER GRAPHICS Forum (1/2017).

6 Jakub Flotynski, Krzysztof Walczak /

independent of particular content formats or languages. Themore generic the description of content transformation to fi-nal encoding, the better are the possibilities of overall dis-semination of 3D content on the web.

3.4. Modeling Activities

On the basis of the available literature, ontology-based mod-eling of 3D content can be regarded as a process encom-passing the following four activities: semantic reflection, se-mantic selection, semantic configuration and semantic trans-formation of 3D content. Semantic reflection and seman-tic transformation are the activities in which 3D contentchanges the format of representation—from a format read-able to 3D browsers to an ontology and vice versa. Theseactivities are necessary because 3D content ontologies arenot the native format understandable to the available 3Dbrowsers. In turn, semantic selection and semantic config-uration are the activities in which 3D content is modified us-ing the concepts specified in the 3D content ontology used.Some approaches enable only single modeling activities,e.g., [KCM06], [DFHP∗07, PDFHH07], whereas other ap-proaches enable several modeling activities combined intocontent creation pipeline, e.g., [AWGH11]. Furthermore, insome approaches, all activities are based on semantic webontologies, e.g., [KLN∗10], whereas in other approachessome activities are not based on ontologies, e.g., [DT-BRS03,BPKDT04,BDTK∗04,BTPK05,KDTM∗05,Man05,PDTB∗05, DTKPB07, DTKM∗07]. The particular activitiesare described below along with an example of modeling avirtual museum exhibition [FW14a]. The modeling processis presented in Figure 2. In description of the activities, 3Dscenes and elements represented using ontologies and 3Dformats are called semantic and syntactic, respectively.

Semantic Reflection. In semantic reflection, semantic con-tent elements corresponding to syntactic elements expressedin a 3D format are created (e.g., shapes, materials and anima-tions). The syntactic elements may be obtained from differ-ent sources and encoded in different formats and languages(e.g., VRML, X3D, MPEG-4) [Wal12a]. The created seman-tic elements are parts of an ontology-based representationand they are typically parametrized to enable the further se-mantic configuration (e.g., [WCW06]). For example, someworks have been devoted to reflecting different parts of hu-man body [ARSF07] and indoor scenes [AWGH11]. If se-mantic reflection is followed by the other modeling activ-ities, semantic elements are encoded using a common se-mantic web technique to preserve cross-compatibility in thefurther configuration and transformation [Wal12b]. Seman-tic elements may represent different features of 3D con-tent, such as geometry, structure, space, appearance, anima-tion and behavior at different levels of abstraction [FW13e,FW14a]. In general, semantic elements can modify, extendand gather the meaning of their prototype syntactic elements,e.g., a piece of furniture reflects a set of meshes; differ-

ent kinds of wood reflect different textures and shininess[FW13b].

For example, in case of a virtual museum (Figure 2), 3Dmodels of museum artifacts encoded in a 3D content format(e.g., X3D) can be obtained from a repository. Semantic el-ements that are counterparts to the models are created andencoded using a semantic web standard, e.g., OWL. The se-mantic elements respect further use cases determined by mu-seum curators, who will use the ontology for modeling vir-tual museum exhibitions. For instance, the cylinder and thebox are combined into the stool, while the sensors and theinterpolator are combined into the artifact animation. Bothconcept—the stool and the animation—are meant to be un-derstandable to museum curators. The result of the reflectionis a set of independent semantic elements and properties thatcan further combined into 3D scenes.

Semantic Selection. In semantic selection, semantic ele-ments are chosen for inclusion in the target content repre-sentation. Semantic selection indicates a subset of all seman-tic elements that are available in a repository [KDTM∗05,DTKM∗07], [KLN∗10]. Selection is performed using con-cepts at the abstraction level that have been determinedduring reflection. The level may be either specific to 3Dcontent—its geometry, structure, space, appearance, anima-tion and behavior (e.g., [SF08]), or specific to an applicationor domain (e.g, factory simulation [KLN∗10] and interiordesign [AWGH11]). Semantic selection typically precedessemantic configuration in the modeling process.

In the semantic selection in the virtual museum example,artifacts (except the seal), stools and the granary, which willbe further combined into the modeled 3D scene, are chosen.The elements still have no properties assigned and they arein no relations.

Semantic Configuration. In semantic configuration, se-mantic properties of selected elements are set with values,and the elements are combined into an ontology-based rep-resentation that is a coherent 3D scene. Configuration isperformed at the level of abstraction that has been used inthe previous activities. This may be specific to 3D content(e.g. [KCM06]), specific to an application or domain (e.g.,[KLN∗10]), or encompass both of them (e.g., [DTKPB07]).

In the semantic configuration in the virtual museum ex-ample, statues are specified as made of wood and glass, an-imation is applied to the glassy statue, and all the selectedartifacts are placed on stools. The result of the configurationis an ontology-based 3D scene comprised of elements withproperties and relations.

Semantic Transformation. In semantic transformation,configured ontology-based 3D scenes are encoded in par-ticular 3D content formats or languages. Therefore, seman-tic transformation can be seen as an inverse semantic re-

submitted to COMPUTER GRAPHICS Forum (1/2017).

Jakub Flotynski, Krzysztof Walczak / 7

Semantic Reflection

x3d: x3d: x3d:x3d: x3d:

x3d:

x3d: x3d:x3d:

owl:

x3d:

x3d:

x3d:x3d:x3d:

x3d:

owl:owl:owl:owl:owl:owl:

owl: owl: owl: owl:

owl:

Semantic Selection

x3d: x3d: x3d:x3d: x3d:

x3d:

x3d: x3d:x3d:

owl:

x3d:

x3d:

x3d:x3d:x3d:

x3d:

owl:owl:owl:owl:owl:owl:

owl: owl: owl: owl:

owl:

Semantic Transformation

Semantic Configuration

Figure 2: Activities of ontology-based modeling of 3D content

flection that pertains to complete semantic scenes. How-ever, in contrast to reflection, transformation produces acoherent content representation that consist of content el-ements with properties set, which enables its presentation

[FW14b, FW14c]. Transformation is often performed with-out the use of semantic web ontologies, e.g., [BQLC03,BBQC10,CTB∗12], [DTBRS03]. In such cases, transforma-

submitted to COMPUTER GRAPHICS Forum (1/2017).

8 Jakub Flotynski, Krzysztof Walczak /

tion is not a part of semantic modeling, however, it is unde-niably a part of a broader modeling process.

In the presented example, the semantic scene is trans-formed to an X3D document, and it can be presented in anX3D browser.

In the approaches to semantic modeling that are discussedin this paper, semantic reflection, selection and configurationare performed either manually (by developers) or automati-cally (by specific software). Automatic reflection usually re-quires analysis of content elements [AWGH11], e.g., contextof use, connections with other elements, etc., in particularwhen it follows 3D content segmentation. Automatic selec-tion and configuration are often used in contextual contentadaptation, which may take into account such elements asinteraction, user preferences and profiles [WRF14].

4. Classification of Ontology-based Approaches

A number of research works have been devoted to ontology-based representation and modeling of 3D content. The ap-proaches vary in several respects. In this section, a classifi-cation along with a taxonomy of the available approaches ispresented. The taxonomy has been elaborated taking into ac-count the aspects mentioned in Sections 2-3. In addition toclassification, it also enables identification of the main chal-lenges and open issues in the area.

The taxonomy is depicted in Figure 3. Every approachis either an ontology of 3D content or a method/tool forontology-based modeling of 3D content. The distinction be-tween ontologies and methods/tools corresponds to the dis-tinction between data representing 3D content and logic—sequences of activities and software used to create the con-tent. The parallel branches in the taxonomy tree representindependent classification criteria (e.g., represented featuresand semantic techniques used). However, some criteria areclosely related one to another and the proper discussion ofone of them requires to mention the second one. For in-stance, the discussion of abstraction levels requires referringto the features represented at these levels. Such criteria havebeen commonly discussed in the following sections.

Reasoning

Abstraction Level

Concrete

Conceptual

Separation of Concerns

Content Adaptation

Modeling Activity

Ontology

Method / Tool

Location of Annotations

Detached

Embedded

Domain of Use

Represented Feature

Geometry

Structure

Space

Appearance

Animation

Behavior

Semantic Technique

Classification group

Classification criteria

Configuration

Reflection

Transformation

Selection

RDF

RDFS

OWL

SWRL

Example of Use

Hybrid

Approach Type

Uses

Data

Process

Figure 3: Taxonomy of approaches to ontology-based repre-sentation and modeling of 3D content

submitted to COMPUTER GRAPHICS Forum (1/2017).

Jakub Flotynski, Krzysztof Walczak / 9Ta

ble

1:C

lass

ifica

tion

of3D

cont

ento

ntol

ogie

s

Sche

me

Lev

elof

Abs

trac

tion

Rep

rese

nted

Feat

ure

Loc

atio

nof

Ann

otat

ions

Sem

antic

Tech

niqu

eD

omai

n/E

xam

ple

ofU

seC

oncr

ete

Con

cept

ual

Geo

met

rySt

ruct

ure

Spac

eA

ppea

ranc

eA

nim

atio

nB

ehav

iour

Det

ache

dE

mbe

dded

1[D

TB

RS0

3,B

PKD

T04

][B

il07,

DT

KPB

07,P

DT

B∗

05]

33

33

33

33

3O

WL

gam

ede

sign

2[F

SA∗

04,A

PP∗

05,S

F08,

aim

]3

33

33

3O

WL

shap

ere

pres

enta

tion

3[G

ut05

,GRV

T∗

06,G

GR

T∗

07,M

PS06

]3

33

33

33

OW

Lhu

man

sim

ulat

ion

4[O

tt05a

,Ott0

5b]

33

33

RD

FSin

teri

orde

sign

5[R

S05]

33

3O

WL

tour

istg

uide

6[K

CM

06]

33

33

33

3O

WL

,Acc

ess-

Lim

ited

Log

icco

mpo

unds

,gen

ome

and

cell

visu

aliz

atio

n7

[PD

F06]

33

33

RD

Fvi

rtua

lmus

eum

8[A

RSF

07,R

ASF

07]

33

33

OW

Lhu

man

sim

ulat

ion

9[D

FHP∗

07,P

DFH

H07

]3

33

33

OW

Lsh

ape

repr

esen

tatio

n10

[CL

08,C

L12

]3

33

33

OW

Lhu

man

sim

ulat

ion

11[Z

KD

T09

]3

33

SWR

Lor

derp

roce

ss12

[KL

N∗

10]

33

33

3R

DFa

,OW

Lfa

ctor

ysi

mul

atio

n

13[V

GR

P∗10

]3

33

33

OW

Lhu

man

sim

ulat

ion,

prod

uctw

ork-

flow

14[A

WG

H11

]3

33

OW

Lin

teri

orde

sign

15[L

T11

,WL

12]

33

33

OW

Lga

me

desi

gn16

[FW

13c,

FW13

a,Fl

o13,

FW13

d]3

33

33

33

33

RD

Fa,O

WL

virt

ualm

useu

m

17[F

W13

b,FW

14a,

Flo1

4]3

33

33

3O

WL

,SW

RL

inte

rior

desi

gn,

virt

ualm

useu

m18

[FW

13f,

FW14

b,FW

14c]

33

33

33

3O

WL

,SW

RL

virt

ualm

useu

m

Tabl

e2:

Cla

ssifi

catio

nof

met

hods

and

tool

sfo

ront

olog

y-ba

sed

mod

elin

gof

3Dco

nten

t

App

roac

hM

etho

d(M

)/To

ol(T

)M

odel

ing

Act

ivity

Sepa

ratio

nof

Con

cern

sR

easo

ning

Con

tent

Ada

ptat

ion

Refl

ectio

nSe

lect

ion

Con

figur

atio

nTr

ansf

orm

atio

n1

[BQ

LC

03,B

BQ

C10

,CT

B∗

12]

T3

33

2

[DT

BR

S03,

BPK

DT

04]

[BD

TK∗

04,B

TPK

05,K

DT

M∗

05]

[Man

05,P

DT

B∗

05]

[DT

KPB

07,D

TK

M∗

07]

M3

33

3[G

ut05

,GT

V05

a]M

3—

——

——

4[C

DT

RK

06,P

KD

T06

,Pel

07]

[VR

C07

,PD

TK

08,P

KD

T09

]T

3—

——

——

5[K

CM

06]

M3

3—

——

——

6[A

RSF

07,R

ASF

07,A

RSF

09]

M3

3—

——

——

7[B

VO

GM

07]

M,T

33

3

8[D

FHP∗

07,P

DFH

H07

]M

3—

——

——

9[L

C07

,Lug

09]

T3

33

——

——

—10

[KL

N∗

10]

T3

33

11[A

WG

H11

]M

33

33

3

12[F

WG

S∗11

,LT

11,W

L12

]T

33

13[C

K13

,ZC

K∗

13]

M3

——

——

—

14[F

W13

b,FW

13e,

FW14

b][F

lo14

,FW

14c,

FW14

a]M

33

33

33

15[W

F14,

WR

F14,

WF1

5a,W

F15b

]M

,T3

33

3

16[P

A14

]T

3—

——

——

submitted to COMPUTER GRAPHICS Forum (1/2017).

10 Jakub Flotynski, Krzysztof Walczak /

4.1. Ontologies of 3D Content

The classification of the ontologies based on the taxonomyis outlined in Table 1. In the following sections, 3D contentontologies are summarized according to the particular clas-sification criteria (Figure 3).

4.1.1. Abstraction Level and Represented Features

The semantics of 3D content elements and properties may berepresented at different (low and high) levels of abstraction,which is the primary distinction of 3D content ontologies inthe proposed classification.

Concrete Representation. Concrete 3D content represen-tations are based on concepts whose meaning is specificto 3D graphics and animation, e.g., texture, dimensions,coordinates and LODs [KCM06] [BPKDT04] [FSA∗04,APP∗05, SF08] as well as VR/AR environments, e.g., in-terfaces, markers and models [RW14, Rum15]. Since suchontologies specify widely accepted classes and properties of3D content, their use is not limited to particular domains, al-though some of them have been presented in particular con-texts, e.g., human simulation [Gut05] [ARSF07] and touristguide [RS05]. Examples of use of concrete representationontologies are given in the Table 1. Concrete 3D contentrepresentations enable low-level access to 3D content ele-ments and properties using semantic web tools, such as Pro-tégé [CL08,CL12] (cf. Section 2.1/1), which is not permittedby the available 3D content formats and languages.

In concrete representations, features of 3D content aredirectly described—using concepts borrowed from widely-used 3D content formats, languages and libraries, such asCOLLADA, X3D and OpenGL. Therefore, concrete repre-sentations can be relatively easily transformed to equivalentfinal 3D content [FW14b] (cf. Section 4.2.1). The featuresthat are most frequently covered by ontologies of 3D contentare: geometry, structure and space, which are inherent to all3D models and scenes. Appearance, animation and behaviorare less addressed features. Animation and behavior requiremore complex content elements (e.g., sensors, interpolatorsand sequencers) [VGRP∗10], or more complex description(e.g., rule-based) [ZKDT09] than the other features. The par-ticular solutions are described in more detail below.

The model proposed in the AIM@SHAPE project [aim]combines 3D content with its corresponding concrete repre-sentations [FSA∗04, APP∗05, SF08]. The model introducesfour levels of content representation. The raw level cov-ers basic content properties related to different features ofthe content such as space and appearance, e.g., dimensionsand colors. The geometric level covers diverse geometri-cal elements, e.g., polygons, parametric surface models andstructured point sets. The structural level organizes both rawand geometrical levels by enabling, e.g., multi-resolution ge-ometry, multi-scale models and topological decomposition.

Finally, the semantic level associates concrete content ele-ments specified at the lower levels with their semantic equiv-alents.

The ontology proposed in [RS05] includes concepts link-ing geometrical models with spatial properties. The exampleuse of the ontology is related to representation of buildingsin a tourist guide.

The ontology proposed in [DFHP∗07, PDFHH07] per-mits concrete representation of non-manifold 3D shapes,e.g., a spider-web on a window, an umbrella with wires, acone touching a plane at a single point. The ontology in-cludes such properties as the number of vertices, numberof non-manifold vertices, number of edges, number of non-manifold edges, number of connected elements, etc. TheCommon Shape Ontology [VGRP∗10], which also stressesrepresentation of shapes, is focused on geometry, structure,shape and animation of 3D content by providing such con-cepts as manifold and non-manifold shapes, point sets, hier-archically structured groups of models, position, orientationand key frame animations.

The ontologies described in [CL08, CL12] enable rep-resentation of multi-user virtual environments and avatars.The ontologies focus on the geometry, space, animation andbehavior of 3D content. The included concepts are seman-tic equivalents to concepts incorporated in widely-used 3Dcontent formats, such as VRML and X3D. Environmentalobjects, which are the main entities of 3D content, are de-scribed by translation, rotation and scale. Avatars are de-scribed by names, statuses and UIs, while their behavior isdescribed by code bases.

Conceptual Representation. Conceptual 3D content rep-resentations are based on concepts whose meaning is not di-rectly related to 3D graphics and animation, but it is specificto an application or domain, e.g., virtual museum [PDF06]and interior design [Ott05a,Ott05b] [AWGH11]. Conceptual3D content representations describe content at a high-levelof abstraction, which is especially useful for users operat-ing with well known concepts (e.g., [ARSF07, RASF07]) ordomain-specific concepts (e.g., [KLN∗10]), without experi-ence in computer science (cf. Section 2.1/2).

In conceptual representations, features of 3D contentare indirectly described, e.g., different pieces of furniturerepresent different geometry [AWGH11]. In such cases,the generation of final 3D content is performed usingformat-specific algorithms implemented in software (e.g.,[BDTK∗04]), or it requires additional mapping of ontologiesto content formats and languages to enable more generic se-mantic transformation [FW14c]. Widespread demand for 3Dcontent representations that hide technical details of contenthas stimulated the development of ontologies in this groupmore than ontologies in the other two groups. The most spe-cific solutions are described in more detail below.

Several ontologies have been designed for representa-

submitted to COMPUTER GRAPHICS Forum (1/2017).

Jakub Flotynski, Krzysztof Walczak / 11

tion of human body. The ontology proposed in [Gut05,GGRT∗07] includes concepts enabling representation of vir-tual humans: geometrical descriptors of vertices and poly-gons, structural descriptors of articulation levels, 3D ani-mations of face and body, and behavior controllers (anima-tion algorithms). The extension of virtual humans with se-mantically represented emotional body expressions is pos-sible by applying the ontology proposed in [GRVT∗06].The ontology is built upon the Whissel’s wheel activation-evaluation space [Whi89]. It includes concepts combiningpassive/active and negative/positive adjectives related to hu-man emotions, e.g., despairing (very passive and very nega-tive), furious, terrified and disgusted (very active and verynegative), serene (very passive and very positive), exhil-arated, delighted and blissful (very active and very posi-tive). Other ontologies of human body have been describedin [ARSF07, RASF07].

An ontology for conceptual 3D content representationthat can be used in game design has been proposed in[WL12,LT11]. In the ontology, 3D content is represented us-ing actors that are the main elements of the created scenes,which manage entities—collections of semantic propertiesdescribing different 3D models. Communication betweenactors is based on events and shared variables.

In [ZKDT09], an OWL- and SWRL-based ontology formodeling features of 3D models in different application do-mains has been proposed. The ontology specifies composi-tions of features (conjunction and alternative), attributes offeatures (variables associated with features), relations be-tween features (mandatory or optional) and constraints onfeatures (e.g., excludes, implies, extends, equal, greater andlesser). Furthermore, the created ontology-based 3D contentrepresentations may be verified in terms of consistency, e.g.,an object that is required by another object cannot excludethe use of that requiring object.

Hybrid Representation. Hybrid 3D content representa-tions are combinations of the previous two types ofrepresentations—they cover 3D content at both concrete andconceptual levels of abstraction. To combine both types ofrepresentations, mapping is typically used [DTBRS03, BP-KDT04, PDTB∗05, Bil07] [FW13b, FW14a]. Therefore, theelaboration of hybrid ontologies of 3D content demandsmore effort, and this still gains little attention from the re-search community. Hybrid representations are convenientfor 3D content that needs to be represented at different lev-els of abstraction, e.g., primitive actions (move, turn, rotate,etc.) are combined to represent composite behaviors under-standable to end users without the knowledge of 3D graph-ics [DTKPB07]. The combined concrete and conceptual3D content representations proposed in [FW14b, FW14c]are mapped to templates encoded in content formats (e.g.,VRML, X3D and ActionScript), which enables automaticgeneration of final 3D scenes.

4.1.2. Semantic Web Techniques Used

Every ontology can be classified in terms of the semanticweb techniques used for its implementation. Since the RDF,RDFS, OWL and SWRL techniques represent consecutivestages of semantic web development, the ontologies basedon the latter techniques are also based on the former ones.

To enable reasoning, the semantic web techniques arebased on first-order logic and logic programming, whichhave been devised long before the development of the se-mantic web. First-order logic is a knowledge representa-tion technique that uses quantifiers and variables to buildstatements [KSH12]. Logic programming is a knowledgerepresentation technique based on rules, which are impli-cations [BG94]. The two approaches differ in expressive-ness, which is determined by the concepts they introduceand permits representation of elements and properties of 3Dcontent at different complexity levels. However, usually, thehigher the expressiveness of a particular knowledge repre-sentation technique, the larger the possibilities of expressingrelations between content elements, but also the more com-plex and time-consuming is reasoning on ontologies. Whilefirst-order logic and logic programming have been intendedfor artificial intelligence and expert systems, the semanticweb techniques are subsets of these formal systems adaptedto the development of the web towards a global databasewith semantically described content. Detailed discussion offirst-order logic and logic programming is out of the scopeof this paper. Focusing only on the expressiveness of the se-mantic web ontologies in 3D modeling, a summary of dif-ferent semantic concepts is presented in Table 3. The list isnot exhaustive, but it covers the most frequently used con-cepts in the reviewed approaches. The table provides infor-mation about the expressiveness of the particular ontologiesof 3D content (described in Table 1), which have been im-plemented using different semantic web techniques.

4.1.3. Location of Semantic Annotations

Another classification criterion is the location of ontology-based annotations that form the content representation. De-tached semantic annotations form content representationsseparated from the content. For example, different OWLknowledge bases can represent separate 3D models (e.g.,[KCM06]). In turn, embedded semantic annotations are in-cluded in the represented 3D content. In comparison to de-tached annotations, embedded annotations are more concise,reduce specification of redundant data (e.g., object IDs inboth representations), and facilitate management of contentthat is inextricably linked with its ontology-based represen-tation. For instance, RDFa-based annotations may be em-bedded in X3D documents [FW13c, FW13a, FW13d]. Themajority of ontologies are used to create detached represen-tations. The most specific embedded representation are de-scribed below.

A model combining conceptual 3D content representa-

submitted to COMPUTER GRAPHICS Forum (1/2017).

12 Jakub Flotynski, Krzysztof Walczak /

Table 3: The expressiveness of semantic web concepts in modeling 3D content

Semanticweb

techniqueConcept Usage Example

RDFStatement

Represents facts about content elements,their classes and properties

Mouse shape includes noseis a statement [RASF07]

typeAssigns properties associatedwith classes to elements

A ball is of the type entityof a virtual world [WL12]

RDFS

domain Denote applicability of propertiesto elements of a particular class

The hasStructuralDescriptor property has domainshape representation and range descriptor [VGRP∗10]range

subClassOfDenote inheritance of elements and propertiesassigned to super-classes/super-properties

Table and chair are subclasses offurniture in a 3D scene [AWGH11],thus they are made of the same material

subPropertyOf

OWL

ClassRepresents content elementswith common properties

Virtual rooms in a scene are connectedby an individual (content element)of the door class [KLN∗10]

Individual Represents a content element

equivalentClassDenote an alias to a class/property that maybe at a different abstraction level

The painting class is an equivalent to a classdescribing a texture on a surface.The made of property is an equivalent tothe hasMaterial property [WRF14]

equivalentProperty

DatatypePropertyRepresents content propertiesdescribed by literal values

Transform elements in a 3D scene havex, y, z coordinates [CL12]

ObjectPropertyRepresents content propertieslinking content elements

HasOutput is an object property that linksreconstructions to annotated meshes [VGRP∗10]

TransitivePropertyRepresents structures of elements linkedby properties of the same semantics

A non-manifold is a mesh, which is a geometry,thus a non-manifold is a geometry [PDFHH07]

SymmetricPropertyRepresents links between elements that arein the same semantic relation one to another

Some topological relations [RS05], e.g., is close to

inverseOf Represents inverse properties linking elements Before is inverse to after [DTKPB07]

allValuesFrom restrictionRepresents uniform structure ofelements, comprised of sub-elements ofa common class

All objects in a virtual museumroom are placed on stands [WF14]

someValuesFrom restrictionRepresents obligatory sub-elements inthe non-empty structure of an element

Every loggia has some walls [PDF06]

complementOfRepresents elements that do nothave some properties

Every round road sign that is not an orderroad sign is white and has red circuit

intersectionOfRepresents elements that inherit propertiesfrom multiple classes

Every round road sign that is blueis an order road sign [Flo14]

SWRL/RuleML

ruleml:Imp Represents semantic rules (implications)If two element properties are dependent but also excludethemselves (body), an exception is raised (head) [ZKDT09]

ruleml:bodyRepresents the ascendant of a rule(conjunction of statements)

ruleml:headRepresents the descendant of a rule(conjunction of statements)

Built-Ins Operations on variables and calculationsPresent only the objects between whichthe distance in the scene is maximal [WF15b]

tions with X3D documents has been proposed in [PDF06].Three concepts are used within the ontologies to specifythe relative positions of 3D models: contained, shared andbounded. The concepts are mapped to particular nodes ofX3D representations using X3D nodes for metadata de-scription (MetadataSet and MetadataString). X3D metadatanodes enable creation of ontology-based representations thatare directly embedded in the 3D content.

In [KLN∗10], conceptual representations may be eitherdetached or embedded in the represented content. Ontology-based representations are encoded in XML using the RDFaand OWL standards, and linked to 3D content encoded inXML3D. Another approach using RDFa to annotate 3Ddocuments has been proposed in [FW13c, FW13a, FW13d].Ontology-based representations are directly embedded in 3D

content using X3D metadata nodes and attributes. The em-bedded representations can be extracted from the content andcombined into detached representations depending on thestructure, types and roles of the content elements [Flo13].

4.2. Methods and Tools for Semantic Modeling

In some works (e.g., [BQLC03,BBQC10,CTB∗12]), the fo-cus is on the methods and tools instead of ontologies, thoughit is clear that there are some ontologies underlying the pre-sented approaches. Such methods and tools are reviewed inthis section, but they have no relevant ontologies summa-rized in Table 1.

A method of ontology-based modeling of 3D content isa set of generic activities (e.g., [FW14a]) accomplished se-quentially or in parallel, which produce 3D content. Typi-

submitted to COMPUTER GRAPHICS Forum (1/2017).

Jakub Flotynski, Krzysztof Walczak / 13

cally, in the activities, ontologies of 3D content are used.Some activities may by performed manually—by a humanusing specific hardware or software tools, while other activ-ities may be performed automatically—by software. For in-stance, content reflection can be performed by a graphic de-signer, while content selection can be performed by a query-processing engine [WF15b, FW16].

A tool for ontology-based modeling of 3D content is a setof interconnected software modules that are used to interactwith users, store, retrieve and process data, in order to enablemodeling of 3D content. For instance, a modeling tool maybe based on client-server architecture with the client basedon a 3D content browser and the server comprised of sev-eral services, e.g., [KLN∗10]. Tools implement methods ofmodeling, but in serveral works, methods are not explicitlypresented, as the focus is on software modules instead of theactivities performed by the modules to produce content.

The idea of knowledge-based modeling of 3D content ar-rived before the advent of the semantic web. It has coveredtools for designing AR applications [FMS93], the influenceof semantically organized virtual spaces on the users’ cogni-tion, interpretation and interaction [CTCC99], modeling ofhistoric monuments using geometrical elements combinedwith semantic structures [GKN99], automatic configurationof interior scenes based on constraints [GS∗99] as well assemantic representation of events and actions in virtual en-vironments [CP00]. Although the approaches did not lever-age the semantic web ontologies, they used schemes withdomain- and application-specific concepts, which could beimplemented as ontologies in follow-up developments.

The classification of the methods and tools based on thetaxonomy is outlined in Table 2. In the following sections,methods and tools of ontology-based modeling are summa-rized according to the particular classification criteria.

4.2.1. Modeling Activities Supported

Methods and tools are classified in terms of the supportedmodeling activities, which may be one or several of: seman-tic reflection, semantic selection, semantic configuration andsemantic transformation (cf. Section 3.4). Several supportedactivities within a method or a tool form content creationpipeline, e.g., [BDTK∗04, BTPK05, KDTM∗05].

The methods and tools usually support only subsets ofall the modeling activities. The largest group encompassesapproaches supporting content selection and configuration.The second group comprises approaches that support onlysemantic reflection and do not support the other modelingactivities. Semantic transformation still does not gain muchattention from the research community. The most specificsolutions are described in more detail below.

The method proposed in [DTBRS03, BP-KDT04,BDTK∗04,BTPK05,KDTM∗05,Man05,PDTB∗05,

DTKPB07, DTKM∗07] enables content selection and con-figuration at the conceptual level using domain-specificontologies, which are mapped to a concrete representationontology. The method proposed in [FW13e] also usesmapping between 3D content ontologies at the concrete andconceptual levels. In addition, the method enables semanticreflection, in which reusable semantic 3D content elementsand properties are created, and semantic transformation. Fi-nal 3D content, which can be encoded in different languages(e.g., VRML, X3D and ActionScript), is automatically gen-erated by linking templates of code in particular languagesto statements included in the ontology-based representation.The transformation is described by an ontology indicatingtemplates of code and their corresponding statements.

The method proposed in [DFHP∗07, PDFHH07] enablesreflection of non-manifold 3D shapes using concrete prop-erties. The method reflects geometrical properties of shapes:non-manifold singularities (e.g., isolated points and curves),one-dimensional parts, connected elements and maximalconnected elements. Once identified, the properties aremapped to a shape ontology and form a concrete ontology-based representation of the shape.

In some works, semantic reflection is performed aftercontent segmentation, in which different elements of thecontent are distinguished on the basis of their properties(geometry, colors, relative locations, etc.). In [ARSF07,RASF07,ARSF09], after automatic segmentation of 3D con-tent, the distinguished elements are semantically reflected.Two modes of reflection have been developed. Automaticreflection is performed by software considering topologicalrelations between content elements (e.g., orientation, size,adjacency and overlapping). Manual reflection is performedby a user equipped with a graphical tool. Moreover, an ex-ample of semantic reflection based on an ontology of humanbody is presented.

The method of modeling 3D content based on pointclouds proposed in [AWGH11] involves several activities.First, in semantic selection, an input point cloud is analyzedto discover planar patches, their properties (e.g., locations)and relations. Then an OWL reasoner processes a domain-specific ontology including conceptual elements that poten-tially match the analyzed patches. Next, matching elementsare selected and configured to build a conceptual representa-tion. The created representation is an ontology-based equiv-alent to the input point cloud.

In the Simulator X [FWGS∗11, LT11, WL12], the selec-tion of elements to be included in the created content is per-formed by a user. The selected elements (actors and entities)are configured using state variables, which can be shared bydifferent actors in the content.

iRep3D [CK13, ZCK∗13] enables semantic reflection of3D models by analyzing its syntactic, conceptual, functionaland geometrical features. The method is used for 3D modelsencoded in X3D, XML3D and COLLADA.

submitted to COMPUTER GRAPHICS Forum (1/2017).

14 Jakub Flotynski, Krzysztof Walczak /

4.2.2. Support for Content Adaptation

The methods and tools that implement several modeling ac-tivities can be classified in terms of support for 3D con-tent adaptation. The methods and tools supporting contentadaptation enable modeling of content with respect to indi-vidual user preferences expressed in ontology-based queries[BVOGM07] [WF14]. Ontology-based adaptation facilitatessharing and reusing of content in comparison to modeling3D content from scratch. Content adaptation can be per-formed across the whole modeling process and affect the re-sults of different modeling activities. The methods and toolsthat do not support content adaptation enable modeling ofcontent in its final form, in which it will be presented toend-users, e.g., [DTBRS03,BPKDT04,BDTK∗04,BTPK05,KDTM∗05, Man05, PDTB∗05, DTKPB07].

Currently, few approaches permit content adaptation. In3DAF [BVOGM07], once queries are issued by users in-voking methods of the communication interface, the invoca-tions are translated into SQL-like queries and processed bya query manager. Next, content elements are retrieved froman annotation repository and a final 3D scene is generated.For example, trees are excluded from the scene, and the ge-ometry of the buildings is simplified (Figure 4).

Figure 4: Adaptation of a campus based on semanticrules. Source: [BVOGM07]. Courtesy of I. M. Bilasco, M.Villanova-Oliver, J. Gensel and H. Martin

In [WRF14,WF15b,FW16], selection and composition ofcontent are performed in response to users’ queries. Queriesare ontologies which are—on demand—combined and pro-cessed with generalized ontology-based 3D content repre-sentations (3D meta-scenes). Examples of 3D content adap-tation with the approach are presented in Figure 5. Themethod has been implemented as a service-oriented toolbased on Blender [WF15a, WF15b, FW16]. In the other an-alyzed approaches, 3D content is created in its final formwithout possibilities of further adaptation.

4.2.3. Support for Reasoning

Due to the use of the semantic web ontologies, every methodand tool can be classified in terms of the available sup-port for reasoning in the modeling process. In the case ofmethods and tools that do not support reasoning on con-tent representations, final 3D content is based only on thestatements explicitly specified while modeling. For instance,in [DTBRS03, BPKDT04], the mapping of domain-specific

concepts to 3D-specific concepts is explicitly specified andfurther used in modeling. In the case of methods and toolsthat support reasoning, final 3D content is based on boththe explicit and implicit (inferred) knowledge. For instance,in [KCM06], software processes rules and statements to vi-sualize a planetary system on the basis of inferred knowl-edge.

Although reasoning is one of the principles of the seman-tic web, this is still used in less than half of the approaches.The approaches that support reasoning require less users’effort in modeling than the approaches that only make useof the explicit knowledge (like in the case of typical meta-data) [FW14a]. For instance, in [LC07, Lug09], ontology-based representations of events and actions are used to in-fer their effects in dynamic 3D scenes (Figure 6). Moreover,such approaches are more flexible in terms of which usersdefine which statements for the content.

Figure 6: An example of a semantic action representationwith the inference of its effects. Source: [LC07]. Courtesyof J. L. Lugrin and M. Cavazza

4.2.4. Support for Separation of Concerns

Encompassing different modeling activities and enabling 3Dcontent representation at different levels of abstraction usu-ally permits separation of concerns between different mod-eling users who have different skills and experience, andwho are equipped with different hardware and software tools(ontology editors, graphical editors, scanners, etc.). For in-stance, basic content elements may be scanned, manipulatedand mapped to high-level concepts by a developer; next, thehigh-level concepts may be used to create 3D content repre-sentations by a domain expert [WRF14, WF15a].

In a few of the available approaches, the modeling pro-cess is separated into activities that may be accomplished bydifferent users. In the methods of ontology-based modelingbased on mapping between concrete and conceptual levelsof abstraction [BDTK∗04, BTPK05, KDTM∗05] [FW13e,FW14a], conceptual content representations are created bydomain experts according to the domain-specific ontologies,while the concrete representation ontologies may be used byan expert in computer graphics.

submitted to COMPUTER GRAPHICS Forum (1/2017).

Jakub Flotynski, Krzysztof Walczak / 15

(a) (b) (c) (d)

Figure 5: City models generated from generalized semantic 3D meta-scenes: (a) roads, buildings and trees during the day, (b)roads, buildings and only palm trees during the day, (c) roads and all road traffic at night, (d) roads and privileged cars only.Source: [WF15b].

Separation of concerns is addressed more often in toolsthen in methods of modeling. In tools, different modelingactivities are typically related to different software mod-ules, which—in turn—can be used by users with differentskills in 3D modeling. In MASCARET [BQLC03,BBQC10,CTB∗12], selection and configuration of content elementsare performed by experts in the semantic web and computergraphics equipped with a 3D modeling tool (Figure 7).

Figure 7: The GUI and example 3D content created in MAS-CARET. Source: [CTB∗12]. Courtesy of P. Chevaillier, T.Trinh, M. Barange, P. D. Loor, F. Devillers, J. Soler and R.Querrec

In the simulation tool presented in [LC07,Lug09], the Un-real Tournament game engine (for rigid-body physics andcontent presentation), an inference engine (for reasoning andupdating the scene representation when events occur) and abehavioral engine (for recognizing actions and changing ob-jects in conceptual terms) are used. Combining different en-gines within one tool creates opportunities for separation ofconcerns between users experienced in their usage.

An example of division of responsibilities between dif-ferent software modules in a client-server architecture hasbeen presented in [KLN∗10]. The tool leverages semanticconcepts, services and hybrid automata to describe behav-ior of 3D content elements. The client is based on a 3D

content presentation tool, e.g., an XML3D browser, whilethe server is built of several services enabling content selec-tion and configuration. A graphical module maintains andrenders 3D scene graphs. A scene module manages globalscene ontologies, which represent the created simulations.A verification module checks spatial and temporal require-ments against properties of content elements. An agent mod-ule manages intelligent avatars, e.g., their perception of thescene. The user interface is capable of communicating withweb-based and immersive virtual reality platforms.

5. Conclusions and Open Challenges

The use of semantic web ontologies to facilitate modelingof synthetic 3D content for VR/AR applications receives in-creasing attention from the research community. Ontology-based representation and modeling of 3D content are lever-aged in various applications and domains, such as culturalheritage, entertainment and simulation. In this paper, theavailable solutions have been reviewed and classified ac-cording to the proposed taxonomy. The following conclu-sions may be drawn from this survey.

3D content ontologies form a common space for repre-senting the semantics of content at concrete and conceptuallevels of abstraction. Ontologies are used within methodsand tools for modeling 3D content. The available methodsand tools typically do not cover the whole modeling pro-cess, but enable only selected modeling activities. The avail-able approaches still do not fully benefit from the possibil-ities of knowledge inference from content representations,and many of them use ontologies as metadata descriptionsof content elements and properties.

Some problems may be indicated in the field of ontology-based representation and modeling of 3D content. Creationof T-Box ontologies is the preliminary step that must be ac-complished before content can be semantically modeled. Inspecific applications and domains, it may be difficult to findappropriate ontologies that provide concepts sufficient forthe analyzed area and use cases. In such cases, it may benecessary that content authors are also involved in the cre-ation of the appropriate ontology. Another limit in the useof ontologies are frequently changing use cases for whichcontent is created. In such cases, T-Box ontologies may be

submitted to COMPUTER GRAPHICS Forum (1/2017).

16 Jakub Flotynski, Krzysztof Walczak /

used few times, which could make their creation unreason-able. Finally, exponential computing complexity of reason-ing algorithms may cause long time of generating results ofmodeling.

The following directions of future research may be indi-cated on the basis of the presented survey.

• More attention should be paid to hybrid 3D content rep-resentations at different levels of abstraction—at the levelthat is specific to 3D graphics and animation and at levelsthat are specific to particular applications or domains. Hy-brid representation enables more flexible content creationand management (indexing, searching and analyzing) incomparison to concrete and conceptual representations,because it enables combination of these two abstractionlevels.

• More effort should be put into the use of inference of im-plicit knowledge from ontology-based 3D content repre-sentations. It could potentially reduce time and costs ofmodeling by liberating users from specifying all elementsof the created content.

• Only a few of the available methods and tools permit se-mantic transformation of content to different content lan-guages. In contrast to syntactic transformation, seman-tic transformation can potentially provide better resultsby taking into account the meaning of particular contentelements and properties, e.g., transformation of selectedfunctional subgroups of semantic 3D scenes.

• Persistent link between ontology-based and final 3D con-tent representations can be proposed to enable real-timesynchronization of the representations and to put morestress on dynamic explorable spatio-temporal content rep-resentations.

• The available works on ontology-based modeling of 3Dcontent do not address generic methods of combining dis-tributed, reusable and possibly stateful content elements(e.g., SOA-based), except the distribution that is an inher-ent element of using ontologies on the web. Ontology- andSOA-based 3D content representation could benefit fromthe available standards (e.g., OWL-S [OWL], WSDL-S[WSD] and SA-WSDL [SAW]) to facilitate descriptionof content behavior and spatio-temporal relations betweendistributed content elements.

• Flexible methods of 3D content adaptation covering bothqueries to content representations and contextual informa-tion, e.g., device use, user location and preferences, can beproposed.

• Semantic web ontologies can also be used for securingvirtual environments [Wój10].

Other open issues are related to automatic reflection ofsemantic representations from 3D content, semantically-oriented content synthesis, documenting the life cycle of 3Dmodels as well as semantically described visualization andinteraction with 3D content [CMSF11].

Acknowledgments

This research work has been supported by the Pol-ish National Science Centre (NCN) Grants No. DEC-2012/07/B/ST6/01523 and DEC-2014/12/T/ST6/00039.

References

[ABK07] ALPCAN T., BAUCKHAGE C., KOTSOVINOS E.: To-wards 3D internet: Why, what, and how? In Cyberworlds, 2007.CW’07. International Conference on (2007), IEEE, pp. 95–99. 2

[AC01] AYLETT R., CAVAZZA M.: Intelligent virtual environ-ments: a state-of-the-art report. In In: Eurographics 2001, STARReports volume (2001), pp. 87–109. 4

[aim] AIM@SHAPE Project. URL: http://cordis.europa.eu/ist/kct/aimatshape_synopsis.htm. 9,10

[APP∗05] ALBERTONI R., PAPALEO L., PITIKAKIS M., ROB-BIANO F., SPAGNUOLO M., VASILAKIS G.: Ontology-basedsearching framework for digital shapes. In On the Move to Mean-ingful Internet Systems 2005: OTM 2005 Workshops (2005),Springer, pp. 896–905. 3, 9, 10

[ARC] ARCO: Virtual museum system. URL: http://www.wirtualne-muzea.pl/en/. 3

[ARSF07] ATTENE M., ROBBIANO F., SPAGNUOLO M., FAL-CIDIENO B.: Semantic annotation of 3D surface meshes basedon feature characterization. In Proceedings of the Semanticand Digital Media Technologies 2Nd International Conferenceon Semantic Multimedia (Berlin, Heidelberg, 2007), SAMT’07,Springer-Verlag, pp. 126–139. 6, 9, 10, 11, 13

[ARSF09] ATTENE M., ROBBIANO F., SPAGNUOLO M., FAL-CIDIENO B.: Characterization of 3D Shape Parts for Se-mantic Annotation. Comput. Aided Des. 41, 10 (Oct. 2009),756–763. URL: http://dx.doi.org/10.1016/j.cad.2009.01.003, doi:10.1016/j.cad.2009.01.003. 9,13

[ATS∗07] ARNDT R., TRONCY R., STAAB S., HARDMAN L.,VACURA M.: Comm: Designing a well-founded multimedia on-tology for the web. In Proceedings of the 6th International TheSemantic Web and 2Nd Asian Conference on Asian Semantic WebConference (Berlin, Heidelberg, 2007), ISWC’07/ASWC’07,Springer-Verlag, pp. 30–43. 3

[Auta] AUTODESK: 3ds Max. URL: http://www.autodesk.pl/products/autodesk-3ds-max/overview. 2

[Autb] AUTODESK: AutoCAD Civil 3D. URL: http://www.autodesk.com/products/autocad-civil-3d/overview. 2

[AWGH11] ALBRECHT S., WIEMANN T., GÜNTHER M.,HERTZBERG J.: Matching CAD object models in semantic map-ping. In Proceedings ICRA 2011 Workshop: Semantic Percep-tion, Mapping and Exploration, SPME (2011). 4, 6, 8, 9, 10, 12,13

[BBQC10] BUCHE C., BOSSARD C., QUERREC R., CHEVAIL-LIER P.: Pegase: A generic and adaptable intelligent system forvirtual reality learning environments. International Journal ofVirtual Reality 9, 2 (2010), 73–85. 7, 9, 12, 15

[BDTK∗04] BILLE W., DE TROYER O., KLEINERMANN F.,PELLENS B., ROMERO R.: Using ontologies to build virtualworlds for the web. In ICWI (2004), Isaías P. T., Karmakar N.,Rodrigues L., Barbosa P., (Eds.), IADIS, pp. 683–690. 6, 9, 10,13, 14

submitted to COMPUTER GRAPHICS Forum (1/2017).

Jakub Flotynski, Krzysztof Walczak / 17

[BG94] BARAL C., GELFOND M.: Logic programming andknowledge representation. The Journal of Logic Programming19 (1994), 73–148. 11

[Bil07] BILLE W.: Conceptual Modeling of Complex Objects forVirtual Environments. PhD thesis, Vrije Universiteit Brussel,2006-2007. 9, 11

[Ble] Blender. URL: http://www.blender.org/. 2

[BLHL∗01] BERNERS-LEE T., HENDLER J., LASSILA O.,ET AL.: The semantic web. Scientific american 284, 5 (2001),28–37. 2

[BPKDT04] BILLE W., PELLENS B., KLEINERMANN F.,DE TROYER O.: Intelligent modelling of virtual worlds usingdomain ontologies. In Proceedings of the Workshop of Intelli-gent Computing (WIC), held in conjunction with the MICAI 2004conference (Mexico City, Mexico, 2004), pp. 272–279. 4, 6, 9,10, 11, 13, 14

[BQLC03] BUCHE C., QUERREC R., LOOR P. D., CHEVAIL-LIER P.: Mascaret: Pedagogical multi-agents system for virtualenvironment for training. In CW (2003), IEEE Computer Society,pp. 423–431. 5, 7, 9, 12, 15

[BTPK05] BILLE W., TROYER O. D., PELLENS B., KLEINER-MANN F.: Conceptual modeling of articulated bodies in virtualenvironments. In Proceedings of the 11th International Confer-ence on Virtual Systems and Multimedia (VSMM) (Ghent, Bel-gium, 2005), Thwaites H., (Ed.), Archaeolingua, Archaeolingua,pp. 17–26. 6, 9, 13, 14

[BVOGM07] BILASCO I. M., VILLANOVA-OLIVER M.,GENSEL J., MARTIN H.: Semantic-based Rules for 3D SceneAdaptation. In Proceedings of the Twelfth International Con-ference on 3D Web Technology (New York, NY, USA, 2007),Web3D ’07, ACM, pp. 97–100. 9, 14

[CDTRK06] CONINX K., DE TROYER O., RAYMAEKERS C.,KLEINERMANN F.: VR-DeMo: a Tool-supported ApproachFacilitating Flexible Development of Virtual Environments us-ing Conceptual Modelling. In Proceedings of Virtual Concept(2006), Springer-Verlag. 9

[CK13] CAO X., KLUSCH M.: Advanced Semantic Deep Searchfor 3D Scenes. In Semantic Computing (ICSC), 2013 IEEE Sev-enth International Conference on (2013), IEEE, pp. 236–243. 9,13

[CL08] CHU Y., LI T.: Using pluggable procedures and ontologyto realize semantic virtual environments 2.0. In Proceedings ofThe 7th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and Its Applications in Industry (New York,NY, USA, 2008), VRCAI ’08, ACM, pp. 27:1–27:6. 9, 10

[CL12] CHU Y. L., LI T. Y.: Realizing semantic virtual environ-ments with ontology and pluggable procedures. Applications ofVirtual Reality (2012). 9, 10, 12

[CMSF11] CATALANO C. E., MORTARA M., SPAGNUOLO M.,FALCIDIENO B.: Semantics and 3D media: Current issues andperspectives. Computers & Graphics 35, 4 (2011), 869–877. 16

[Col] COLLADA - Digital Asset Schema Release 1.4.1, Specifi-cation (2nd Edition). URL: https://www.khronos.org/files/collada_spec_1_4.pdf. 1

[CP00] CAVAZZA M., PALMER I.: High-level interpretation invirtual environments. Applied Artificial Intelligence 14, 1 (2000),125–144. 13

[CTB∗12] CHEVAILLIER P., TRINH T., BARANGE M., LOORP. D., DEVILLERS F., SOLER J., QUERREC R.: Semantic mod-eling of virtual environments using mascaret. In SEARIS (2012),IEEE, pp. 1–8. 7, 9, 12, 15

[CTCC99] CHEN C., THOMAS L., COLE J., CHENNAWASIN C.:Representing the semantics of virtual spaces. IEEE MultiMedia6, 2 (1999), 54–63. 13

[DAD∗11] DARAS P., AXENOPOULOS A., DARLAGIANNIS V.,TZOVARAS D., BOURDON X. L., JOYEUX L., VERROUST-BLONDET A., CROCE V., STEINER T., MASSARI A., CA-MURRI A., MORIN S., MEZAOUR A., SUTTON L., SPILLERS.: Introducing a unified framework for content object descrip-tion. IJMIS 2, 3/4 (2011), 351–375. 3

[DFHP∗07] DE FLORIANI L., HUI A., PAPALEO L., HUANGM., HENDLER J.: A semantic web environment for digitalshapes understanding. In Semantic Multimedia. Springer, 2007,pp. 226–239. 4, 6, 9, 10, 13

[DFS07] DE FLORIANI L., SPAGNUOLO M.: Shape analysis andstructuring. Springer, 2007. 2

[DHM02] DACHSELT R., HINZ M., MEISSNER K.: Contigra:An XML-based Architecture for Component-oriented 3D Appli-cations. In Proceedings of the Seventh International Conferenceon 3D Web Technology (New York, NY, USA, 2002), Web3D ’02,ACM, pp. 155–163. doi:10.1145/504502.504527. 2

[dir] Direct3D 11.1 Features. URL: https://msdn.microsoft.com/en-us/library/windows/desktop/hh404562%28v=vs.85%29.aspx. 1

[DTBRS03] DE TROYER O., BILLE W., ROMERO R., STUERP.: On generating virtual worlds from domain ontologies. InProceedings of the 9th International Conference on Multi-MediaModeling (Taipei, Taiwan, 2003), pp. 279–294. 6, 7, 9, 11, 13,14

[DTKM∗07] DE TROYER O., KLEINERMANN F., MANSOURIH., PELLENS B., BILLE W., FOMENKO V.: Developing seman-tic VR-shops for e-Commerce. Virtual Reality 11, 2-3 (2007),89–106. 4, 5, 6, 9, 13

[DTKPB07] DE TROYER O., KLEINERMANN F., PELLENS B.,BILLE W.: Conceptual modeling for virtual reality. In Tutorials,posters, panels and industrial contributions at the 26th Interna-tional Conference on Conceptual Modeling - ER 2007 (Auck-land, New Zealand, 2007), Grundy J., Hartmann S., Laender A.H. F., Maciaszek L., Roddick J. F., (Eds.), vol. 83 of CRPIT, ACS,pp. 3–18. 6, 9, 11, 12, 13, 14

[Flo13] FLOTYNSKI J.: Harvesting of Semantic Metadata fromDistributed 3D Web Content. In Proceedings of the 6th Inter-national Conference on Human System Interaction (HSI), June06-08, 2013, Sopot (Poland) (2013), IEEE. 9, 12

[Flo14] FLOTYNSKI J.: Semantic modelling of interactive 3Dcontent with domain-specific ontologies. Procedia ComputerScience 35 (2014), 531–540. 18th International Conferenceon Knowledge-Based and Intelligent Information & EngineeringSystems. 5, 9, 12

[FMS93] FEINER S., MACINTYRE B., SELIGMANN D.:Knowledge-based augmented reality. Commun. ACM 36, 7 (July1993), 53–62. doi:10.1145/159544.159587. 13

[FS98] FALCIDIENO B., SPAGNUOLO M.: A shape abstractionparadigm for modeling geometry and semantics. In ComputerGraphics International (1998), IEEE Computer Society, pp. 646–. 4, 5

[FSA∗04] FALCIDIENO B., SPAGNUOLO M., ALLIEZ P., QUAKE., VAVALIS E., HOUSTIS C.: Towards the semantics of digitalshapes: The aim@shape approach. In EWIMT (2004). 3, 9, 10

[FW13a] FLOTYNSKI J., WALCZAK K.: Attribute-based Seman-tic Descriptions of Interactive 3D Web Content. In Informa-tion Technologies in Organizations - Management and Applica-tions of Multimedia (2013), Kiełtyka L., (Ed.), Wydawnictwa To-

submitted to COMPUTER GRAPHICS Forum (1/2017).

18 Jakub Flotynski, Krzysztof Walczak /

warzystwa Naukowego Organizacji i Kierownictwa - Dom Orga-nizatora, pp. 111–138. 9, 11, 12

[FW13b] FLOTYNSKI J., WALCZAK K.: Conceptual SemanticRepresentation of 3D Content. Lecture Notes in Business Infor-mation Processing: 16th International Conference on BusinessInformation Systems, Poznan, Poland, 19 - 20 June, 2013 160(2013), 244–257. 6, 9, 11

[FW13c] FLOTYNSKI J., WALCZAK K.: Describing Semanticsof 3D Web Content with RDFa. In The First International Con-ference on Building and Exploring Web Based Environments,Sevilla (Spain), January 27 - February 1, 2013 (2013), Think-Mind, pp. 63–68. 3, 9, 11, 12

[FW13d] FLOTYNSKI J., WALCZAK K.: Microformat and Mi-crodata Schemas for Interactive 3D Web Content. In Proceed-ings of the 2013 Federated Conference on Computer Science andInformation Systems Kraków, Poland, 8 - 11 September, 2013(2013), Ganzha M., Maciaszek L., Paprzycki M., (Eds.), vol. 1,Polskie Towarzystwo Informatyczne, pp. 549–556. 3, 9, 11, 12

[FW13e] FLOTYNSKI J., WALCZAK K.: Semantic Modelling ofInteractive 3D Content. In Proceedings of the 5th Joint VirtualReality Conference (Paris, France, December 11-13, 2013). 6, 9,13, 14

[FW13f] FLOTYNSKI J., WALCZAK K.: Semantic Multi-layeredDesign of Interactive 3D Presentations. In Proceedings ofthe Federated Conference on Computer Science and Informa-tion Systems (Kraków, Poland, September 8-11, 2013), IEEE,pp. 541–548. 9

[FW14a] FLOTYNSKI J., WALCZAK K.: Conceptual knowledge-based modeling of interactive 3D content. The Vi-sual Computer (August 2014), 1–20. doi:10.1007/s00371-014-1011-9. 6, 9, 11, 12, 14

[FW14b] FLOTYNSKI J., WALCZAK K.: Multi-platform Seman-tic Representation of 3D Content. In Proceedings of the 5th Doc-toral Conference on Computing, Electrical and Industrial Sys-tems (Lisbon, Portugal, April 7-9 2014). 7, 9, 10, 11

[FW14c] FLOTYNSKI J., WALCZAK K.: Semantic Representa-tion of Multi-platform 3D Content. Computer Science and Infor-mation Systems 11, No 4, October 2014 (2014), 1555–1580. 5,7, 9, 10, 11

[FW16] FLOTYNSKI J., WALCZAK K.: Customization of 3Dcontent with semantic meta-scenes. Graphical Models (2016),in print. doi:http://dx.doi.org/10.1016/j.gmod.2016.07.001. 3, 5, 13, 14

[FWGS∗11] FISCHBACH M., WIEBUSCH D., GIEBLER-SCHUBERT A., LATOSCHIK M. E., REHFELD S., TRAM-BEREND H.: SiXton’s curse - Simulator X demonstration. In Vir-tual Reality Conference (VR), 2011 IEEE (2011), Hirose M., LokB., Majumder A., Schmalstieg D., (Eds.), pp. 255–256. URL:http://dx.doi.org/10.1109/VR.2011.5759495.9, 13

[Gai07] GAILDRAT V.: Declarative modelling of virtual environ-ments, overview of issues and applications. In International Con-ference on Computer Graphics and Artificial Intelligence (3IA),Athenes, Grece (2007), vol. 10, pp. 5–15. 4

[GGRT∗07] GUTIÉRREZ M., GARCÍA-ROJAS A., THALMANND., VEXO F., MOCCOZET L., MAGNENAT-THALMANN N.,MORTARA M., SPAGNUOLO M.: An ontology of virtual hu-mans: Incorporating semantics into human shapes. Vis. Comput.23, 3 (Feb. 2007), 207–218. 9, 11

[GKN99] GRUSSENMEYER P., KOEHL M., NOUREL M.: 3Dgeometric and semantic modelling in historic sites, 1999. 13

[GL96] GIACOMO G. D., LENZERINI M.: Tbox and abox rea-soning in expressive description logics. In In Proc. of KR-96(1996), Morgan Kaufmann, pp. 316–327. 3

[GN87] GENESERETH M. R., NILSSON N. J.: Logical founda-tions of artificial. Intelligence. Morgan Kaufmann (1987). 2

[gp1] Ghost productions. URL: http://www.ghostproductions.com/. 2

[GP12] GRÖGER G., PLÜMER L.: CityGML–Interoperable se-mantic 3D city models. ISPRS Journal of Photogrammetry andRemote Sensing 71 (2012), 12–33. 2

[Gru95] GRUBER T. R.: Toward principles for the design of on-tologies used for knowledge sharing? International journal ofhuman-computer studies 43, 5 (1995), 907–928. 2

[GRVT∗06] GARCÍA-ROJAS A., VEXO F., THALMANN D.,RAOUZAIOU A., KARPOUZIS K., KOLLIAS S.: Emotionalbody expression parameters in virtual human ontology. In Pro-ceedings of 1st International Workshop on Shapes and Seman-tics, Matsushima, Japan, June 2006, pp. 63-70. URL: http://www.image.ece.ntua.gr/publications.php. 9,11

[GS∗99] GOSELE M., STUERZLINGER W., ET AL.: Semanticconstraints for scene manipulation. In in Proceedings SpringConference in Computer Graphics’ 99 (Budmerice, Slovak Re-public (1999), Citeseer. 13

[GTV05a] GUTIÉRREZ M., THALMANN D., VEXO F.: Seman-tic virtual environments with adaptive multimodal interfaces. InMMM (2005), Chen Y.-P. P., (Ed.), IEEE Computer Society,pp. 277–283. 9

[GTV∗05b] GUTIÉRREZ M., THALMANN D., VEXO F., MOC-COZET L., MAGNENAT-THALMANN N., MORTARA M., SPAG-NUOLO M.: An ontology of virtual humans: incorporating se-mantics into human shapes. In Proceedings Workshop towardsSemantic Virtual Environments (SVE05) (2005), no. VRLAB-CONF-2007-004, pp. 56–57. 5

[Gut05] GUTIÉRREZ M.: Semantic virtual environments. 5, 9,10, 11

[Hun01] HUNTER J.: Adding multimedia to the semantic web:Building an mpeg-7 ontology. In SWWS (2001), Cruz I. F.,Decker S., Euzenat J., McGuinness D. L., (Eds.), pp. 261–283.2

[KCM06] KALOGERAKIS E., CHRISTODOULAKIS S.,MOUMOUTZIS N.: Coupling ontologies with graphics contentfor knowledge driven visualization. In VR ’06 Proceedings ofthe IEEE conference on Virtual Reality (Alexandria, Virginia,USA, March 25-29, 2006), pp. 43–50. 4, 6, 9, 10, 11, 14

[KDTM∗05] KLEINERMANN F., DE TROYER O., MANSOURIH., ROMERO R., PELLENS B., BILLE W.: Designing semanticvirtual reality applications. In In Proceedings of the 2nd INTU-ITION International Workshop, Senlis (2005), pp. 5–10. 6, 9, 13,14

[KLN∗10] KAPAHNKE P., LIEDTKE P., NESBIGALL S., WAR-WAS S., KLUSCH M.: ISReal: An Open Platform for Semantic-Based 3D Simulations in the 3D Internet. In International Se-mantic Web Conference (2) (2010), pp. 161–176. 5, 6, 9, 10, 12,13, 15

[KSH12] KRÖTZSCH M., SIMANCIK F., HORROCKS I.: A de-scription logic primer. arXiv preprint arXiv:1201.4089 (2012).11

[LA00] LUCK M., AYLETT R.: Applying artificial intelligence tovirtual reality: Intelligent virtual environments. Applied ArtificialIntelligence 14, 1 (2000), 3–32. 4

submitted to COMPUTER GRAPHICS Forum (1/2017).

Jakub Flotynski, Krzysztof Walczak / 19

[LB08] LATOSCHIK M. E., BLACH R.: Semantic modelling forvirtual worlds – a novel paradigm for realtime interactive sys-tems? In Proceedings of the ACM VRST 2008 (2008), pp. 17–20.2

[LC07] LUGRIN J.-L., CAVAZZA M.: Making sense of virtualenvironments: Action representation, grounding and commonsense. In Proceedings of the 12th international conference onIntelligent user interfaces (New York, NY, USA, 2007), IUI ’07,ACM, pp. 225–234. 9, 14, 15

[Llo87] LLOYD J. W.: Foundations of logic programming. 4

[LRGC04] LE ROUX O., GAILDRAT V., CAUBE R.: Constraintsatisfaction techniques for the generation phase in declarativemodeling. In Geometric modeling: techniques, applications, sys-tems and tools. Springer, 2004, pp. 193–215. 4

[LT11] LATOSCHIK M. E., TRAMBEREND H.: Simulator X: AScalable and Concurrent Software Platform for Intelligent Real-time Interactive Systems. In Proceedings of the IEEE VR 2011(2011). 9, 11, 13

[Lug09] LUGRIN J.-L.: Alternative Reality and Causality in Vir-tual Environments. PhD thesis, University of Teesside, Middles-brough, United Kingdom, 2009. 9, 14, 15

[Man05] MANSOURI H.: Using Semantic Descriptions for Build-ing and Querying Virtual Environments. PhD thesis, Vrije Uni-versiteit Brussel, 2004-2005. 6, 9, 13, 14

[MPS06] MORTARA M., PATANÉ G., SPAGNUOLO M.: Fromgeometric to semantic human body models. Comput. Graph. 30,2 (Apr. 2006), 185–196. 9

[ope] OpenGL. URL: https://www.opengl.org/. 1

[Ora] ORACLE: Java3D. URL: http://www.oracle.com/.1

[Ott05a] OTTO K.: Semantic virtual environments. In Specialinterest tracks and posters of the 14th international conferenceon World Wide Web (Chiba, Japan, May 10-14, 2005), pp. 1036–1037. 5, 9, 10

[Ott05b] OTTO K.: The semantics of multi-user virtual environ-ments. In Proceedings of the Workshop towards Semantic VirtualEnvironments (2005). 9, 10

[OWL] OWL-S: Semantic Markup for Web Services. URL:http://www.w3.org/Submission/OWL-S/. 16

[PA14] PELKEY C. D., ALLBECK J. M.: Populating semanticvirtual environments. Computer Animation and Virtual Worlds25, 3-4 (2014), 405–412. 9

[PAMR05] PAPALEO L., ALBERTONI R., MARINI S., ROB-BIANO F.: An ontology-based Approach to Acquisition and Re-construction. In Workshop towards Semantic Virtual Environ-ment, Villars, Switzerland (2005). 4

[PDF06] PITTARELLO F., DE FAVERI A.: Semantic Descrip-tion of 3D Environments: A Proposal Based on Web Standards.In Proceedings of the Eleventh International Conference on 3DWeb Technology (New York, NY, USA, 2006), Web3D ’06, ACM,pp. 85–95. 5, 9, 10, 12

[PDFHH07] PAPALEO L., DE FLORIANI L., HENDLER J., HUIA.: Towards a semantic web system for understanding real worldrepresentations. In Proceedings of the Tenth International Con-ference on Computer Graphics and Artificial Intelligence (2007).6, 9, 10, 12, 13

[PDTB∗05] PELLENS B., DE TROYER O., BILLE W., KLEIN-ERMANN F., ROMERO R.: An ontology-driven approach formodeling behavior in virtual environments. In Proceedings of Onthe Move to Meaningful Internet Systems 2005: Ontology Min-ing and Engineering and its Use for Virtual Reality (WOMEUVR

2005) Workshop (Agia Napa, Cyprus, 2005), Meersman R., TariZ., Herrero P., (Eds.), no. 3762 in Lecture Notes in ComputerScience, Springer-Verlag, Springer-Verlag, pp. 1215–1224. 6, 9,11, 13, 14

[PDTK08] PELLENS B., DE TROYER O., KLEINERMANN F.:CoDePA: a conceptual design pattern approach to model behav-ior for X3D worlds. In Proceedings of the 13th InternationalSymposium on 3D web technology (Los Angeles, August 09-10,2008), pp. 91–99. 9

[Pel07] PELLENS B.: A Conceptual Modelling Approach for Be-haviour in Virtual Environments using a Graphical Notation andGenerative Design Patterns. PhD thesis, Vrije Universiteit Brus-sel, 2006-2007. 9

[PKDT06] PELLENS B., KLEINERMANN F., DE TROYER O.: In-tuitively Specifying Object Dynamics in Virtual EnvironmentsUsing VR-WISE. In Proceedings of the ACM Symposium onVirtual Reality Software and Technology (New York, NY, USA,2006), VRST ’06, ACM, pp. 334–337. 9

[PKDT09] PELLENS B., KLEINERMANN F., DE TROYER O.:A Development Environment Using Behavior Patterns to Facili-tate Building 3D/VR Applications. In Proceedings of the SixthAustralasian Conference on Interactive Entertainment (2009), IE’09, ACM, pp. 8:1–8:8. 9

[Ple99] PLEMENOS D.: Using artificial intelligence techniquesin computer graphics. In International Conference Graphicon(1999). 5

[PM08] PLEMENOS D., MIAOULIS G.: Artificial intelligencetechniques for computer graphics, vol. 159. Springer, 2008. 5

[pro] Protégé. URL: http://protege.stanford.edu/. 5

[RASF07] ROBBIANO F., ATTENE M., SPAGNUOLO M., FAL-CIDIENO B.: Part-Based Annotation of Virtual 3D Shapes. 2013International Conference on Cyberworlds 0 (2007), 427–436. 9,10, 11, 12, 13

[RS05] REITMAYR G., SCHMALSTIEG D.: Semantic world mod-els for ubiquitous augmented reality. In Proceedings of Workshoptowards Semantic Virtual Environments’ (SVE) 2005 (2005). 9,10, 12

[rul] RuleML. URL: http://wiki.ruleml.org/index.php/RuleML_Home. 4

[Rum15] RUMINSKI D.: An experimental study of spatialsound usefulness in searching and navigating through ar envi-ronments. Virtual Reality 19 (2015), 223–233. URL: http://dx.doi.org/10.1007/s10055-015-0274-4, doi:10.1007/s10055-015-0274-4. 10

[RW14] RUMINSKI D., WALCZAK K.: Semantic contextual aug-mented reality environments. In The 13th IEEE InternationalSymposium on Mixed and Augmented Reality (ISMAR 2014),ISMAR 2014 (2014), IEEE, pp. 401 – 404. doi:10.1109/ISMAR.2014.6948506. 10

[SAW] Semantic Annotations for WSDL and XML Schema- Usage Guide. URL: http://www.w3.org/TR/2007/NOTE-sawsdl-guide-20070828/. 16

[SF08] SPAGNUOLO M., FALCIDIENO B.: The Role of Ontolo-gies for 3D Media Applications. Springer, 2008. 6, 9, 10

[SF09] SPAGNUOLO M., FALCIDIENO B.: 3D Media and theSemantic Web. IEEE Intelligent Systems 24, 2 (2009), 90–96. 2

[sh1] Sweet Home 3D. URL: http://www.sweethome3d.com/. 2

[TBSDK08] TUTENEL T., BIDARRA R., SMELIK R. M.,DE KRAKER K. J.: The role of semantics in games and sim-ulations. Computers in Entertainment (CIE) 6, 4 (2008), 57. 2

submitted to COMPUTER GRAPHICS Forum (1/2017).

20 Jakub Flotynski, Krzysztof Walczak /

[TQDLC10] TRINH T.-H., QUERREC R., DE LOOR P.,CHEVAILLIER P.: Ensuring semantic spatial constraints in vir-tual environments using uml/ocl. In Proceedings of the 17th ACMSymposium on Virtual Reality Software and Technology (2010),ACM, pp. 219–226. 2

[VGLM∗07] VAN GOOL L., LEIBE B., MÜLLER P., VER-GAUWEN M., WEISE T.: 3D Challenges and a Non-In-DepthOverview of Recent Progress. In 3DIM (2007), pp. 118–132. 2

[VGRP∗10] VASILAKIS G., GARCÍA-ROJAS A., PAPALEO L.,CATALANO C. E., ROBBIANO F., SPAGNUOLO M., VAVALISM., PITIKAKIS M.: Knowledge-Based Representation of 3DMedia. International Journal of Software Engineering andKnowledge Engineering 20, 5 (2010), 739–760. 4, 9, 10, 12

[VRC07] VANACKEN L., RAYMAEKERS C., CONINX K.: In-troducing semantic information during conceptual modelling ofinteraction for virtual environments. In Proceedings of the2007 Workshop on Multimodal Interfaces in Semantic Interac-tion (New York, NY, USA, 2007), WMISI ’07, ACM, pp. 17–24.9

[VRH04] VAN-ROY P., HARIDI S.: Concepts, techniques, andmodels of computer programming. MIT press, 2004. 5

[W3Ca] W3C: A Core Ontology for Multimedia. URL: http://comm.semanticweb.org/. 3

[W3Cb] W3C: Building the Web of Data. URL: http://www.w3.org/2013/data/. 3

[W3Cc] W3C: ISO/IEC 14772-1:1997. The Virtual Reality Mod-eling Language. URL: http://www.web3d.org/x3d/specifications/. 1

[W3Cd] W3C: ISO/IEC 19775-1:2008. Extensible 3D(X3D) (2008). URL: http://web3d.org/x3d/specifications/. 1

[W3Ce] W3C: Ontology for Media Resources 1.0. URL: http://www.w3.org/TR/mediaont-10/. 3

[W3Cf] W3C: OWL. URL: http://www.w3.org/2001/sw/wiki/OWL. 4

[W3Cg] W3C: RDF. URL: http://www.w3.org/TR/2004/REC-rdf-concepts-20040210/. 4

[W3Ch] W3C: RDFa 1.1 Primer - Second Edition. URL: http://www.w3.org/TR/xhtml-rdfa-primer/. 4

[W3Ci] W3C: RDFS. URL: http://www.w3.org/TR/2000/CR-rdf-schema-20000327/. 4

[W3Cj] W3C: SWRL. URL: http://www.w3.org/Submission/SWRL/. 4

[Wal06] WALCZAK K.: Beh-VR: modeling behavior of dynamicvirtual reality contents. In Interactive Technologies and So-ciotechnical Systems. Springer, 2006, pp. 40–51. 2

[Wal08] WALCZAK K.: Flex-VR: configurable 3D web applica-tions. In Proceedings of the Conference on Human System Inter-actions (2008), IEEE, pp. 135–140. 2

[Wal12a] WALCZAK K.: Building Configurable 3D Web Appli-cations with Flex-VR. In Interactive 3D Multimedia Content,Cellary W., Walczak K., (Eds.). Springer, 2012, pp. 103–136. 2,6

[Wal12b] WALCZAK K.: Dynamic Database Modeling of 3DMultimedia Content. In Interactive 3D Multimedia Content, Cel-lary W., Walczak K., (Eds.). Springer, 2012, pp. 55–102. 2, 6

[WC03] WALCZAK K., CELLARY W.: X-VRML for AdvancedVirtual Reality Applications. Computer 36, 3 (Mar. 2003), 89–92. 2

[WCSS06] WALCZAK K., CHMIELEWSKI J., STAWNIAK M.,STRYKOWSKI S.: Extensible metadata framework for describingvirtual reality and multimedia contents. In Proc. of the IASTEDInt. Conf. on Databases and Applications (2006), Hamza, MH,(Ed.), Int Assoc Sci & Technol Dev; IASTED Tech CommDatabases, pp. 168–175. IASTED Int. Conf. on Databases andApplications, Innsbruck, Austria, Feb 14-16, 2006. 3

[WCW06] WALCZAK K., CELLARY W., WHITE M.: Virtual mu-seum exhibitions. Computer 39, 3 (Mar. 2006), 93–95. 6

[WF14] WALCZAK K., FLOTYNSKI J.: On-Demand Generationof 3D Content Based on Semantic Meta-Scenes. In Lecture Notesin Computer Science; Augmented and Virtual Reality; First Inter-national Conference, AVR 2014, Lecce, Italy, September 17-20,2014 (2014), Springer International Publishing, pp. 313–332. 9,12, 14

[WF15a] WALCZAK K., FLOTYNSKI J.: Ontology-Based Cre-ation of 3D Content in a Service-Oriented Environment. In Lec-ture Notes in Business Information Processing: Business Infor-mation Systems (2015), vol. 208, Springer Verlag, pp. 77–89.doi:10.1007/978-3-319-19027-3. 9, 14

[WF15b] WALCZAK K., FLOTYNSKI J.: Semantic Query-basedGeneration of Customized 3D Scenes. In Proceedings of the 20thInternational Conference on 3D Web Technology (New York, NY,USA, 2015), Web3D ’15, ACM, pp. 123–131. 5, 9, 12, 13, 14,15

[Whi89] WHISSEL C. M.: Emotion: Theory, research and experi-ence. In The dictionary of affect in language (New York, 1989),vol. 4. 11

[WL12] WIEBUSCH D., LATOSCHIK M. E.: Enhanced Decou-pling of Components in Intelligent Realtime Interactive Systemsusing Ontologies. In Software Engineering and Architectures forRealtime Interactive Systems (SEARIS), proceedings of the IEEEVirtual Reality 2012 workshop (2012). 9, 11, 12, 13

[WMD∗04] WHITE M., MOURKOUSSIS N., DARCY J.,PETRIDIS P., LIAROKAPIS F., LISTER P. F., WALCZAK K.,WOJCIECHOWSKI R., CELLARY W., CHMIELEWSKI J., STAW-NIAK M., WIZA W., PATEL M., STEVENSON J., MANLEY J.,GIORGINI F., SAYD P., GASPARD F.: Arco - an architecture fordigitization, management and presentation of virtual exhibitions.In Computer Graphics International (2004), IEEE ComputerSociety, pp. 622–625. 3

[Wój10] WÓJTOWICZ A.: Selective Semantic Modeling MethodSSM for Building Secure Multiaccess Virtual Environments. PhDthesis, Gdansk University of Technology, 2010. 16

[WRF14] WALCZAK K., RUMINSKI D., FLOTYNSKI J.: Build-ing contextual augmented reality environments with semantics.In Proceedings of the 20th International Conference on VirtualSystems & Multimedia VSMM 2014, Hong Kong, 9-12 Decem-ber 2014, (2014), IEEE, pp. 353–361. doi:10.1109/VSMM.2014.7136656. 5, 8, 9, 12, 14

[WSD] Web Service Semantics - WSDL-S. URL: http://www.w3.org/Submission/WSDL-S/. 16

[WWWC04] WOJCIECHOWSKI R., WALCZAK K., WHITE M.,CELLARY W.: Building virtual and augmented reality museumexhibitions. In Proceedings of the Ninth International Con-ference on 3D Web Technology (New York, NY, USA, 2004),Web3D ’04, ACM, pp. 135–144. doi:10.1145/985040.985060. 3

[XSF02] XU K., STEWART J., FIUME E.: Constraint-based au-tomatic placement for scene composition. In Graphics Interface(2002), vol. 2. 4

[ZCK∗13] ZINNIKUS I., CAO X., KLUSCH M., KRAUSS C.,

submitted to COMPUTER GRAPHICS Forum (1/2017).

Jakub Flotynski, Krzysztof Walczak / 21

NONNENGART A., SPIELDENNER T., SLUSALLEK P.: A col-laborative virtual workspace for factory configuration and eval-uation. In Collaborative Computing: Networking, Applicationsand Worksharing (Collaboratecom), 2013 9th International Con-ference Conference on (2013), IEEE, pp. 353–362. 9, 13

[ZDLB08] ZAHARIADIS T., DARAS P., LASO-BALLESTEROSI.: Towards future 3D media internet. NEM Summit (2008), 13–15. 2

[ZKDT09] ZAID L. A., KLEINERMANN F., DE TROYER O.: Ap-plying semantic web technology to feature modeling. In Proceed-ings of the 2009 ACM Symposium on Applied Computing (2009),SAC ’09, ACM, pp. 1252–1256. 9, 10, 11, 12

submitted to COMPUTER GRAPHICS Forum (1/2017).