of 63 /63
© Fraunhofer IGD Sebastian Pena Serna Enriching 3D Collections Fraunhofer-Institut für Graphische Datenverarbeitung IGD Fraunhoferstraße 5 64283 Darmstadt Tel +49 6151 155 – 468 [email protected] www.igd.fraunhofer.de

Enriching 3D Collections

Embed Size (px)



Text of Enriching 3D Collections

  • 1. Enriching 3D CollectionsSebastian Pena Serna Fraunhofer-Institut fr Graphische Datenverarbeitung IGD Fraunhoferstrae 5 64283 Darmstadt Tel +49 6151 155 468 [email protected] www.igd.fraunhofer.de Fraunhofer IGD

2. Definitions3D Collection Digital archive with multimedia material and 3D artifacts, which isassociated with semantic informationBuilding Acquisition and ingestion of digital assets and their correspondingprovenance informationAccessing Browsing and exploration of digital assets in the 3D collectionEnriching Increasing the associations within the semantic network2 Fraunhofer IGD 3. Workflow with 3D collections3 Fraunhofer IGD 4. Workflow with 3D collections : s d ing rocesuil and p Beq uir ac4 Fraunhofer IGD 5. Workflow with 3D collections Accessing: search and browse : s d ing rocesuil and p Beq uir ac5 Fraunhofer IGD 6. Workflow with 3D collections Accessing: search and browse : s vie En d ing rocesw ric an huil and p Bed inan guir no :q ta ac te6 Fraunhofer IGD 7. Building a 3D collection : s d ing rocesuil and p Beq uir ac7 Fraunhofer IGD 8. Multimedia Information Collections management Conservation ImagesBibliographic8 Fraunhofer IGD 9. Digitization 3D geometry Material properties Digital provenance9 Fraunhofer IGD 10. Processing Improve the quality of 3D artifacts Process 3D artifacts for different purposes (e.g. research, presentation)10 Fraunhofer IGD 11. Provenance Legacy and rich processing metadataused_as_derivation_source A.15-1955-dome-out.zip used_as_derivation_sourceIvoryPanel 3IvoPan_LegacyData.rdf 3IvoPan_LegacyData.rdf2009CA5307vColoured.ply4Ivory_Arc3DPro4Ivory_Arc3DPro Arc3D-A.15-1955_dmy.v3d5Ivory_MeshLa 5Ivory_MeshLacEvent.rdfcEvent.rdf bProcEvent.rdf bProcEvent.rdf has_createdcreated_derivative digitized created_derivative LegendA.15-1955-dome-A.15-1955-dome-forms_part_of out.rdfout.rdf2009CR4851_0.rdf2009CR4851_0.rdfhas_created Digitization_Process1IvoryPanel_Ob 1IvoryPanel_ObjAcqEvent.rdf jAcqEvent.rdf forms_part_of 2009CR4851_0.tif Formal_Derivation2009CA5306_0.rdf2009CA5306_0.rdf forms_part_of 2IvoryPanel_ 2IvoryPanel_forms_part_ofSub-events DocEvent.rdf DocEvent.rdf has_created 2009CA5306_0.tifData_ObjectMan_Made_Object11 Fraunhofer IGD 12. Ingestion Individual objects with high- Large acquisition campaigns quality metadatawith similar structures12 Fraunhofer IGD 13. Accessing a 3D collectionAccessing:search and browse13 Fraunhofer IGD 14. Metadata Accessing Stanford Repository 3D artifacts without searchable metadatahttp://www-graphics.stanford.edu/data/3Dscanrep/14 Fraunhofer IGD 15. Metadata Accessing [email protected] 3D artifacts with basic searchable metadata,e.g. categories, keywordshttp://shapes.aim-at-shape.net/15 Fraunhofer IGD 16. Metadata Accessing 3D-COFORM 3D artifacts with rich metadata Fundamental categories and relationships Searchable material and shape properties16 Fraunhofer IGD 17. User Accessing Administrator17 Fraunhofer IGD 18. User Accessing CH professional18 Fraunhofer IGD 19. User Accessing Internet user19 Fraunhofer IGD 20. Enriching a 3D collection vie Enw ric an h d inan gno :tate20 Fraunhofer IGD 21. 3D Shape AnnotationAim: associate digital 3D shapes with related information and knowledgeon the represented objectAnnotation: mechanism for enriching digital 3D shapes with semanticsResult: annotated shape or a semantically enriched shape, combining: the geometric description contextual information knowledge of the represented object the created relationships Fraunhofer IGD 22. SponsorsProjects: [email protected] (http://www.aimatshape.net/) Focus K3D (http://www.focusk3d.eu/) 3D-COFORM (www.3d-coform.eu) V-MusT (http://www.v-must.net/) Enhancing Engagement with 3D Heritage Data through Semantic Annotation (http://www.ddsgsa.net/projects/empire/Empire/Home.html) Semantic Annotations for 3D Artefacts (http://itee.uq.edu.au/~eresearch/projects/3dsa)Technologies: Linking Open Data (http://esw.w3.org/SweoIG/TaskForces/CommunityProjects/LinkingOpenData) 3D Internet (Alpcan et al. 2007 [33]) Fraunhofer IGD 23. Annotation Process Fraunhofer IGD 24. Annotation Process Fraunhofer IGD 25. Geometric DefinitionAim: Understand the intrinsicstructure of the digital 3Dshape (Attene et al. 2006[1], De Floriani et al. 2010[2]) Associate semantics withrelevant part(s) of the digital3D shape (Spagnuolo andFelcidieno 2009 [3]) Fraunhofer IGD 26. Geometric DefinitionTechniques: Sketching, painting, outlining, fitting, segmenting, and structuring These are driven by different principles (Attene at al. 2006 [4],Shamir 2008 [5] and Chen et al. 2009 [6]) Fraunhofer IGD 27. Geometric DefinitionPrinciples: RANSAC (Schnabel et al. 2007 [7]) Curvature analysis (Madeira et al. 2007 [8]) Contour analysis (Liu and Zhang 2007 [9]) Discrete operators (Reuter et al. 2009 [10]) Physics (Fang et al. 2011 [11]) Concavity (Au et al. 2011 [12]) Fraunhofer IGD 28. Geometric DefinitionStrategies:Hierarchical segmentation (Shapira et al. 2010 [13], Wang et al. 2011 [14], Ho and Chuang 2011 [15]) Fraunhofer IGD 29. Geometric DefinitionStrategies: Combination of geometric principles with other concepts aboutthe represented shape (Attene et al. 2009 [16], Golovinsliy andFankhouser 2009 [17], Kalogerakis et al. 2010 [18]). Fraunhofer IGD 30. Geometric DefinitionStrategies: Skeletons to identify the structure of the digital 3D (Tierny et al. 2007[19], Shapira et al. 2008 [20]) and/or by means of fitting primitives(Attene et al. 2006 [21]). Fraunhofer IGD 31. Geometric DefinitionStrategies: User assisted segmentationfor complex digital 3D shapesor for additional requirements,e.g. functions or styles (DeFloriani et al. 2008 [22], Miaoet al. 2009 [23], Bergamascoet al. 2011 [24]). Fraunhofer IGD 32. Geometric DefinitionStrategies:Manual segmentation, sketching (Ji et al. 2006 [25]), painting (Papaleo and De Floriani 2010 [26]) or outlining regions (Pena Serna et al. 2011 [27]). Fraunhofer IGD 33. Geometric DefinitionStrategies:Segmentation refinement (Klaplansky and Tal 2009 [28]). Fraunhofer IGD 34. Geometric DefinitionSpecific Requirements:Scenes (Knopp et al. 2011 [29])Developable segments (Julius et al. 2005 [30])Best view (Mortara and Spagnuolo 2009 [31]).Identify adjectives (Simari et al. 2009 [32]) Fraunhofer IGD 35. Geometric DefinitionChallenges:Difficult to generate a plausible and context-aware geometricdefinition for different classes of objects.The current strategies cannot easily be mapped to the differentapplications requirements within a given domain.There are few approaches trying to map principles to specificapplications requirements.A combination of principles, strategies and user guidance couldgenerate the expected results. Fraunhofer IGD 36. Annotation Process Fraunhofer IGD 37. Structured Information and KnowledgeThere is a vast amount of existent information and knowledge related to any digital 3D shape:Information related to the intrinsic structure of the 3D shapeInformation related to the meaning of the represented objectInformation related to the digital provenanceKnowledge related to the application domain Fraunhofer IGD 38. Structured Information and KnowledgeStructured Information for describing the intrinsic structure of the digital 3D shape (Papaleo and De Floriani 2010 [26], Attene et al. 2009 [16]). Fraunhofer IGD 39. Structured Information and KnowledgeStructured Information for describing digital 3D shapes using concepts within a particular domain (Catalano et al. 2009 [34], De Luca et al. 2011 [35], Mortara et al. 2006 [36]). Fraunhofer IGD 40. Structured Information and KnowledgeStructured Information in the engineering domain Product and Manufacturing Information (PMI) Geometric Dimensions and Tolerances (GD&T) Functional Tolerancing and Annotation (FT&A). Standard ASME Y14.41-2003 Digital ProductData Definition Practices ISO 1101:2004 Geometrical ProductSpecifications (GPS) - Geometrical tolerancing.(Spatial Corp.) Fraunhofer IGD 41. Structured Information and KnowledgeStructured Information in the Cultural Heritage domain based on CIDOC-CRM http://cidoc.ics.forth.gr/ (Rodriguez- Echavarria et al. 2009 [37], Havemann et al. 2009 [38]). Fraunhofer IGD 42. Annotation Process Fraunhofer IGD 43. Mechanisms for AnnotatingDifferent mechanisms have been proposed, which vary depending on:application domaindegree of user intervention that they requiretechnology supporting themdegree of structured information which they involve. Fraunhofer IGD 44. Mechanisms for AnnotatingApplication domain Product design (Andre and Sorito2002 [39]) Architecture (Pittarello and Gatto2011 [40]) Cultural Heritage (Hunter and Gerber2010 [41]) Chemistry (Gawronski and Dumontier2011 [42]) Medicine (Trzupek et al. 2011 [43]) Fraunhofer IGD 45. Mechanisms for AnnotatingUser interventionSemi-automatic mechanisms normally require of a degree of user intervention to define an annotation (Shapira et al. 2010 [13], Kalogerakis et al. 2010 [18]). Fraunhofer IGD 46. Mechanisms for AnnotatingSupporting technology:stand-alone modeling systemsstand-alone 3D viewersSiemens NX (Pena Serna et al. 2011 [27])web based viewers (Hunter et al. 2010 [44]) Fraunhofer IGD 47. Annotation Process Fraunhofer IGD 48. Representation of the AnnotationApproach to structure, store and transmit the annotating process outputImportant for the annotations indexing, retrieval and reutilization.There is no agreed format for this. Fraunhofer IGD 49. Representation of the AnnotationStrategies: Persistent annotationsStore the annotation in a database based on a semantic model.The model describes the associations or relations between different media ([16], [27], Hunter et al. 2010 [45]). Fraunhofer IGD 50. Representation of the AnnotationStrategies: Transient annotations Store and transmit annotations in a data file. MPEG-7 (Bilasco et al. 2006 [46]) VRML / X3D (Pittarello and Faveri 2006 [47], [40], [26]) Jupiter (JT) Data Format Product Representation Compact (PRC) Data Format COLLADA ([37], [38]) Universal 3D Data Format ASME Y14.41 Digital Product Definition Data Practices Fraunhofer IGD 51. Representation of the AnnotationIssues: Stability, flexibility and easy of use There is no notion of annotation representation. It is considered as a piece of text, which is stored in a database or as atag on a digital 3D shape. Annotations interoperability Degree of independency from transient digital 3D shapes. Fraunhofer IGD 52. Enriching a 3D collectionChallenges and OpportunitiesThis remains an active area of research. Different challenges need to be solved to fully support a semantic enrichment pipeline: Automatically extracting information from a digital 3D shape Modeling semantic information Automatically linking it to the digital 3D shape Using standards to store, interoperate, and preserve annotations in thelong term Fraunhofer IGD 53. Enriching a 3D collectionChallenges and OpportunitiesOpportunities of using semantically aware 3D shapes: searching 3D shapes intelligently interacting with semantically aware 3D shapes shape matching or deriving meaning of new shapes high-level editing goal oriented 3D synthesizing knowledge management semantic visualization and interaction Fraunhofer IGD 54. Workflow with 3D collectionsAccessing:search and browse: s vie End ing rocesw rican h uil and pBed in an g uir no : q taac te54 Fraunhofer IGD 55. Enabling Technologies Cloud Computing Storage and computation capacity online 3D Internet Visualization of 3D artifacts on standard web browsers Mobile devices Access and visualization on the move55 Fraunhofer IGD 56. Emerging Challenges Define workflows Create services Enable intuitive access Provide contextualized interfaces User involvement and engagement56 Fraunhofer IGD 57. References [1] ATTENE M., BIASOTTI S., MORTARA M., PATAN G., SPAGNUOLO M., FALCIDIENO B.: Computational methods for understanding 3D shapes. Computers & Graphics30, 3 (June 2006), 323333. [2] DE FLORIANI L., MAGILLO P., PAPALEO L., PUPPO E.: Shape modeling and understanding: Research trends and results of the G3 group at DISI. [3] SPAGNUOLO M., FALCIDIENO B.: 3D media and the semantic web. IEEE Intelligent Systems (March/April 2009), 9096. [4] ATTENE M., KATZ S., MORTARA M., PATAN G., SPAGNUOLO M., TAL A.: Mesh segmentation - a comparative study. In Shape Modeling International (2006). [5] SHAMIR A.: A survey on mesh segmentation techniques. Computer Graphics Forum 27, 6 (2008), 15391556. [6] CHEN X., GOLOVINSKIY A., FUNKHOUSER T.: A benchmark for 3D mesh segmentation. In ACM SIGGRAPH 2009 papers (New Orleans, Louisiana, 2009), ACM, pp. 112. [7] SCHNABEL R., WAHL R., KLEIN R.: Efficient RANSAC for Point-Cloud shape detection. Computer Graphics forum 26, Number 2 (June 2007), 214226. [8] MADEIRA J., SILVA S., STORK A., PENA SERNA S.: Principal Curvature-Driven segmentation of mesh models: A preliminary assessment. In 15 EPCG - EncontroPortugus de Computao Grfica. (2007). [9] LIU R., ZHANG H.: Mesh segmentation via spectral embedding and contour analysis. Volume 26 (2007), Number 3. [10] REUTER M., BIASOTTI S., GIORGI D., PATAN G., SPAGNUOLO M.: Discrete Laplace-Beltrami operators for shape analysis and segmentation. Computers & Graphics33, 3 (June 2009), 381390. [11] FANG Y., SUN M., KIM M.: Heat-Mapping: a robust approach toward perceptually consistent mesh segmentation. IEEE Computer Vision and Pattern Recognition (CVPR)2011 (2011), pp 21452152. [12] AU O. K., ZHENG Y., CHEN M., XU P., TAI C.: Mesh segmentation with concavity-aware fields. IEEE Trans. Vis. Comp. Graphics (2011). [13] SHAPIRA L., SHALOM S., SHAMIR A., COHEN-OR D., ZHANG H.: Contextual part analogies in 3D objects. Int. J. Comput. Vision 89, 2-3 (2010), 309326. [14] WANG Y., XU K., LI J., ZHANG H., SHAMIR A., LIU L., CHENG Z., XIONG Y.: Symmetry hierarchy of Man-Made objects. Computer Graphics Forum 30, 2 (2011), 287296. [15] HO T., CHUANG J.: Volume based mesh segmentation. Journal of Information Science and Engineering 27 (2011). [16] ATTENE M., ROBBIANO F., SPAGNUOLO M., FALCIDIENO B.: Characterization of 3D shape parts for semantic annotation. Computer-Aided Design 41, 10 (Oct. 2009),756763. [17] GOLOVINSKIY A., FUNKHOUSER T.: Consistent segmentation of 3D models. Computers & Graphics 33, 3 (June 2009), 262269. [18] KALOGERAKIS E., HERTZMANN A., SINGH K.: Learning 3D Mesh Segmentation and Labeling. ACM Transactions on Graphics 29, 3 (2010). Fraunhofer IGD 58. References [19] TIERNY J., VANDEBORRE J.-P., DAOUDI M.: Topology driven 3d mesh hierarchical segmentation. In Proceedings of the IEEE International Conference on ShapeModeling and Applications 2007 (Washington, DC, USA, 2007), IEEE Computer Society, pp. 215220. [20] SHAPIRA L., SHAMIR A., COHEN-OR D.: Consistent mesh partitioning and skeletonisation using the shape diameter function. The Visual Computer: International Journalof Computer Graphics 24, 4 (Mar. 2008). [21] ATTENE M., FALCIDIENO B., SPAGNUOLO M.: Hierarchical mesh segmentation based on fitting primitives. The Visual Computer: International Journal of ComputerGraphics 22 (2006), 181193. [22] DE FLORIANI L., PAPALEO L., CARISSIMI N.: A Java3D framework for inspecting and segmenting 3D models. In Proceedings of the 13th international symposium on 3Dweb technology (Los Angeles, California, 2008), ACM, pp. 6774. [23] MIAO Y., FENG J., WANG J., JIN X.: User-controllable mesh segmentation using shape harmonic signature. Progress in Natural Science 19, 4 (Apr. 2009), 471478. [24] BERGAMASCO F., ALBARELLI A., TORSELLO A.: Semi-supervised segmentation of 3D surfaces using a weighted graph representation. In Proceedings of the 8thinternational conference on Graph-based representations in pattern recognition (GbRPR11) (2011). [25] JI Z., LIU L., CHEN Z., WANG G.: Easy mesh cutting. Computer Graphics Forum 25, 3 (2006), 283291. [26] PAPALEO L., DE FLORIANI L.: Manual segmentation and semantic-based hierarchical tagging of 3D models. (2010) pp. 2532. [27] PENA SERNA S., SCOPIGNO R., DOERR M., THEODORIDOU M., GEORGIS C., PONCHIO F., STORK A.: 3D-centered media linking and semantic enrichment throughintegrated searching, browsing, viewing and annotating. In VAST11: The 12th International Symposium on Virtual Reality, Archaeology and Intelligent Cultural Heritage (Prato,Italy, 2011). [28] KAPLANSKY L., TAL A.: Mesh segmentation refinement. In Computer Graphics Forum (Pacific Graphics), 28(7) (Oct. 2009), pp. 19952003. [29] KNOPP J., PRASAD M. , VAN GOOL L. : Scene Cut: Class-specific Object Detection and Segmentation in 3D Scenes. In 3DIMPVT, Hangzhou, 2011 [30] JULIUS D., KRAEVOY V., SHEFFER A.: D-charts: Quasi-developable mesh segmentation. In Computer Graphics Forum, Proceedings of Eurographics 2005 (Dublin,Ireland, 2005), vol. 24, Eurographics, Blackwell, pp. 581590. [31] MORTARA M., SPAGNUOLO M.: Semantics-driven best view of 3D shapes. Computers & Graphics 33, 3 (June 2009), 280290. [32] SIMARI P., NOWROUZEZAHRAI D., KALOGERAKIS E., SINGH K.: Multi-objective shape segmentation and labeling. In Proceedings of the Symposium on GeometryProcessing (Berlin, Germany, 2009), Eurographics Association, pp. 14151425. [33] ALPCAN T., BAUCKHAGE C., KOTSOVINOS E.: Towards 3d internet: Why, what, and how? In Proceedings of the International Conference on Cyberworlds CW 07(October 2007), pp. 95 99. [34] CATALANO C., CAMOSSI E., FERRANDES R., CHEUTET V., SEVILMIS N.: A product design ontology for enhancing shape processing in design workflows. Journal ofIntelligent Manufacturing 20, 5 (Oct. 2009), 553567. 3 Fraunhofer IGD 59. References [35] LUCA L. D., BUSAYARAT C., STEFANI C., VRON P., FLORENZANO M.: A semantic-based platform for the digital analysis of architectural heritage. Computers &Graphics 35, 2 (Apr. 2011), 227241. [36] MORTARA M., PATAN G., SPAGNUOLO M.: From geometric to semantic human body models. Computers&Graphics 30, 2 (Apr. 2006), 185196. [37] RODRIGUEZ ECHAVARRIA K., MORRIS D., ARNOLD D.: Web based presentation of semantically tagged 3D content for public sculptures and monuments in the UK. InProceedings of the 14th International Conference on 3D Web Technology (Darmstadt, Germany, 2009), ACM, pp. 119126. [38] HAVEMANN S., SETTGAST V., BERNDT R., EIDE., FELLNER D. W.: The Arrigo showcase reloaded - towards a sustainable link between 3D and semantics. J. Comput.Cult. Herit. 2, 1 (2009), 113. [39] ANDRE P., SORITO R.: Product manufacturing information (PMI) in 3D models: a basis for collaborative engineering in product creation process (PCP). In 14th EuropeanSimulation Symposium and Exhibition (2002). [40] PITTARELLO F., GATTO I.: ToBoA-3D: an architecture for managing top-down and bottom-up annotated 3D objects and spaces on the web. In Web3D 11 Proceedings ofthe 16th International Conference on 3D Web Technology (2011). [41] HUNTER J., GERBER A.: Harvesting community annotations on 3D models of museum artefacts to enhance knowledge, discovery and re-use. Journal of Cultural Heritage11, 1 (2010), 8190. [42] GAWRONSKI A., DUMONTIER M.: MoSuMo: a semantic web service to generate electrostatic potentials across solvent excluded protein surfaces and binding pockets.Computers & Graphics 35, 4 (Aug. 2011), 823830. [43] TRZUPEK M., OGIELA M. R., TADEUSIEWICZ R.: Intelligent image content semantic description for cardiac 3D visualisations. Engineering Applications of ArtificialIntelligence In Press, Corrected Proof (2011). [44] HUNTER J., YU C.-H., NAKATSU R., TOSA N., NAGHDY F., WONG K., CODOGNET P.: Supporting multiple perspectives on 3D museum artefacts through interoperableannotations. Vol. 333 of IFIP Advances in Information and Communication Technology. Springer Boston, 2010, pp. 149159. [45] HUNTER J., COLE T., SANDERSON R., VAN DE SOMPEL H.: The open annotation collaboration: A data model to support sharing and interoperability of scholarlyannotations. (2010) [46] BILASCO I. M., GENSEL J., VILLANOVA-OLIVER M., MARTIN H.: An MPEG-7 framework enhancing the reuse of 3D models. In Proceedings of the eleventh internationalconference on 3D web technology (Columbia, Maryland, 2006), ACM, pp. 6574. [47] PITTARELLO F., FAVERI A. D.: Semantic description of 3D environments: a proposal based on web standards. In Proceedings of the eleventh international conference on3D web technology (Columbia, Maryland, 2006), ACM, pp. 8595. Fraunhofer IGD 60. Thank You!Sebastian Pena SernaFraunhofer-Institut fr Graphische DatenverarbeitungIGDFraunhoferstrae 564283 DarmstadtTel +49 6151 155 468seb[email protected] Fraunhofer IGD 61. IVB: Integrated Viewer /Browser Access and enrichment of 3D collections Searching and browsing Searching: flexible formulation of queries Browsing: exploration of multiple results and query refinement Viewing and Annotating Viewing: inspection and analysis of multimedia objects Annotating: building and enrichment of semantic relationships61 Fraunhofer IGD 62. IVB: Searching and Browsing Interface62 Fraunhofer IGD 63. IVB: Viewing and Annotating Interface63 Fraunhofer IGD