Upload
others
View
7
Download
0
Embed Size (px)
Citation preview
Copyright University of Southampton IT Innovation Centre 2017 1 www.gravitate-project.eu
Geometric and Semantic Matching for Cultural
Heritage Artefacts
Paul Walland
IT Innovation
ResearchSpace Symposium
28-07-2017
Copyright University of Southampton IT Innovation Centre 2017 2 www.gravitate-project.eu
Initial Ambition of GRAVITATE Project
Re-Assembly
Re-Unification
Re-Association
------------------------
Similarity Search
Copyright University of Southampton IT Innovation Centre 2017 3 www.gravitate-project.eu
Copyright University of Southampton IT Innovation Centre 2017 4 www.gravitate-project.eu
British Museum, London Cyprus Museum, Cyprus
Ashmolean Museum, Oxford Fitzwilliam Museum, Cambridge
Copyright University of Southampton IT Innovation Centre 2017 5 www.gravitate-project.eu
Re-Unification
Head in the British Museum
Statue in the Cyprus Museum
Re-unify 3D models of the statue
3D print copies for re-unification
Re-unify statue in a virtual museum
Copyright University of Southampton IT Innovation Centre 2017 6 www.gravitate-project.eu
Re-Association
Artefact with semantic description Material, decoration, size, shape, period, style, glaze, …
Data in another museum catalogue
Copyright University of Southampton IT Innovation Centre 2017 7 www.gravitate-project.eu
Re-Association
Find similarities leading to new insights about past cultures
Copyright University of Southampton IT Innovation Centre 2017 8 www.gravitate-project.eu
Copyright University of Southampton IT Innovation Centre 2017 9 www.gravitate-project.eu
Data on Artifacts is in many different forms:
Physical Descriptors: Image (2D and 3D) Geometrical Material Pattern Colour … etc
Abstract (non-physical, semantic) Age Condition Style Location Part (e.g. foot or ear) Specialised curatorial terms … etc
And in many different places!
Copyright University of Southampton IT Innovation Centre 2017 10 www.gravitate-project.eu
Fragments of Terracotta Statues from Salamis
Copyright University of Southampton IT Innovation Centre 2017 11 www.gravitate-project.eu
Problem Statement British Museum, London Cyprus Museum, Cyprus
Ashmolean Museum, Oxford Fitzwilliam Museum, Cambridge
Copyright University of Southampton IT Innovation Centre 2017 12 www.gravitate-project.eu
Problem Statement
Jigsaws
in 3D
most pieces missing
all the edges worn off
many puzzles jumbled together
pieces spread across many countries
Copyright University of Southampton IT Innovation Centre 2017 13 www.gravitate-project.eu
Problem Statement
Front Back
Faded
Cracked
Eroded
Copyright University of Southampton IT Innovation Centre 2017 14 www.gravitate-project.eu
Problem Statement Re-Assembly
You have some (eroded) pieces.
The computer fits them together.
This is hard in 2 dimensions, never mind in 3D!
Localised geometry
3D descriptors
Localised detail
Colour descriptors
Copyright University of Southampton IT Innovation Centre 2017 15 www.gravitate-project.eu
We need to Gather all the Knowledge
Catalogue data with text
descriptions
Archaeological papers
Excavation notes
Chemical analysis
X-ray fluorescence
3D scanning models
Copyright University of Southampton IT Innovation Centre 2017 16 www.gravitate-project.eu
We need to Gather all the Knowledge
Catalogue data with text
descriptions
Archaeological papers
Excavation notes
Chemical analysis
X-ray fluorescence
3D scanning models
Copyright University of Southampton IT Innovation Centre 2017 17 www.gravitate-project.eu
We need to Gather all the Knowledge
Catalogue data with text
descriptions
Archaeological papers
Excavation notes
The British Museum has 2.5M objects described in CIDOC CRM Catalogue data from Ashmolean, Fitzwilliam and Cyprus museums is now mapped to the same data model
Object hasNote “…lots of free-form text added by the curator”
Using Natural Language Processing to extract and encode meaning from this text: • References to papers, to catalogue entries • Parts and features • Conservation condition • Measurements • … CIDOC CRM / CRMarchaeo / etc
Copyright University of Southampton IT Innovation Centre 2017 18 www.gravitate-project.eu
We need to Gather all the Knowledge
Chemical analysis
X-ray fluorescence
3D scanning models
Volume, area thickness,
curvature, …
Colour, texture, distance from convex hull, …
Feature detection
Faceting: front / back
/ fracture
Part annotation
CRMdig CH Artefact Partonomy
All fragments (211) are scanned, in a
variety of resolutions and with a variety of
scanners
Copyright University of Southampton IT Innovation Centre 2017 19 www.gravitate-project.eu
What is our raw data?
Semantic data
From catalogues
From curators
Measurements of the fragments
3D scanning 3D models
Chemical analysis, pigment composition
X-ray fluorescence
…etc
Copyright University of Southampton IT Innovation Centre 2017 20 www.gravitate-project.eu
Fundamental Geometric Properties and Descriptors
Library of tools for the analysis of the geometric and colorimetric properties of a fragment’s shape Curvature-based
Distance-based
Volume, area, thickness
..
Library of descriptors and signatures to evaluate similarity geometry
color
textures
Mean Shape index
Distance from convex hull
Average geodesic distance
Copyright University of Southampton IT Innovation Centre 2017 21 www.gravitate-project.eu
Faceting Faceting partitions a fragment into
distinct facets by sharpness and roughness
Each facet will be classified into: Skin (original) boundary surface of
the unbroken object
Fracture created when the object broke
Re-Assembly modules use the facets differently: Skin facets local continuity
in color and shape
Fracture facets complementarity of abraded local shape
(SotA faceting from predecessor PRESIOUS)
Copyright University of Southampton IT Innovation Centre 2017 22 www.gravitate-project.eu
NLP Example Head from Toumba, Ashmolean Museum, Oxford, Inv. no. 1909.837=1891.468 (Tubbs and Munro 1891, 149, fig. 7; SCE IV:2, pl. IX bottom; Hermary 1991, pl. XXXIX:a). The lower part of the beard is restored. Restored height of the head: 39 cm. Upper part of (conical?) helmet missing; upturned cheek-pieces; preserved part of helmet is covered with stamped circles. The face is rectangular; almond-shaped eyes, ridged eyelids, broad 'feathered' eyebrows. The beard is neatly separated from the rest of the face by a 'rope' line as on Cat. no. 73. The restored lower part of the beard must originally have been square with rounded corners, like that of Cat. no. 73. It is grooved vertically with each groove terminating in a curl (cf. a beard fragment from Samos, no. T324, Schmidt 1968, pl. 60). A fringe of curly hair appears on the forehead below the lower edge of the helmet and at the temples, as on Cat. no. 73. Trapezoidal moustache with grooved strokes. The ears are rendered in the same way as on Cat. no. 73. Black paint on beard, hair, moustache, eyebrows and irises; red paint on lips. This head was assigned by Gjerstad (SCE IV: 2, 108) to the Western Neo-Cypriote style. It has several stylistic similarities with Tamassos heads Cat. nos 68-69, though the expression of Cat. no. 74 is more smiling (Karageorghis 1993, 32, Cat. no. 74).
Ashmolean reference AN1909.837
The discourse styles for Salamis artefacts in BM, Ashmolean, Fitzwilliam and Cyprus museums are similar, but different
We attempt to automate annotation of the artefact semantics from such sources
Copyright University of Southampton IT Innovation Centre 2017 23 www.gravitate-project.eu
ResearchSpace
• British Museum’s platform impementing CIDOC CRM as framework
• Components: Search, Clipboard, Data Annotation, Data Argumentation, Image Annotation/ Argumentation, Workflow, User Defined Pages/ Publication • Shows faceted relationships
• Combines results from multiple datasets
• Supports contextual search
Copyright University of Southampton IT Innovation Centre 2017 24 www.gravitate-project.eu
Similarity Search
Similarity matching of artefacts Start with search criteria or set of example objects
Geometric descriptors » size, part, colour, texture, edge/surface » Geometric matching algorithms
Semantic metadata » type, style, location, size, part, colour, texture, damage, hypothesis/conjecture » Graph matching
Fusion of match results Aggregation of the result space + Uncertainty analysis
Not a black box approach
Iterative expansion / refinement of match results with expert feedback
Copyright University of Southampton IT Innovation Centre 2017 25 www.gravitate-project.eu
Demonstration of GRAVITATE Platform
Copyright University of Southampton IT Innovation Centre 2017 26 www.gravitate-project.eu
Copyright University of Southampton IT Innovation Centre 2017 27 www.gravitate-project.eu
Copyright University of Southampton IT Innovation Centre 2017 28 www.gravitate-project.eu
Copyright University of Southampton IT Innovation Centre 2017 29 www.gravitate-project.eu
Copyright University of Southampton IT Innovation Centre 2017 30 www.gravitate-project.eu
Copyright University of Southampton IT Innovation Centre 2017 31 www.gravitate-project.eu
Copyright University of Southampton IT Innovation Centre 2017 32 www.gravitate-project.eu
Copyright University of Southampton IT Innovation Centre 2017 33 www.gravitate-project.eu
Copyright University of Southampton IT Innovation Centre 2017 34 www.gravitate-project.eu
Copyright University of Southampton IT Innovation Centre 2017 35 www.gravitate-project.eu
Copyright University of Southampton IT Innovation Centre 2017 36 www.gravitate-project.eu
Copyright University of Southampton IT Innovation Centre 2017 37 www.gravitate-project.eu
Copyright University of Southampton IT Innovation Centre 2017 38 www.gravitate-project.eu
Copyright University of Southampton IT Innovation Centre 2017 39 www.gravitate-project.eu
Copyright University of Southampton IT Innovation Centre 2017 40 www.gravitate-project.eu
Copyright University of Southampton IT Innovation Centre 2017 41 www.gravitate-project.eu
Copyright University of Southampton IT Innovation Centre 2017 42 www.gravitate-project.eu
Copyright University of Southampton IT Innovation Centre 2017 43 www.gravitate-project.eu
Copyright University of Southampton IT Innovation Centre 2017 44 www.gravitate-project.eu
Copyright University of Southampton IT Innovation Centre 2017 45 www.gravitate-project.eu
Desktop app
Copyright University of Southampton IT Innovation Centre 2017 46 www.gravitate-project.eu
GRAVITATE-PROJECT.EU
Thank you for your attention!