1

Minerals Intelligence Network for Europe

Minerals4EU deliverable D5.3: The European Union

Minerals Knowledge Data Platform (EU-MKDP) Title of the project: Minerals Intelligence Network for Europe –

Minerals4EU

Grant Agreement number: 608921

Funding Scheme: FP7-NMP-2013: 4.1-3 –CSA (COORDINATING)

Start date: 01.09.2013

Duration: 24 months

Document title: Minerals4EU deliverable D5.3: the EU-MKDP

Work package: WP5

Author(s): Compiled by Daniel Cassard (BRGM) and WP5 partners

Date of delivery: 04/09/2015

Dissemination level: PU/PP/RE/CO1

Reviewed by: Co-authors

Status of the document: Draft/Final Document location: https://vyvi-some2.vy-verkko.fi/gtk/Minerals4EU/Deliverables

Video location: http://minerals4eu.brgm-rec.fr/videos/00-EU-

MKDP_Final2.mp4

Project web site: http://www.minerals4eu.eu

1 PU=public, PP= Restricted to other programme participants (including the Commission Services), RE= Restricted to a group specified by the consortium (including the Commission Services), CO= Confidential, only for members of the consortium (including the Commission Services)

2

Contents 1 Introduction ..................................................................................................................................... 3

2 The EU-MKDP general architecture................................................................................................. 3

3 Development of the EU-MKDP system............................................................................................ 4

4 The EU-MKDP Web Portal and dedicated services ......................................................................... 8

5 Support to geological surveys for web services implementation ................................................... 8

6 References ..................................................................................................................................... 10

3

1 Introduction Data related to raw materials, either metallic, industrial or construction materials, primary or from wastes (mining and industrial) are, most of the time, available in Europe. However, they are often scattered amongst a variety of institutions, including governmental agencies, universities, NGOs and industries. These data are often stored in databases with their own design/architecture and vocabulary, making any attempt of merging – even partially - in view of a compilation difficult and time consuming. The problems regarding availability, quality, organization, accessibility and sharing of data are common to a number of policies and are experienced across the various levels of public authorities. Solving these problems requires measures that address exchange, sharing, access and use of interoperable spatial data and services both at national and European levels. This is the aim of the INSPIRE Directive (2007, "Establishing an Infrastructure for Spatial Information in the European Community"), but its implementation in the Member States has just started and achieving those objectives represents a major challenge.

The development of the EU-MKDP (Minerals Knowledge Data Platform) aims to give a simplified, user-friendly and efficient access to all available and new data related to mineral resources from national geological surveys, scientific institutes and universities, relevant industries and professional organizations, as well as from former European projects such as ProMine (information on both mineral deposits and anthropogenic concentrations resulting from mining and downstream activities - Serrano et al., 2010 ; Cassard et al., in press) and EuroGeoSource (information on energy and mineral resources, extraction locations, production, reserves - Gruijters and Fruijtier, 2011; EuroGeoSource, 2013). The system is also designed to accommodate and manage semi- and non-structured data (e.g., syntheses and statistics in the form of graph charts, time-series related to exploration and primary reserves and resources, secondary resources, exploitation technologies including ore beneficiation, extraction technologies, end product development and waste management practices, European market survey and raw material demand…).

The role of the EU-MKDP is to provide the end user with a seamless access to data related to the whole value chain from deposit exploration, mining and extraction of ore, exploitation technologies to treatment of end-of-life products and the generation of "new" materials, consumption and demand, with the ability to combine all spatial and non-spatial pertinent information in a single reference system.

2 The EU-MKDP general architecture

The principles of the EU-MKDP architecture follow the Implementing rules of the European INSPIRE Directive (2007) to make data and services interoperable across Europe over a distributed infrastructure (Cassard et al., 2015).

The open architecture defines various components connected together to build the EU-MKDP, taking into account state-of-the-art developments to enhance the performance, stability, sustainability and user friendliness of the system. The connections between the components are specified by standardized interfaces.

4

The main components of the EU-MKDP architecture are the followings:

• Data on primary and secondary resources from databases provided by geological surveys and former European projects such as ProMine and EuroGeoSource. These data, semantically harmonized using common terminology (Schubert et al., 2014), are delivered through INSPIRE/OGC compliant web services and according to the common data model that extends the INSPIRE core data model (INSPIRE Data specification on MR - INSPIRE Thematic Working Group Mineral Resources, 2013) which is based on the ERML v.2 data model (EarthResourceML, 2013; Vuollo et al., 2010, 2012; Cassard et al., 2012; Raymond et al., 2012).

• A system which gives access to available pertinent layers (e.g., geological, geochemical, geophysical, geographical, land use maps…) provided as interoperable services that are registered in a metadata catalogue accessible via an INSPIRE discovery service.

• New data and information delivered during the project by other work packages (statistics

analyses, analyses of supply and demand in the EU, stocks, flows, including trends in what products are put on the market and their composition, analyses of the composition of waste products and wastes arising from pre-processing and their geographic location in EU) in various formats will be accessible through the portal. This applies also to any "open data" of interest for the project. This semi- or non-structured information will be enriched by the provider with a metadata record (e.g., Dublin Core type, see below), then processed and indexed, first extracting relevant information from the documents (e.g., named-entity recognition on atomic elements such as names of locations, expressions of time, thematic specific terms…), then classifying this information according to three facets (spatial, temporal and thematic) and creating indexes for each facet. The way to display the list of selected documents will take into account these three facets.

• Besides a map viewer, the EU-MKDP also includes a search engine for all data, layers and documents delivered by the project and all external pertinent documents.

3 Development of the EU-MKDP system Principle: Participating surveys have set up their National Provider databases and populated them with data. Each database contains the data of the geological survey, transformed from their own original format to the INSPIRE data model and semantically harmonised using commonly agreed terminology resulting in 50+ code lists (see INSPIRE Thematic Working Group Mineral Resources, 2013; EarthResourceML, 2013; Schubert et al., 2014; GEUS, 2015). The data are then transformed into an INSPIRE compliant WFS format using the Deegree3 framework.

► The harvesting system consists of a harvesting application and a database. The harvesting application retrieves WFS formatted data from each provider, harvests data from it, transforms it into proper form and then stores it in the Central Harvesting database. Based on the EuroGeoSource project, it was initially decided to develop the harvesting system using the GeoKettle ETL tool. During the development of the harvesting scripts it turned out that the GeoKettle ETL tool could not properly parse WFS 2.0 formatted XML files. The biggest obstacle was parsing XPath from M4EU XML files.

5

For this reason it could not properly navigate through the XML elements and attributes of the XML document. Due to these insufficiencies of the GeoKettle ETL tool it was decided to test other solutions. One of those was JAXB (Java Architecture for XML Binding API) which was tested both with NetBeans and Eclipse SDK using their own JAXB components and several different JAXB libraries. The JAXB offers easy access to XML documents from within applications written with Java programming language. In order to access and harvest data from an XML document with JAXB, the first task is to generate several Java classes that represent the M4EU schema. Then it is necessary to unmarshal the XML document into Java content objects. The Java content objects represent the content and organization of the XML document. It is then possible to store this java objects in the database. One of the advantages is that the objects contain all the relations to other objects. Aggregation and other manipulation of data in the harvesting system, if needed, will be performed by procedures within the harvesting application and PostgreSQL/PostGIS database. The harvesting database has the same structure as the database that each national provider has setup. The delivery of data from the harvesting system to the diffusion system is carried out by sending SQL update scripts.

NOTE: The harvesting system has been developed and implemented by the Geological Survey of Slovenia (GeoZS) and it is currently running on GeoZS servers for both Minerals4EU and EURare projects. In accordance with the commitment taken by the WP5 Team, this system is based on open software and is self-contained. Following the decisions which will be taken at the end of the project, it can easily be moved – and the diffusion system as well - to other installation/platforms if needed. ► The Diffusion Database is a copy, specialized for data delivery and computations based on the data stored in the Harvesting Database. This database is kept updated with the Harvesting Database using database synchronization. This synchronization relies on SQL scripts forwarded from the Harvesting Database to the Diffusion Database. The specialization of this database aims to optimize the response time of the services proposed to the users.

The services on top of the Diffusion Database are:

• Simple map visualizations using WMS’s (Web Map Service). (Operational) • Search facilities: these facilities allow the user to search in the whole database and

documents. (Operational) • Minerals4EU’s dedicated services (e.g., the e-Minerals Yearbook). (Operational) • Data download: in some case (and depending on the access rights to the data) the data will

be available for download so that they can be processed in other applications (e.g., desktop GIS…) Because the structure of the database is extremely complex, only selected parts of this one (called "portrayals") will be made available for download. (Operational)

► Besides the Diffusion database, an additional interface allows experts to feed the knowledge base with non-structured data (reports, synthesis notes, time series…, in various formats). This interface allows the expert either to add a document (the document will be part of the EU-MKDP and retrievable within the EU-MKDP) or a link to an existing document in some (perennial) place accessible via the Internet. To be able to integrate it into the knowledge base, the expert will have to include metadata to this document. These metadata are based on Standard Dublin Core (http://dublincore.org/). The Dublin Core Metadata Element Set has 15 so-called "mandatory" elements covering the most important properties to describe a document (title, creator, subject, coverage – temporal or geographical…). These metadata are then used in the search facilities to

6

retrieve the documents. The metadata application is operational (more than 70 documents with their metadata have already been entered at mid-July 2015). ►The Minerals4EU Metadata Catalogue (for structured data) is the central access point to metadata concerning European mineral resources and related topics. It has been developed by CGS based on similar principles as MIcKA metadata catalogue, serving to the OneGeology-Europe Metadata Catalogue. It is fully operational at CGS: http://m4eu.geology.cz/metadata/. It provides tools for compilation of those metadata in a standardized format that will allow users to effectively search through the database.

In order to make the data discoverable in the most efficient way, the catalogue is fully compliant with international standards and supports the distributed system of metadata administration. Only digital and structured information (spatial datasets or dataset series and spatial data services - WMS, WFS) related to the mineral resources and relevant topics should be described by metadata in this catalogue. In order to display a metadata record for which an on-line map service is available, the metadata will be used in the search facilities. The catalogue has enabled systematic discovery, viewing and use of data on mineral resources across Europe.

To distinguish the specific harmonised Minerals4EU datasets and services, a new term "Minerals4EU harmonised" has been added to the list of Specification in Data Quality section of the metadata record. This enables searching only the harmonised records.

All data providers have been encouraged to fill in the metadata both on all relevant data and services already available and on newly created harmonized data sets and services. Procedure on how to insert the specific information concerning the compliance to Minerals4EU standards was presented at the Bled meeting (Slovenia, March 2015) and detailed description is available in the updated cookbook.

91 public records had been inserted into the database until August 18th, 2015 (Table 1).

Organization abbreviation Country Metadata

record Harvested Datasets Services Harmonised

AGS Albania BGR Germany BGS United Kingdom 2 private rec.

BRGM France 3 private rec. CGS Czech Republic 31 Y 17 14 1 EGS Belgium

Fraunhofer Germany GBA Austria 1 Y 1

GEOINFORM Ukraine 2 2 GEOZS Slovenia 3 Y 3 GEUS Denmark 1 1 GSD Cyprus GSI Ireland 2 1 1 GTK Finland 1 1

7

HGI-CGS Croatia IGME_G (EKBAA) Greece 1 1

IGME_S Spain 11 6 5 1 IGR Romania 2 2

ISPRA Italy JRC Belgium

LNEG Portugal 1 1 MFGI Hungary 5 3 2 NGU Norway 5 2 3 1

PGI-NRI Poland 8 8 RBINS Belgium RMG Sweden

SELOR Netherlands SGU Sweden 10 4 6 1

SGUDS Slovensko 7 7 TNO Netherlands 1

VBS (DDPS) Switzerland Wuppertal Institute

for Climate… Germany

TOTAL 91 public rec. 3 53 39 4 Table 1: Number of metadata records inserted by participants

The code lists for the metadata based on INSPIRE Data Specification Mineral Resources code lists have been amended for the needs of the Minerals4EU project as published in Schubert et al., 2014. Therefore the Metadata Keywords application has been amended http://m4eu.geology.cz/codelist/#. Any keyword is coded as URI (semantic web) according to the latest INSPIRE Metadata Technical Guidelines (October 2013).

To support the metadata authorized editors, the Cookbook has been updated and is available at http://m4eu.geology.cz/metadata/?ak=cookbook. The guidelines on how to use the Metadata Catalogue Application and Metadata Keywords Code List Application are described in detail so metadata editors from any data provider can easily insert, maintain and search for the metadata. ► The search facilities are based on a search engine indexing the Central Diffusion Database, the documents corpus (non-structured data) and some external databases (if any). The user interface allows the end users to retrieve data from the whole EU-MKDP using a simple input (Google-like search, a simple sentence will search in the whole EU-MKDP), or using some specialized interfaces (search specific concepts using their main attributes). Almost all the indexed concepts will have a geographic and temporal extents (coming from INSPIRE MR/ERML for the diffusion database, coming from Dublin Core metadata for the documents). These extents will allow the user to retrieve the most accurate response for his search. The search facilities are operational.

8

4 The EU-MKDP Web Portal and dedicated services

The EU-MKDP Web portal (http://minerals4eu.brgm-rec.fr/) is in the process of being finalized for the main functionalities (July 2015). The ongoing work now focuses on the way to represent and use data (representation, (multiple) selection, getFeature info [information obtained when clicking on a site], download capabilities…).

BRGM, in collaboration with BGS, developed dedicated interfaces in order to dynamically consult the Minerals Yearbook created within Minerals4EU WP4. This new e-Minerals Yearbook (http://minerals4eu.brgm-rec.fr/m4eu-yearbook/theme_selection.html) allows the end user to navigate in the Yearbook and visualize it in an interactive way. The user is able to select a country of Europe and view the data related to its production, trades, resources, reserves for each commodities; he is also able to view, by commodities, these different data at an EU scale and to consult the information about the secondary resources sector.

5 Support to geological surveys for web services implementation Several countries are currently implementing the INSPIRE based Web Feature Services that they should provide within the framework of this project and if the data are newly aggregated before the end of 2015. In order to help setting up the PostgreSQL database, the use of the ETL GeoKettle, code lists, the server interface Deegree3 and the harvesting system, a dedicated session was organized in Copenhagen on June 2nd, 2014. From February 2015 onwards, GEUS, TNO, BRGM and GeoZS have, improved the setup, included some new features and helped other countries setting up their own National Provider Databases.

At the end of August 2015, the status for implementation was as follows:

Organization Implementation status BGS Done BGR Preliminary stages BRGM Done CGS Done EKBAA Ongoing, has WFS GBA Done

Geoinform Implementation expected in 2016, in collaboration with NGU

GeoZS Done GEUS Done

GSD Done (ProMine data for Cyprus [updated] served under BRGM supervision)

GSI Done GTK Done IGME Done

9

IGR Done ISPRA Preliminary stages LNEG Ongoing MFGI Done NGU Done PGI Ongoing SGU Done SGUDS Done TNO Done VBS-DDPS Done

Table 2: Status for the implementation of the Minerals4EU harvesting system

Those countries with functioning web feature services have been used to test the harvester software and most of them can be harvested from. Most countries will, as a first stage, only implement the mineral resources core part of the INSPIRE model for mineral resources and mines, if such information is available. Some countries will also create services for mineral resource extensions related to mining waste, mining products, rock materials and minerals and information from area management is also included for some of them.

At the time this final report related to D5.3 is written (end of August 2015) the following 17 URLs have been successfully harvested: AT, CH, CY, CZ, DK, ES, FI, FR, GB, HU, IE, NL, NO, RO, SE, SI, SK. Belgium, Croatia, Greece and Poland will publish their services in the forthcoming days/weeks.

DELIVERABLE D5.3

SEE THE VIDEO OF THE EU-MKDP AT:

http://minerals4eu.brgm-rec.fr/videos/00-EU-MKDP_Final2.mp4

10

6 References Cassard D., Bertrand G., Billa M., Serrano J.-J., Tourlière B., Angel, J.-M., Gaál G † (2015). ProMine mineral databases: new tools to assess primary and secondary mineral resources in Europe in: P. Weihed (ed.), 3D, 4D and Predictive Modelling of Major Mineral Belts in Europe, Mineral Resource Reviews. ProMine Special Volume, pp. 9-58. Springer Editor.

Cassard D., Serrano J.-J., Vuollo J. (2012). The use of the standard exchange EarthResourceML in the ProMine project. 34th International Geological Congress, Brisbane, August 2012, Abstracts on CD-ROM.

Cassard D., Tertre F., Bertrand G., Tellez-Arenas A., Schjøth F., Heijboer T., Vuollo J., Čápová D., Šinigoj J., Gruijters S., Bogaard P., Waardenburg F., Tomas R., Schubert Ch. (2015). Minerals4EU - INSPIRE in Action: Pan-European Raw Materials Knowledge Base Data Platform. Geospatial World Forum – INSPIRE Conference, Lisbon, Portugal, 25-29 May, 2015.

EarthResourceML (2013).

http://www.earthresourceml.org/earthresourceml/2.0/doc/ERML_HTML_Documentation/

EuroGeoSource (2013). EuroGeoSource homepage http://eurogeosource.eu/.

In the menu EuroGeoSource Deliverables:

1) Publications and presentations: http://eurogeosource.eu/deliverables/publications-and-presentations

2) Public Reports: http://eurogeosource.eu/deliverables/public-reports

GEUS (2015). The M4EU data model, database and tool stack cookbook for Minerals4EU-WP5 and EURare-WP1.Minerals4EU and EURare Projects internal documents, unpublished.

Gruijters S. and Fruijtier S. (2011). Using cloud computing to increase the usability of distributed Web GIS Systems. eChallenges e-2011 Conference Proceedings, 26-28 October 2011, Florence, Italy. Paul Cunningham & Miriam Cunningham (Eds.), IIMC International Information Management Corporation, 2011, ISBN: 978-1-905824-27-4, 8 p.

INSPIRE Directive (2007).

http://inspire.jrc.ec.europa.eu/

http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:2007:108:0001:0014:EN:PDF

INSPIRE Thematic Working Group Mineral Resources (2013). D2.8.III.21 Data Specification on Mineral Resources – Technical Guidelines. 156 p. European Commission Joint Research Center Publisher. http://inspire.ec.europa.eu/documents/Data_Specifications/INSPIRE_DataSpecification_MR_v3.0.pdf

11

Raymond O., Vuollo J., Richard S. and CGI Interoperability Working Group (2012). Global Geoscience Data Transfer Standards - GeoSciML and EarthResourceML. eResearch Australasia Conference, Brisbane, Australia. http://conference.eresearch.edu.au/eres2012/posters-2012/

Schubert Ch., Vuollo J., Tomas R., Cassard D. and WP5 Partners (2014). Minerals Intelligence Network for Europe – Minerals4EU-WP5. WP5: Common terminology for Minerals4EU – Version 1.0, Minerals4EU Report, 330 p.

Serrano J.-J., Kauniskangas E., Cassard D. (2010). ProMine: architecture, portal and web services to provide a European mineral resources information system, 4th INSPIRE Conference. Krakow, Poland, 22-25 June.

Vuollo J., Cassard D., Simons B., Seymon A. (2010). The Earth resource data exchange model (EarthResourceML) – a tool for delivering INSPIRE and ProMine mineral resource data, 4th INSPIRE Conference. Krakow, Poland, 22-25 June.

Vuollo J., Simons B., Laxton J., Cassard D., Seymon A. (2012). EarthResourceML v.2.0 – an upgrade of the CGI-IUGS Earth Resource Data Model due to INSPIRE Data Specification. 34th International Geological Congress, Brisbane, August 2012, Abstracts on CD-ROM.