Quarterly Implementation Report - Copernicus · No parts of this document may be photocopied, reproduced, stored in retrieval system, or transmitted, in any form or by any means whether

Service contract for the provision of assistance to the EEA in

setting up and coordinating the Copernicus in situ

component – In situ Database review.

Deliverable 6. Final evaluation report

11 March 2016

http://www.bing.com/images/search?q=European+Commission&view=detailv2&&id=BE07C147F3052749F773A60E0D2B0598F76F9163&selectedIndex=1&ccid=DPcMIPbN&simid=608003851937448111&thid=OIP.M0cf70c20f6cd74f1c0e6e1f7d0afd8eeH0

2 EU - EEA DELEGATION AGREEMENT

Cross-cutting activities for coordination of the in situ component of the Copernicus Programme 2014 - 2020



This report is prepared in response to the tasks described in the Technical Annex to the

Delegation Agreement with the EEA for the implementation of the Copernicus Land

monitoring service and the In situ component. The Delegation Agreement is funded by the

European Union in the context of the Copernicus programme and implemented by the

European Environment Agency.

© 01/01/2015 European Environment Agency

All Rights Reserved. No parts of this document may be photocopied, reproduced, stored in

retrieval system, or transmitted, in any form or by any means whether electronic,

mechanical, or otherwise without the prior written permission of the European Environment

Agency.

DISCLAIMER

The views expressed herein can in no way be taken to reflect the official opinion of the

European Union.



Report information

Prepared by Johnny te Roller (Alterra)

Project number

Project manager Johnny te Roller (Alterra)

Project supervisor

Prepared for EEA

Project manager Henrik Steen Andersen

Specific contract number EEA/IDM/15/019

Relation to

Type of document Final evaluation report

Description

Review and contribution Sander Mucher, Daniel van Kraalingen en Rob Lokers

(Alterra)

Creation date 28 January 2016

Status

Access

Version history:

Date Name Comment

31 January 2016 Johnny te Roller Initial version

3 February 2016 Johnny te Roller Updates according to internal

review comments and corrections.

14 February 2016 Johnny te Roller Updates according to

teleconference with Henrik Steen



Andersen (EEA)

16 February 2016 Johnny te Roller Updates according to remarks

from Henrik Steen Andersen (EEA)

11 March 2016 Johnny te Roller Updates according to remarks

from Copernicus cross-cutting in

situ data coordination Workshop

(24-25 February 2016)



Executive summary

This report is the final version to describe the results of the technical review of the

Copernicus in situ database. It includes recommendations for improvement and

remarks and comments from the Copernicus cross-cutting in situ data coordination

Workshop.



Table of Contents

1. INTRODUCTION .................................................................................................................................................. 9

1.1 BACKGROUND ....................................................................................................................................................... 9

1.2 PROBLEM DEFINITION.............................................................................................................................................. 9

1.3 MAIN OBJECTIVE OF THE PROJECT ............................................................................................................................ 10

1.4 REPORT OUTLINE .................................................................................................................................................. 10

1.5 OUTCOME OF THE WORKSHOP ................................................................................................................................ 10

2. DATABASE TECHNICAL EVALUATION .................................................................................................................12

2.1 INTRODUCTION .................................................................................................................................................... 12

Design principles ......................................................................................................................................... 12 2.1.1

Implementation principles .......................................................................................................................... 13 2.1.2

2.2 WHAT IS IN SITU DATA? ......................................................................................................................................... 13

2.3 TABLE NAMES ...................................................................................................................................................... 13

Singular or Plural ......................................................................................................................................... 13 2.3.1

Many-to-many relations ............................................................................................................................. 14 2.3.2

2.4 FIELD NAMES ....................................................................................................................................................... 14

ID fields ....................................................................................................................................................... 14 2.4.1

Timestamps ................................................................................................................................................. 15 2.4.2

Miscellaneous ............................................................................................................................................. 15 2.4.3

2.5 USE OF FOREIGN KEY RELATIONS .............................................................................................................................. 16

Missing foreign key relations ...................................................................................................................... 16 2.5.1

Enforce referential integrities ..................................................................................................................... 16 2.5.2

2.6 INDEX DEFINITIONS ............................................................................................................................................... 17

2.7 USE OF FIELD DATA TYPES ....................................................................................................................................... 18

2.8 USE OF PICK LIST TABLES ........................................................................................................................................ 18

2.9 NORMALISATION .................................................................................................................................................. 19

2.10 USE OF UNITS ...................................................................................................................................................... 20

2.11 COPERNICUS SERVICES STRUCTURE .......................................................................................................................... 20

2.12 GAP ANALYSES ..................................................................................................................................................... 21

2.13 MISCELLANEOUS .................................................................................................................................................. 23

3. TEST CURRENT AND FUTURE DATASETS (FUNCTIONAL EVALUATION) ...............................................................24

3.1 LAND MONITORING .............................................................................................................................................. 25

3.2 EMERGENCY MANAGEMENT ................................................................................................................................... 26

Critical infrastructures - utilities .................................................................................................................. 26 3.2.1

Large scale population information ............................................................................................................ 27 3.2.2

3.3 ATMOSPHERE MONITORING ................................................................................................................................... 29

Aerosol optical depth .................................................................................................................................. 30 3.3.1

Surface near-real-time measurements – CO ............................................................................................... 31 3.3.2

3.4 MARITIME ENVIRONMENT MONITORING .................................................................................................................. 33



3.5 CLIMATE CHANGE ................................................................................................................................................ 34

Future requirements ................................................................................................................................... 34 3.5.1

3.6 SECURITY ............................................................................................................................................................ 35

ANNEX 1. PROPOSAL FOR RENAMING OF TABLE AND FIELD NAMES .........................................................................36



1. Introduction

1.1 Background

In the GISC project (GMES in situ coordination) the current Copernicus in situ

database has been developed. This project ended in 2013. The Copernicus In-Situ

Coordination (GISC) project aimed at linking in-situ data providers and Copernicus

service providers to ensure access to in-situ data for Copernicus services.

Afterwards a Delegation Agreement (DA) between the European Environment Agency

(EEA) and the European Commission (EC) was signed: Copernicus land monitoring

service and the in situ component 2014 – 2020.

In the Delegation Agreement four primary cross-cutting in situ coordination tasks are

described. (REGULATION (EU) No 377/2014 OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL of 3 April 2014)

1. Establishing and maintaining an overview of the current state of play of in situ

data for Copernicus services.

2. Operational provision of in situ data including access to reference data for

Copernicus services.

3. Managing partnerships with data providers to improve access and use

conditions of in situ data for Copernicus services.

4. Support to the EC and Copernicus services on in situ data issues.

The first task, Establishing and maintaining an overview of the current state of play of

in situ data for Copernicus services, is based on 3 elements:

a) Overview of the in situ data requirements for Copernicus service production;

b) A database comprising an overview of already available and used in situ data

including their accessibility and web-service performance as indicated by the

Copernicus service operators;

c) Crossing demand and offer for in situ, in order to identify gaps, priorities and

the potential for improvement of in situ data access.

1.2 Problem definition

One of the main products defined in the delegation agreement is the in situ

requirements database. This should provide up-to-date information across the

Copernicus services on in situ data requirements (current and expected), data used,

gaps, data providers, access arrangements, and partnerships.

The current Copernicus in situ database has been developed at the GISC project,

which ended in 2013. At this moment there is a need to have a professional review on

the database to ensure future operational use, and is the major objective of this

report.



1.3 Main objective of the project

The main objective of this service contract is to evaluate whether the current

Copernicus in situ database, developed by the GISC project, is sufficiently capable of

a) providing a suitable overview of the Copernicus in situ component, b) capturing

information (metadata) about all Copernicus services’ in situ data requirements and

relevant data sources in an appropriate and efficient manner, and c) providing the

necessary basis for in situ data gap analyses. Moreover, based on the evaluation

results, to provide recommendations for an effective working process with efficient

work sharing, and for improvements of the current database structure to support this

work process.

1.4 Report outline

This report describes the results of the evaluation of the current Copernicus in situ

database.

The next chapters are dedicated to the technical and functional review of the current

version of the in situ database. This includes recommendations for improvement.

The evaluation is also being carried out for all six Copernicus services based on a

limited but typical set of requirements and data sources. This results in one chapter

for each service.

This report also reflects the outcome of the Copernicus cross-cutting in situ data

coordination workshop on 24-25 February 2016 at the European Environment Agency

in Copenhagen. Next paragraph describes a summary of the major conclusions related

to the implementation of the database.

1.5 Outcome of the workshop

Representatives of all services were present at the workshop to discuss the cross-

cutting activities of the EEA. The first task of the cross-cutting activities is to establish

and maintaine an overview of the current state of play of in situ data for Copernicus

services. A database comprising an overview of already available and used in situ data

is considered as one of the elements to do this task.

Starting point for the discussion was ‘What would we like to achieve from creating and

maintaining this data base’. Why do we need this database? From the services’ point



of view it was not clear what was their incentive to deliver the data to keep the

database up to date or to have this database at all. It was considered a tool especially

useful for the EEA. It would cost them a lot of time and energy to deliver all requested

information to update the database. Therefore the data in the database should not be

too detailed, a global overview should be sufficient for the major purposes.

For the EEA the major purpose of the database is to have a up to date overview of all

services, not only of the requirements and used datasets, but especially of the gaps

and overlaps. This to be able to assist the services to fill the data gaps.

Based on the comments made during the workshop the EEA reconsidered the added

value of setting up an in situ database at this point in time. It will be too resource

demanding to specify, implement, and populate a database usable across all services.

Whereas the services clearly indicated the priority on other issues and asked the EEA

to focus on gaps and solutions, and start awareness raising activities.

The EEA will create, instead of the database, the overview through a set of fact

sheets, report and inventories per service. (see figure 1.) The need of a database

might be reconsidered at a later stage. But then the proposed set of factsheets, report

and inventories will be a better basis the build on.

Figure 1. Overview of Copernicus in situ component



2. Database technical evaluation

2.1 Introduction

The purpose of the Copernicus in-situ data requirements database (further in the text

called ‘database’) is to enable efficient management of information on in-situ data

requirements. The reviewed database (MS Access 2007) contains data of 4 Copernicus

services (Land Monitoring, Emergency Management, Atmosphere Monitoring and

Maritime Environment Monitoring). The in situ requirements of the Copernicus

Services Climate Change and Security will be added in the future as well.

This chapter describes the results of the technical review of the current in situ

database. As the database doesn’t contain any version info it is only possible to

indicate the version of the database by its last modification date, 12 July 2013.

This leads to our first recommendation. Introduce a new system table (e.g.

SysDB_info) with meta data of the database. The table should contain at least the

version number and the (last) modification date with respect to structural changes.

For easy management of the content of the database forms are built upon the

database in MS Access. These forms are not part of this review.

For a relational database guidelines and rules of thumb are well established. An

entity-relationship model (ER model) is a method to describe the data or information

aspects of a business domain or its process requirements. It is an abstract model that

lends itself to ultimately being implemented in a (relational) database. The main

components of ER models are entities (things) and the relationships that can exist

among them. We assessed the database and the ER-model for several aspects. The

result is described in the next paragraphs.

Design principles 2.1.1

At the GISC project the following objectives have governed database design

principles:

deliver a comprehensive harmonised documentation and specification of the in-

situ data requirements reflecting the current status of Copernicus service

deliveries;

establish and maintain a consistent overview of in-situ data requirements,

taking into account synergies, gaps, overlaps, constraints on priorities and

other issues such as restriction to access or use, intellectual property rights

(IPR), infrastructure and architecture ensuring sustainable data provision;

prioritise in-situ data needs in light of their urgency and contribution to

Copernicus services;

provide the basis for cost estimations;



identify data provider organisations or networks with whom a dialogue is

needed.

Implementation principles 2.1.2

In the GISC project, the following principles have been considered as basis for

choosing the database implementation approach (database model and operating

platform):

Copernicus in-situ data requirements must be traceable to the individual

Copernicus services and their products;

the database must allow to perform an analysis of Copernicus in-situ data

requirements, e.g. in terms of criticality, multiple use, coverage etc.;

the database must allow to classify Copernicus in-situ data requirements in

terms of required quality, coverage, timeliness, accessibility and intellectual

property rights;

the database must enable the identification of gaps, overlaps, critical

constraints and issues (such as intellectual property rights (IPR) obstacles and

sustainability);

the database must support the dialogue with Copernicus services, data

providers and stakeholders;

the database should be flexible and easy to manage both for technical

personnel and end users;

the database must provide the necessary reporting tools and possibilities for

performing queries;

the database should be accessible from within EEA, but external (via internet)

multiuser access should not be excluded.

2.2 What is in situ data?

The database is called Copernicus in-situ requirements database. It should be clear

what is meant with in-situ data is this context and what kind of data requirements

should be in the database. Outcome of the workshop is next definition:

‘in situ data’ means observation data from ground-, sea-or air-borne sensors as well

as reference and ancillary data licensed or provided for use in Copernicus.

2.3 Table names

Singular or Plural 2.3.1

Table names are considered appropriate. There is not a general consensus on the use

of singular or plural for table names. However it is recommended to make a choice



and apply this consequently. It seems that in this database all pick-list tables are

singular and the rest plural. This seems a clear strategy.

Many-to-many relations 2.3.2

Implementation of many-to-many relations is done in coupling tables. Both names of

the tables of the relations should be part of the name of the coupling table. The

coupling table of the tables insitu_requirements and products should be

insitu_requirements_products or insitu_requirements_x_products to clearly indicate

its (technical) purpose.

Note. In some cases the coupling table represent a recognizable object. In those

cases it is preferred to use the name of the object. E.g. Consider the case of an online

shop, where orders are actually a coupling table implementing the many-to-many

relation between customers and products.

2.4 Field names

ID fields 2.4.1

It is common practice to work with a meaningless primary key. Every table has an ID

field defined as the primary key, which is correct. For the name of this field there are

two options:

i. ID. For every table the same name. This means that the fieldname is not

unique and should always be referred to in combination with or related to the

table name. Other table with a foreign key relation to this ID field should name

this foreign key field combined with the table name: <table name>_ID.

ii. <table name>_ID. (use singular of table name) This option is preferred as

the name is unique and can have the same value as the foreign key fields in

related tables. This means many id-fields (and foreign key fields) have to be

renamed. (see Annex 1)

The used data type of the id fields is Long Integer. In many cases the number of

records doesn’t exceed 10 (especially pick-list tables) which means that a ’cheaper’

data type like integer or byte could also be used. But as the database will not contain

very large datasets, memory use and performance will not be an issues and the use of

Long Integer is justifiable as being the safest option.

In the database for every primary ID field the AutoNumber datatype is used. This is

not recommended since it is a Microsoft Access specific field type that needs to be

converted to some other field type when porting the database to a different database

platform. Several options then exist for creating a unique id-value for a new record:

i. Use of sequences and insert-triggers.



ii. Generate ID numbers from a specific sys-table managing ID numbers.

iii. Use of UID’s

Timestamps 2.4.2

Some inconsistencies are introduced by adding fields named datemodified and

datamodified. By naming a field datamodified it suggests that it is a Boolean field

which indicated whether a record is changed (or not). But it is a date/time field so

most likely datemodified is intended.

An effort has been done to let every record in most tables have some sort of

timestamp through the addition of timemodified fields. This field is redundant as the

datemodified field is of type date/time and contains already a timestamp.

For a number of entries in some tables the value of datemodified and/or timemodified

is empty. It is recommended to make the field datemodified mandatory (timemodified

can be removed from database). Most database systems have the option to add a

trigger to a table which updates the datemodified field each time the record is

changed (or inserted).

Miscellaneous 2.4.3

The table available_datasets has a field named target_accuracy. It is not clear why

the part ’target’ is added to the fieldname. Isn’t just accuracy a better name?

The table insitu_requirements has a field Name. It contains in most cases a long

description of the requirement. Perhaps add extra fields, acronym and description,

similar to other tables. Consider if both fields description and characteristics are

necessary.

Name and Type are often reserved words in RDBMS systems and should not be used

as a field name.

Name is a field in the tables consortiums, service_components, themes,

core_services, products, insitu_requirements, available_datasets and data_providers.

The fieldname of the field Name can be replaced by putting the singular form of the

table name in front of it.

Type is used in many picklist tables and can be replaced by caption (see also

paragraph 2.6).



2.5 Use of foreign key relations

Missing foreign key relations 2.5.1

There is (probably) a missing relation between table data_type (field data_type_id)

and table insitu_requirements (field type_id). Enforcing this relation with referential

integrity and full cascading did not give an error, so we assume this was a missing

relation.

Enforce referential integrities 2.5.2

The relations between many tables are not setup to enforce referential integrity. This

is considered bad design and can easily cause ‘ghost’ relations. Also, cascading delete

and update relations were not setup, leading to data corruption in case where the

master of a relation is deleted or where the master field is changed. All 30 relations

were verified and enforcement of referential integrity was done causing 18 relations to

be investigated and repaired. This could only successfully be done by changing some

foreign key data. Below is the list of manual changes necessary on each of the 18

relations (per table alphabetically).

Many existing relations where referential integrity was switched on, did not have

“Cascade update related fields” switched on. The can also lead to referential integrity

problems and it was manually fixed wherever this situation occurred.

Relation between coverage and available_datasets (field coverage_id):

- 8 records in which the illegal value zero was used were set to 7 (TBD).

- 10 records in which the field was empty is set to 7 (TBD).

Relation between coverage and insitu_requirements (coverage_id):

- No integrity relation was setup

Relation between criticality and requirement_cases (field data_use_id):


Relation between data_policy and available_datasets (field policy_id):



Relation between data_providers and available_datasets (field data_sust_id): - No integrity relation was setup

Relation between data_security and available_datasets (field security_id):



Relation between data_sustainability and available_datasets (field data_sust_id):


- 2 record in which the field was empty is set to 5 (TBD).

Relation between data_type and insitu_requirements (field data_type_id):


Relation between data_use and products (field data_use_id):





Relation between data_use and requirement_cases (field data_use_id):


Relation between dissemination and available_datasets (field dissemination_id):



Relation between dissemination and insitu_requirements (field frequency_id):



Relation between eionet_data_flows and eionet_data_flows_links (field eionet_id):


Relation between frequency and available_datasets (field frequency_id):



Relation between frequency and insitu_requirements (field frequency_id)


Relation between status and products (field status_id)



Relation between inspire_annexes and inspire_links (field inspire_id)


Relation between timeliness and available_datasets (field timeliness_id):



Relation between timeliness and insitu_requirements (field insitu_id):



2.6 Index definitions

Next requirements checked (and considered OK)

- Every table has a primary key.

- Tables realizing many to many relations (requirement_cases, provider_cases)

have a unique multi field index.

- All ID fields that are not a primary key (= foreign key) must have a non-unique

index.

The table available_datasets has a primary key on the field dataset_id as well as a

non-unique index. The latter is considered unnecessary. Similarly table consortiums

(field cons_id), table core_services (field cs_id), table coverage (field coverage_id),

table dataproviders (field provider_id), table data_type (field data_type_id), table

data_use (field data_use_id), table insitu_requirements (field insitu_id), table

inspire_annexes (field inspire_id), table inspire_links (field inspire_link_id), table

products (field prod_id), table service_components (field sc_id) and table status (field

status_id).



2.7 Use of field data types

The database contains many different field lengths for text fields. A maximum length

is only required in cases where there is a true maximum length. In other cases, a

maximum length is much harder to determine and is set according to some gut feeling

(so is arbitrary). To hit such a maximum length while entering data can be annoying

to the user. Our advice would be to set all such fields to 255 chars. Again a maximum,

but beyond 255, user interface components can cause troubles. Most database

platforms nowadays are capable to store only the true data, so extending the

maximum length does not necessarily increase the database size.

2.8 Use of pick list tables

Many of the tables in the database are used for pick lists (e.g. status, criticality).

Except for table specific items they all have an entry for ‘N/A’ (not applicable) and

‘TBD’ (to be determined). Unfortunately the id’s corresponding with these items differ

in value among the tables (for table status it is 4 and 5, for table criticality it is 5 and

6). Furthermore the default status on insert of a new record in table products is 0,

which is in an undefined entry in the status table. It is a better approach if ‘N/A’ would

have corresponded with ID value = -1 in ALL pick list tables, and if ‘TBD’ would have

corresponded with ID value = 0 in ALL pick list tables. In doing so, pick list items are

automatically set to ‘TBD’ whenever a new entry is made in a table. Another option is

to set no default so the user is forced to make a choice.

These pick list tables can be seen as a special kind of tables. Therefor it is

recommended to structure these tables similar besides having the same id-values for

the common entries (N/A and TBD) as mentioned above. All these tables have the ID-

field and sort_order field in common. Besides this the tables have a text-field and/or a

description-field or even a third variant of the field. Also the data type of these fields

differs. The next table gives an overview of all used fields and data types for these

tables.

fields description type <other>

picklist tables

status text

data_use text

criticality memo text

dissemination text

coverage text



frequency text

timeliness text

data_type text

data_sustainability text

data_policy text

data_security memo text

Table 1. Pick list table fields

We suggest giving all tables a caption field (not type because it is a SQL reserved

word) and a description field (of data type text), where caption should be a short (one

or two words) caption of the entry. The field description may contain a longer

description of the entry.

Each entry in the table is considered an allowed option of a field of a related table.

The user selects from a list of all options. In the list the caption will be shown which

means the caption should unambiguously describe the option. To enforce this the field

caption should have a unique index.

2.9 Normalisation

Normalisation in the database is performed on every possible table except for the

costs in the insitu_requirements table. It contains 4 pairs of cost related fields for

setup, operational, data access and coordination. This should be stored in a zero-to-

many detail table costs. This means that every insitu_requirement can have zero or

more costs.

The costs table should have at least the following fields:

Field name description

cost_id Primary key

insitu_requirement_id Foreign key to related insitu_requirement

cost_type Type of costs (allowed values Setup, Operational, Data

access, Coordination or Other)

currency Currency used to describe amount

amount The actual costs

reoccurrence How many times should it be paid (allowed values Once,

Monthly, Quarterly, Biannual, Yearly, Biennial, Triennial,...)

notes Description of the costs

For the allowed values of the fields cost_type and reoccurrence a pick-list table should

be introduced to define the allowed values. This will make the values of the field a

foreing key field and the names should be cost_type_id and reoccurrence_id.



The normalisation might require extra checks in the User Interface. E.g. a

insitu_requirement can have only one detail cost record of each cost_type.

To be able to summarize costs (per requirement and/or product) it is necessarry that

all costs are defined in the same currency or extra conversion information is added to

the table.

2.10 Use of units

In many tables values are entered together with the unit in which they are expressed

in one field. E.g. the field target_accuracy contains values like ‘0.2 m’. This makes it

very difficult (or even impossible) to query on these fields. Also it is possible to ‘forget’

to enter the used unit which makes it impossible to interpret the entered value in a

correct way.

It is more solid to introduce an extra table Units with the fields symbol and

description. Every relevant field in the database where a value and a unit are

expected should be split in two fields: amount and unit_id, where the field amount

contains the numeric value and the field unit_id a link to a unit in the units table.

The insitu_requirements table has the fields setup_costs, oper_costs, data_acc_costs

and coord_costs are of type Currency. It is shown as £ which probably depends on the

setup of MS Access of the user. However a description in the technical document says:

costs in Eur. This problem will also be solved with the use of a Units table. The costs

field will be split in two fields: amount and currency. Currency will be a link to a value

in the Units table. (To be consequent this field should be named unit_id.)

2.11 Copernicus Services structure

The database should reflect the structure of the Copernicus Services part of the

Copernicus Programme. In the current structure the required tables are included but

not related in the correct way. Also the table consortiums should be renamed to

entrusted_entities. Figure 2 shows the proposed relations. Each service has one or

more service_components. However some services don’t have a service component.

There are some options to solve this problem in the database (i) add a dummy service

component to fill the gap, (ii) link the entrusted_entity direct to the service. The latter

requires extra checks to prevent the unwanted situation that a services with a service

component is also direct linked to an entrusted entity.

Each service component is always linked to one entrusted entity.



Figure 2. Copernicus in situ component structure

The table consortiums may contain extra information like ModeOfImplementation with

possible options direct or indirect management.

Table service_components: Order of fields acronym and name differs from other

similar tables. (Figure 2 shows the proposed order)

To be discussed the need of the table themes between products and

service_components. If it is required to have the option to categorise products in

themes then a separate table only linked to product is a more obvious option. This

makes the categorization optional. In the current situation it is mandatory.

2.12 Gap analyses

One of the main objectives of this service contract is to evaluate whether the current

Copernicus in situ database is sufficiently capable of providing the necessary basis for

in situ data gap analyses.

Our first approximation is that the gap can be defined as a mismatch between the

preferred data sets and the available and/or used datasets. This mismatch can be on

several levels:

i. It is unknown if the preferred dataset is available or even if it exists.

ii. The preferred dataset is known but not available. E.g. the potential data

provider does not want to deliver the data or there is no contract with the data

provider (yet).

iii. The dataset is available but does not cover the whole required spatial area or

time line.

iv. The dataset is available but the delivery of the data is not in time.

v. The dataset is available but the costs are too high.

vi. ...

In the table insitu_requirements the services define the required characteristics of the

data need. In the table available_datasets the used data sets are stored. This may be



a dataset that doesn’t cover all the required characteristics. This gap can be defined in

a new table insitu_requirement_gaps. (Figure 3)

Figure 3. define the gap.

Each insitu_requirement can have zero or more insitu_requirement_gaps. The

insitu_requirement_gaps are categorised by a gaptype. The new picklist table is a

predefined list with options which relate to the characteristics of the

insitu_requirement, e.g. accuracy, dissemination, coverage, timeliness, frequency,

type, costs. For gaps that are not related to any of these options a ‘general’ option

should be available as well.

The field description of the insitu_requirement_gaps table is a free text field to specify

the gap. E.g. a gap of type coverage can have description ‘data of Germany and

Poland is missing’.

The field description may also contain information on a possible sollution or an

indication to a possible direction of the sollution. E.g. if the service already knows a

existing dataset but not yet available. This information can also be stored in an extra

field.

Not every gap may be critical. Through the field criticality_id a gap can be ranked. The

use of the existing table criticality is not necessary. Another way of ordering the

criticality is also possible.

Optional is a status field to store the status of a gap. This enables archieving of all

gaps. Filled gaps can be marked as ’Filled’ or ’Solved’ in the status field. Otherwise the

filled gaps have to be deleted to maintain an up to date database, which includes that

’old’ gaps can not be traced back.



2.13 Miscellaneous

- Extra field should be added to the table available_datasets. Additional information

is required on the reliability (e.g. irregular delivery, will stop in 1 year, ..) and the

context of the dataset (e.g. research project, network,..).

- Services maintain their own systems to administer their products and data

requirements. It may be preferable to have a link to these systems stored in the in

situ requirements database. Therefore an extra optional field ‘external_reference’

can be added to the table products and/or insitu_requirements.

- The table insitu_requirements has fields target_accuracy and thresh_accuracy.

Some records contain a range (e.g. 5-10 %) where a value is expected.

- It is not clear what is meant with the table data_type. It contains records related

to the type of data (observation, forecast) but also related to the format/structure

of the data (xml, asci, map, raster data, ...). Should this be split in 2 separate

tables?

- The table data_use is linked to 2 tables, products and requirement_cases. This

means a product can have two different data_uses: one direct and one related to

an insitu_requirement. Is this correct or should there only be a relation from

data_use to products?



3. Test current and future datasets (functional evaluation)

In the previous chapter we proposed adaptations on the current structure of the in

situ database. This is done from a technical point of view. This leaves us with the

question if the current data in the database will still fit in the database after the

proposed changes.

Another question to answer is if this database structure (or the proposed) is the most

optimal structure to store the data of all services? The services cover all a different

domain which means the in-situ data requirements relate to another domain. So is it

possible to describe the data for all domains with the same set of attributes or do we

need to make a specialization for each service?

In the next chapters a pair of representative data requirements per service is selected

to test if the proposed database structure is still capable of storing the data. For the

services Climate Change and Security there is no data available in the current

database so we will create new test data.



3.1 Land Monitoring

The Copernicus land monitoring service provides geographical information on land

cover and on variables related, for instance, to the vegetation state or the water

cycle. It supports applications in a variety of domains such as spatial planning, forest

management, water management, agriculture and food security, etc.

The service became operational in 2012. It consists of three main components:

A global component;

A Pan-European component;

A local component.

The global component is coordinated by the European Commission DG Joint Research

Centre (JRC). It produces data across a wide range of biophysical variables at a global

scale (i.e. worldwide), which describe the state of vegetation (e.g. leaf area index,

fraction of green vegetation cover, vegetation condition index), the energy budget

(e.g. albedo, land surface temperature, top of canopy reflectance) and the water cycle

(e.g. soil water index, water bodies).

The Pan-European component is coordinated by the European Environment Agency

and will produce 5 high resolution data sets describing the main land cover types:

artificial surfaces (e.g. roads and paved areas), forest areas, agricultural areas

(grasslands), wetlands, and small water bodies. The pan-European component is also

updating the Corine Land Cover dataset to the reference year 2012.

The local component is coordinated by the European Environment Agency and aims to

provide specific and more detailed information that is complementary to the

information obtained through the Pan-European component. It focuses on "hotspots"

which are prone to specific environmental challenges. The local component provides

detailed land cover and land used information (over major European cities, which are

the first type of "hotspots". This is the so-called Urban Atlas. Besides an update of

the Urban Atlas, the next local component will address biodiversity in areas around

rivers (riparian areas).



3.2 Emergency Management

The Copernicus emergency management service (Copernicus EMS) provides all actors

involved in the management of natural disasters, man-made emergency situations,

and humanitarian crises with timely and accurate geo-spatial information derived from

satellite remote sensing and completed by available in situ or open data sources.

The mapping component of the service (Copernicus EMS - Mapping) has a worldwide

coverage and provides the above-mentioned actors (mainly Civil Protection Authorities

and Humanitarian Aid Agencies) with maps based on satellite imagery. The service is

implemented by the European Commission DG Joint Research Centre (JRC).

Another component of this service is the Early Warning System, also implemented by

European Commission DG Joint Research Centre (JRC).

From the database next two requirements were selected to test them against the

current and the proposed database structure.

Critical infrastructures - utilities 3.2.1

In-situ data requirement:

Critical infrastructures - utilities - scale 1: 10 000, global coverage Characteristics: Power plants, transmission networks, water treatment plants, pipelines, bridges, hazardous installations

Target accuracy: >1: 10 000

Threshold accuracy: 1: 50 000

Dissemination: Coverage: Timeliness: Frequency: View and Download Global Latest version available Irregular

Note: Latest version available, preferably not older than 3 years

OGC vector format

National and/or UTM projections

Relevant Copernicus product in-situ data is required for: Criticality of in-situ data: Delineation maps - overview Product generation Essential

Delineation maps - detail Product generation Essential

Reference maps - overview Product generation Essential

Reference maps - detail Product generation Essential

Grading maps - overview Product generation Essential

Grading maps - detail Product generation Essential



Pre-disaster situation maps - overview Product generation Essential

Pre-disaster situation maps - detail Product generation Essential

Post-disaster situation maps - overview Product generation Essential

Post-disaster situation maps - detail Product generation Essential

Links to Inspire annex(s) and theme(s):



Annex III: Utility and governmental services

Potential in-situ data providers Dataset/Product name National authorities in Member states National data

UN Geographic Information Working Group UN Spatial Data Infrastructure-Transport-V1.2

Remarks For this requirement the accuracy is a scale as the required data type is a map. For

the Land Monitoring Service the accuracy was related to measurements. In that

case the accuracy has a different meaning. This makes the accuracy field hybrid

and not suitable for queries or filters.

Timeliness is ‘latest version available’, the note says ‘not older than 3 years’.

Better define this with one value in the timeliness field: ‘< 3 years’.

Large scale population information 3.2.2


Large scale population information - accuracy 0,01 km2 (100 m x 100 m), for disaster affected areas only Characteristics: Number of people, forecasted population numbers, gender, literacy, resilience level, poverty levels

Target accuracy: 0,01 km2 (100 m x 100 m)

Threshold accuracy: 0,0625 km2 (250 m x 250 m)

Dissemination: Coverage: Timeliness: Frequency: N/A Disaster affected areas Latest version available Irregular

Note: At admin level 3 or higher/urban level, based on administrative units, statistics and/or maps

Relevant Copernicus product in-situ data is required for: Criticality of in-situ data: Delineation maps - detail Product generation Essential

Delineation maps - overview Product generation Essential



Grading maps - overview Product generation Essential

Grading maps - detail Product generation Essential



Pre-disaster situation maps - overview Product generation Essential

Pre-disaster situation maps - detail Product generation Essential

Post-disaster situation maps - overview Product generation Essential

Post-disaster situation maps - detail Product generation Essential

Links to Inspire annex(s) and theme(s): Annex III: Population distribution and demography

Potential in-situ data providers Dataset/Product name

National authorities in Member states National data



Remarks

For this requirement the accuracy is a resolution as the required data type is a

map. As mentioned before his makes the accuracy field hybrid and not suitable for

queries or filters.

Coverage is disaster affected areas. In case this area is very small, does this affect

the target_accuracy. Is more accurate data necessary?



3.3 Atmosphere Monitoring

The Copernicus Atmosphere Monitoring Service (CAMS) provides continuous data and

information on atmospheric composition. The service describes the current situation,

forecasts the situation a few days ahead, and analyses consistently retrospective data

records for recent years.

The Copernicus atmosphere monitoring service supports many applications in a

variety of domains including health, environmental monitoring, renewables energies,

meteorology, and climatology.

It provides daily information on the global atmospheric composition by monitoring and

forecasting constituents such as greenhouse gases (carbon dioxide and methane),

reactive gases (e.g. carbon monoxide, oxidised nitrogen compounds, sulphur

dioxide), ozone and aerosols.

The service also provides near-real-time analysis and 4-day forecasts, as well as

reanalysis, of the European air quality, thus enabling a permanent assessment of the

air we breathe.

The monitoring and reanalysis of greenhouse gases and aerosols contribute to climate

change studies by describing climate forcing.

Thanks to daily analysis and forecasts of UV and stratospheric ozone, the service

supports public health policies (e.g. skin cancer prevention).

Solar radiation is playing a key role in domains like health, agriculture and renewable

energies. The Copernicus atmosphere monitoring service provides public and private

organisations involved in solar energy usage with suitable and accurate information on

the solar radiation resources at the Earth's surface.

In addition to the above-mentioned services, the Copernicus atmosphere monitoring

service compiles emission inventories which serve as input to the atmospheric

chemistry-transport models and estimates net fluxes of CO2 and CH4 at the Earth's

surface. Knowing emissions and surface fluxes are prerequisite for understanding the

composition of the atmosphere.

ECMWF (European Centre for Medium-Range Weather Forecast) is the entrusted entity

for the implementation of the service.



From the database next two requirements were selected to test them against the

current and the proposed database structure

Aerosol optical depth 3.3.1


Aerosol optical depth - timeliness 1 day, accuracy 0.005, global coverage Characteristics: not specified

Target accuracy: 0.005

Threshold accuracy: 0.01

Dissemination: Coverage: Timeliness: Frequency: View and Download Global 1 day 15 min

Note: Currently aerosol optical depth is the only required parameter but other aerosol parameters, including vertical profiles,

are being used to improve understanding of atmospheric processes and to develop new products.

Target timeliness requirement is 3 h.

Relevant Copernicus product in-situ data is required for: Criticality of in-situ data: Reanalysis of greenhouse gases and fluxes Product generation Essential

Monitoring of surface solar irradiance Product generation Essential

Total ozone record Product generation Essential

Observed aerosol optical depth and aerosol Product generation Essential

speciation record

NO2 measurements made by OMI Product generation Essential

Global fire emissions Product generation Essential

Monitoring of greenhouse gases and fluxes Product generation Essential

Monitoring of global aerosols Product generation Essential

Support for scientific observational campaigns Product generation Essential

Reanalysis of global atmospheric composition Product generation Essential

Near-real-time ozone monitoring Product generation Essential

Near-real-time ozone forecasts Product generation Essential

Monitoring and forecasting of total ozone Product generation Essential

Monitoring and forecasting of global atmospheric Product generation Essential

composition

Links to Inspire annex(s) and theme(s): Annex III: Atmospheric conditions

Annex III: Environmental monitoring facilities

Potential in-situ data providers Dataset/Product name AErosol RObotic NETwork AERONET - aerosol optical depth

European Aerosol Research Lidar Network EARLINET - aerosol optical profile

European Supersites for Atmospheric Aerosol Research EUSAAR - aerosol properties

WMO Global Atmosphere Watch GAW-WDCA data


SKYNET SKYNET - aerosol properties



Remarks As the proposed structure doesn’t change much from the current, this requirement

will fit in the proposed structure. Normalisation of the costs doesn’t affect this

requirement as it has no costs defined.

However optical depth is dimensionless, it is not clear that the value for Target

accuracy and Threshold accuracy requires no unit or that it was forgotten or

unknown. In case an extra mandatory field ‘unit’ is added it should have the value

‘-‘ which ensures the dimensionless value of it.

The field name of the requirement contains information of several other fields of

the requirement: timeliness, accuracy and coverage. Why?

The value of the field timeliness is 1 day. The field note contains ‘Target timeliness

requirement is 3 h.’. What is the meaning of the field timeliness, is this the

used/available timeliness? Why does the table has a field target accuracy and not

a field target timeliness?

It is not clear what the status is of the dataset and the data providers. The current

name of the table is available_datasets, the proposed name is used_datasets. In

case the datasets are used then why the call the data providers ‘potential’ in the

report? In the database the table is called data_providers.

Surface near-real-time measurements – CO 3.3.2


Surface near-real-time measurements - CO - timeliness 6 months, EU27, EEA39, global coverage Characteristics: not specified

Target accuracy: TBD

Threshold accuracy: TBD

Dissemination: Coverage: Timeliness: Frequency: NRT Service Global 6 months <1 hr

Note: Seven regional air-quality analysis and forecasting systems are operated routinely for MACC and require surface air

quality measurements for assimilation and validation.

Surface air quality measurements delivered in near real time are un-validated.

Target timeliness requirement is 1 month.

In the future global coverage may be needed.

Relevant Copernicus product in-situ data is required for: Criticality of in-situ data: Reanalysis of European air quality Calibration and validation Essential

European air quality monitoring and forecasting Calibration and validation Essential

Links to Inspire annex(s) and theme(s): Annex III: Atmospheric conditions

Annex III: Environmental monitoring facilities



Potential in-situ data providers Dataset/Product name United States Environmental Protection Agency AirNow

WMO Global Atmosphere Watch Data from WMO GAW

Co-operative Programme for Monitoring and Evaluation of the EMEP - CO

Long-range Transmission of Air Pollutants in Europe

Integrated Carbon Observation System ICOS - CO


European Environment Agency NRT Airbase

Remarks

As the proposed structure doesn’t change much from the current, this requirement

will fit in the proposed structure. Normalisation of the costs doesn’t affect this

requirement as it has no costs defined.

The name includes three coverages, where in the field coverage only one coverage

can be defined. Should it be possible to define more than one coverage to a

requirement? If so, is this possible with one target_accuracy field? Each coverage

may have its own target_accuracy.

Timeliness: see remark previous requirement.



3.4 Maritime Environment Monitoring

The Copernicus Marine Environment Monitoring Service (CMEMS) provides regular and

systematic reference information on the physical state, variability and dynamics of the

ocean and marine ecosystems for the global ocean and the European regional seas.

The observations and forecasts produced by the service support all marine

applications. For instance, the provision of data on currents, winds and sea ice help to

improve ship routing services, offshore operations or search and rescue operations,

thus contributing to marine safety.

The service also contributes to the protection and the sustainable management of

living marine resources in particular for aquaculture, fishery research or regional

fishery organisations.

Physical and marine biogeochemical components are useful for water quality

monitoring and pollution control. Sea level rise helps to assess coastal erosion. Sea

surface temperature is one of the primary physical impacts of climate change and has

direct consequences on marine ecosystems. As a result of this, the service supports a

wide range of coastal and marine environment applications.

Many of the data delivered by the service (e.g. temperature, salinity, sea level,

currents, wind and sea ice) also play a crucial role in the domain of weather, climate

and seasonal forecasting.

Mercator Océan is the entrusted entity for the implementation of the service.



3.5 Climate Change

The Copernicus Climate Change service responds to environmental and societal

challenges associated with human-induced climate changes.

The service will give access to information for monitoring and predicting climate

change and will, therefore, help to support adaptation and mitigation. It benefits from

a sustained network of in situ and satellite-based observations, re-analysis of the

Earth climate and modelling scenarios, based on a variety of climate projections.

The service will provide access to several climate indicators (e.g. temperature

increase, sea level rise, ice sheet melting, warming up of the ocean) and climate

indices (e.g. based on records of temperature, precipitation, drought event) for both

the identified climate drivers and the expected climate impacts.

The Copernicus Climate Change service is under implementation. In November 2014,

the European Commission signed a Delegation Agreement with ECMWF (European

Centre for Medium-Range Weather Forecasts) for the implementation of the service.

Future requirements 3.5.1

The current version of the database doesn’t contain requirements of the Climate

Change service. So no data is available to test. Instead we can figure out what kind of

requirements are to be expected from this service.

Climate indicators will be based on a observations of a long historical period and/or

result from climate models which produce simulation result for a long period in the

future. This may require an extra field to define the temporal coverage of a

requirement. E.g. a dataset should cover at least a historic period of 30 years.

Climate models simulate several scenarios. This may require an extra field to define

the minimum number of scenarios.



3.6 Security

The Copernicus service for Security applications aims to support European Union

policies by providing information in response to Europe’s security challenges. It

improves crisis prevention, preparedness and response in three key areas:

Border security;

Maritime surveillance;

Support to EU External Action.

Border Security.

In the area of border security, main objectives are to reduce the number of illegal

immigrants entering the EU undetected, to reduce the death toll of illegal immigrants

by rescuing more lives at sea and to increase internal security of the European Union

as a whole by contributing to the prevention of cross-border crime.

FRONTEX is the entrusted entity to implement the border surveillance component of

the Copernicus Security Service. The objective is to support the EU’s external border

surveillance information exchange framework (EUROSUR) by providing real time data

on what is happening on land and sea around the EU’s borders.

Maritime Surveillance

In the area of maritime surveillance, the overall objective of the European Union is to

ensure the safe use of the sea and to secure Europe’s maritime borders. The

corresponding challenges mainly relate to safety of navigation, marine pollution, law

enforcement, and overall security.

EMSA is the entrusted entity to implement the operation of the maritime surveillance

component of the Copernicus Security Service. EMSA uses space data from

Copernicus Sentinel 1 satellites combined with other sources of maritime information

to effectively monitor maritime areas both within and outside the EU’s borders.

Support to EU External Action

As a global actor, Europe has a responsibility in promoting stable conditions for

human and economic development, human rights, democracy and fundamental

freedoms. In this context, a main objective of the EU is to assist third countries in a

situation of crisis or emerging crisis and to prevent global and trans-regional threats

having a destabilising effect.



Annex 1. Proposal for renaming of table and field names

For several reasons we advise to change table names of field names.

This table gives an overview for all table names.

current table name Proposed table name

core_services services

consortiums entrusted_entities

available_datasets datasets

requirement_cases insitu_requirements_x_products

provider_cases insitu_requirements_x_datasets

inspire_links insitu_requirements_x_inspire_annexes

eionet_data_flows_links insitu_requirements_x_eionet_data_flows

Table 2. Table renaming

This table gives an overview for all field names.

Table name Current field

name Proposed field name

insitu_requirements insitu_id insitu_requirement_id

type_id data_type_id

thresh_accuracy threshold_accuracy

core_services cs_id core_service_id

entrusted_entities cons_id entrusted_entity_id

cs_id service_component_id

service_component sc_id service_component_id

cons_id entrusted_entity_id

themes sc_id service_component_id

products prod_id product_id

insitu_requirements_x_products case_id insitu_requirement_x_pro

duct_id

prod_id product_id

insitu_id insitu_requirement_id

data_policy policy_id data_policy_id

data_sustainability data_sust_id data_sustainability_id

data_provider provider_id data_provider_id

data_security security_id data_security_id

insitu_requirements_x_datasets pcase_id insitu_requirement_x_dat

aset_id




datasets policy_id data_policy_id

data_sust_id data_sustainability_id

provider_id data_provider_id

type_id data_type_id

target_accuracy accuracy

inspire_annexes inspire_id inspire_annex_id

insitu_requirements_x_inspire_annexe

s

inspire_link_id insitu_requirement_x_insp

ire_annex_id


inspire_theme_id inspire_annex_id

eionet_data_flows eionet_id eionet_data_flow_id

insitu_requirements_x_eionet_data_flo

ws

eionet_link id insitu_requirement_x_eio

net_data_flow_id


eionet_id eionet_data_flow_id

Table 3. Field renaming

Documents

Quarterly Implementation Report - Copernicus · No parts of this document may be photocopied, reproduced, stored in retrieval system, or transmitted, in any form or by any means whether