Producing INSPIRE compliant datasets

INSPIRE Data Models

Finnish Meteorological Insitute

Finnish Meteorological Institute

Roope Tervo

Finnish Meteorological Institute opened its data in 2013.

Basically everything that FMI has property rights was opened.

Data is provided in freely in machine readable format.

29.4.2015 INSPIRE Data Models | Finnish Meteorological Insitute | Roope Tervo 2

FMI Open Data

https://en.ilmatieteenlaitos.fi/open-data


FMI Open Data Portal follows INSPIRE requirements.

29.4.2015 3

FMI Open Data Portal

Meta data

Data Models

Services

The very same data portal works as Open Data and

INSPIRE portal.

INSPIRE Data Models | Finnish Meteorological Insitute | Roope Tervo

Catalog Service

(CSW)

o Based on GeoNetwork

29.4.2015 4

View Service (WMS)

o Based on GeoServer

o Only the most common layers

published


Download Service

(WFS 2.0)

o Web Feature Service (WFS) 2.0 Simple Profile

o Based on stored queries

o Predefined data sets with possibility for additional parameters (i.e. time and area)

o In-house production

29.4.2015 5INSPIRE Data Models | Finnish Meteorological Insitute | Roope Tervo

Data set Description Time

Interval

Estimated

publish date

Weather

Observations

Temperature, Wind,

Humidity, Ground

Temperature…

10 min Open,

older data to be

added

Sun Radiation UV, Short and Long

Term Radiation…1 min Open

Marine

Observations

Waves, Sea

Temperature, Sea

Level…

1 h Open

Weather Radars Precipitation Rate,

Precipitation Amount…5 min Open,

older data to be

added

Lightning Thunder Strikes in

Finland5 min Open

29.4.2015 INSPIRE Data Models | Finnish Meteorological Insitute | Roope Tervo 6

Example of Data Sets

29.4.2015 7


Data set Description Time Interval Estimated

publish date

Real Time

Observations

Real Time Observations from

specific location(s)

AWS 2010 –

Soundings 1959 –

Flashes 1998 –

Sea Level 1971 –

Waves 2005 –

Open

older data will

be added

Climatological

Observations

Dayly and monthly

temperature mean and

extreme values from weather

stations

1959 - Open

Climatological

Observations

Monthly temperature and

precipitation rate mean

values interpolated to grid

1961 - Open

Climatological

Reference

Climatological Reference.

Temperature, humidity,

pressure, precipitation

amount and snow depth.

Reference seasons:

1971-2000 1981-

2010

Open


29.4.2015 8



publish date

Weather forecast

model HIRLAM RCR

Point forecasts and grid

dataLatest model

run

(4 times a day)

0…54 h

Open

Sea forecast models Sea level point

forecasts, Wave (WAM)

and current (HBM) as

grid data

Latest model

run

(4 times a day)

0...54 h

Open

Environmental

Monitoring Facilities

Weather observation

stations, radars…2015

Aviation

Observations

METAR 30 min open

Ground & mast

observations

Special observations

from ground and masts2015


29.4.2015 9



publish date

Air Quality

Observations

Air Quality Observations 1h 2015-2016

Silam Model Dispersion Model for Air

Quality, Forest Fire and

Pollen

Latest model

run (once a day)

0…96h

2015

HELMI Ice Model Ice forecast model Latest model

run

(4 times a day)

0...54 h

open

Soundings Temperature, Humidity,

Pressure, Wind from

ground to 25 km height

2 times a day 2015


Data Models

o Observations and point forecasts as GML

o The same data is published in:

o MultiPointCoverage

o MeasurementTimeSeries

o SimpleFeature

o Gridded data is provided in

appropriate binary format (Grib,

NetCDF, GeoTiff…)

o WFS members contains the metadata ‘envelope’ with a link to a actual data


Data Models

gmlcov:MultiPointCoverage

29.4.2015 11

gml:rangeSet

gml:doubleOrNilReasonTupleList

The data is listed for every

point defined in domain set.

gml:domainSet

gmlcov:simpleMultiPoint

The coverage is

defined as a list of

points in 4

dimensional grid (lat,

lon, height, time).

gmlcov:rangeType

The parameters

listed in range set

are defined in

separate element.


Cons

- Not intuitive

- No natural

structure of XML

XSLT and

Xpath don’t work

Pros

+ Compact

+ Efficient

+ Small file size

+ Works for many

data types

29.4.2015 12

Data Models



Data Models

wml2:MeasurementTimeseries

29.4.2015 13

wml2:MeasurementTimeseries

One member contains time

series for one parameter

and one location


Cons

- Lots of repetition

- Large file size

- Heavy for DOM-

based parsers

- Don’t work i.e. for

thunder strikes

Pros

+ Intuitive

+ Easy to use

+ XSLT & XPath

works

29.4.2015 14

Data Models

wml2:MeasurementTime

series


Data Models

SimpleFeature

29.4.2015 15

SimpleFeature

One member contains one

time, one parameter and

one location


Cons

- Lots of repetition

- Very large file size

- Heavy for DOM-

based parsers

Pros

+ Intuitive

+ Easy to use

+ XSLT & XPath

works

+ Ready client

support

29.4.2015 16

Data Models

SimpleFeature


29.4.2015 17

Data Type Data Format

Observations wml2:MeasurementTimeseries


SimpleFeature

Point Forecasts wml2:MeasurementTimeseries


SimpleFeature

Lighting Observations gmlcov:MultiPointCoverage

SimpleFeature

Grid Forecasts XML Envelope + Grib2/NetCDF

Radar Images GeoTiff / PNG images

METAR IWXXM


29.4.2015 18

Data Models File size Comparison

81.7

52.9

1.81.3 1.2 0.20

102030405060708090

Document Size[MB]

CompressedDocumentSize[MB]


29.4.2015 19

Data Models Popularity

Comparison

80

19.8

0.20

102030405060708090

Downloads[%]


Intranet

DMZ

Server 1

GS 1 GS 2 GS 3

Server 2

GS 1 GS 2 GS 3

Server 3

GS 1 GS 2 GS 3

Load Balancer

Configuration

GeoServer

Data

(NFS)

Configuration

(NFS)Database


Intranet

DMZ

Backend

(WFS)

Load Balancer

Data

(NFS)

Configuration

(NFS)Database


Frontend Frontend

Backend

(WFS)

Backend

(binary data)Backend

(binary data)

Open Data Service

Cluster

S1 S2 S3

Client Data Service

Cluster

S1 S2 S3

Load Balancer

Configuration

Data

(NFS)

Configuration

(NFS)Database


Weather Data – Volumes

29.4.2015 23

• In-situ weather measurement 1 TB

• Weather radar data 50 TB

• NWP model gridded data (FMI)

• HIRLAM 120 TB

• AROME 230 TB

• Satellite image data (FMI)

• Globsnow 32 TB

• Other 50 TB

• Climate model data (FMI) 29 TB

• Other models

• SILAM 100 TB

• Tuuliatlas 21 TB

• Other 300 TB


Producing INSPIRE Data Products

Observations


Database BS Data ServerBS Data Server

WFS Plugin


Point Forecasts


File

SystemBS Data Server

BS Data Server

WFS Plugin


Grid Forecasts 1/2

File


BS Data Server

WFS Plugin



Grid Forecasts 2/2

File


BS Data Server

Download

Plugin



Radar Images 1/2

PostGIS

DBBS Data Server

BS Data Server

WFS Plugin



Radar Images 2/2

PostGIS

DBGeoServer


INSPIRE Data Sets

How to define a data set?

o All weather observations from

Finland?

Would cause over 50 000 000 Observations (XML file size ~37 G)

o All observations from one

observation station?

Would cause over 200 data sets

o Even one year’s observations cause

too large data set to handle


INSPIRE Data Sets

Meteorological data is a constant

flow of observations

FMI has one data set per data

type, i.e. one for ground weather,

observations, one for Hirlam

weather forecasts, etc…

Every data set have predefined

area and time range.


INSPIRE Data Sets

It is notable that data set

response depends on time it’s

requested

Unique identifiers are not

reasonable


And a little over300 000 data downloads

per day(3,7 req/s)

At the moment about 7200

registered users

29.4.2015 33

Some Experiences


Practicallyno clientsupportscomplexfeatures

Although standards are followed, there’s

a gap between provided data model

and clients’ capabilities


Some Experiences

GeoServer is modified to support

stored queries in WFS 2.0 (released

in version 2.7)

FMI is going to open the same data as simple features to

support clients


Some Experiences

Industry ishappy to usestandardized

services

Amateur andfreelancer coders

would prefer simple JSON API


Some Experiences

So far maybe

even more

professional

interest than

private

Quite many

expected a user

interface to load data

to i.e. to Excel

instead of machine

readable interface


Some Experiences

…but suites quite

well for exchanging

(subsets of) data.


Data format is too

verbose for clients to

use directly…

Some Experiences

For now,

very few have

been interested in

forecast models

as a grid data

Point forecasts,

observations and

radar images are the

most interesting data

types


Some Experiences

www.fmi.fi

http://www.slideshare.net/tervo/


http://www.slideshare.net/tervo/


Documents

Producing INSPIRE compliant datasets