Real-Time ToolchainGSN - MeteoIO - GEOtop

Thomas EggerLausanne, 10.12.2009

GOAL

● Collect from Sensorscope stations● Channel data through GSN● Feed GEOtop with real-time data from GSN● Publish real-time simulation results

Step-by-Step

● Making GEOtop a real-time model● GEOtop & MeteoIO interaction● Toolchain (data acquisition &

simulation)● Using GSN Webservice (SOAP/WSDL)● Demonstration● Outlook

GEOtop Initial Condition

● Strictly linear processing● No usable start, pause and resume

mechanisms● Difficult configuration of parameters● Meticulous, time-consuming preparation

of data necessary, no safeguards (segfaults)

● How to make a model that can be used reliably with real-time data?

GEOtop's Dark Side● Time required for an expert to run a reference

simulation: 4 days● Extremely complicated internal data structures:

tricky even for senior developers● No checks on parameters files: when format changes

-> crashes without any hint of what is wrong● Moving target: heavy development occurring on

GEOtop➢ requires an expert to run, no automatic,

unattended execution possible

GEOtop Goals● Start, pause and resume mechanism ● Use different data sources, without manual

preparation (robustness, automatic filtering)● Easier configuration of simulation● Increased stability (unattended run)

Real-Time GEOtop

● Start, pause and resume through recovery mechanism

● Access to different data sources, data cleaning & filtering through MeteoIO

● Command-Line-Interface for configuration of simulation

● Making code more robust through debugging (segmentation faults, leaking, uninitialized variables)

GEOtop Recovery Mechanism

● Save status of simulation after each timestep● Recovery files in human readable form● Restore status for a given timestep● Pitfalls

– Global variables (in C: extern AND static vars)

– Precision of outputfprintf(f, “%.*f”, (int)LDBL_DIG, doublevalue);

GEOtop & MeteoIO (1)● MeteoIO is a library that provides

– access to different meteorological data sources through different plugins (meteo data, 2D/3D grids)

– advanced methods for data cleaning & filtering as well as spatial interpolation algorithms

● MeteoIO is written in C++, thus wrappers were necessary to “translate” data

● Two MeteoIO plugins developed– GSNIO (accessing GSN data through web service)

– GeotopIO (parsing CSV files)

GEOtop & MeteoIO (2)

Mixing C/C++

● Using compatible compilers (e.g. gcc)● Declare linkage for functions (extern keyword)

● Concept of classes compatible with structs● Deal with C++ exceptions

#include <iostream>extern "C" int print(int i, double d){ std::cout << "i = " << i << ", d = " << d;}

Toolchain (1)

1.Meteo stations transmit data to Sensorscope servers

2.GSN fetches data by listening to serial data stream, using web service or CSV wrapper

3.GSN triggers a simulation (GEOtop/A3D) if enough data present for next timestep

4.GEOtop/A3D access the data from GSN through the MeteoIO library

5.Simulation results are returned to GSN

Toolchain (2)

Sensorscope weather stations La Fouly

Sensorscope database Lausanne

GSN Network Lausanne

Simulation server running GEOtop/A3D

Lausanne

Toolchain (3)

Sensorscope weather stations La Fouly

Sensorscope database Lausanne

GSN Network Lausanne

Simulation server running GEOtop/A3D

Lausanne

[1] GPRS

[2] CSV, Serial, WS

[3] RSH

[4] SOAP

Post-processing[5] SFTP

File IO

What is GSN?

● GSN is a data acquisition network (JAVA)● It is build around the concept of “sensors”

– Real sensors (e.g. thermometer)

– Virtual sensors (e.g. averaging filter, GEOtop)

● Many wrappers for different sensors exist

Sensorscope GSN Interaction

● For GSN Sensorscope stations are sensors● CSV Wrapper

Importing data by parsing CSV files● Serial Listener

Listening to data packages as they come on a a TCP port (event based messaging)

● Future: Web service or database access

Web Services (1)

● By using web services an application can publish its functions on the internet

● Definition of what functions are available in web service description language (WSDL) in XML

● XML used to code and decode data and SOAP to transport it (over HTTP)

● Heterogeneous environments can easily be connected

Web Services (2)

● Machine-to-Machine interface (automation)● 2 ways of development:

– bottom-up (start with implementation, generate WSDL)

– top-down (start with design, generate skeleton)

● Similar technologies: RMI, CORBA, XML-RPC

GSN Web Service

● MeteoIO plugin GSNIO accesses data on GSN servers by means of a web service

● Simple access functions:– getSensors()

– getSensorInfo(stationname)

– getSensorLocation(stationname)

– getMeteoData(station, startDate, endDate)

● GSNWebService.wsdl

MeteoIO & GSN Webservice

● wsdl2h generates C++ header file● soapcpp2 generates C++ stubs● Incorporated into plugin GSNIO

● Example: Accessing GSN webservice through MeteoIO

Demonstration

● Retrieving data through MeteoIO– GEOtopIO plugin

– GSNIO plugin

● Script run.pl (Invokes A3D/GEOtop and post-processing)

● GSN webpage displaying results of sensors

Outlook

● GSN web interface for starting and running models

● GSN support for reading 2D Grids (DEM)● Session support (so different users can execute

simulation at same time) in both GEOtop and A3D