Upload
others
View
9
Download
0
Embed Size (px)
Citation preview
SmartMet Server Providing MetOcean Data
Roope Tervo, Mika Heiskanen | Finnish Meteorological Institute
In a Nutshell • Data and product server for MetOcean data • High capacity & availability
• FMI installation handles over 30 000 000 requests each day • Data is extracted and products generated on-demand • INSPIRE Compliant • Operative since 2008
• FMI client services (since 2008) • Finnish Meteorological Institute (FMI) Open Data Portal (since 2013) • Going to be used at Copernicus C3S Climate Data Store (ECMWF)
• Open source
22/3/2017 SmartMet Server | Roope Tervo, Mika Heiskanen 2
In a Nutshell • Several input sources
• GRIB-, NetCDF-, etc. files (multi-dimensional grid data)
• PostGIS database (vectors)
• Point database (point observations)
• Several output interfaces and formats • WMS, WFS 2.0
• JSON, XML, ASCII, HTML, SERIAL
• GRIB1, GRIB2, NetCDF
22/3/2017 SmartMet Server | Roope Tervo, Mika Heiskanen 3
Usage
22/3/2017 SmartMet Server | Roope Tervo, Mika Heiskanen 4
FMI Open Data Portal & FMI INSPIRE Data Services
Backend for clients’ web services
Integration to clients’ systems Backend for
mobile applications
Backend for FMI Client Services
Backend for FMI public pages
• Basis of most FMI product generation
• Published in 2016 in GitHub • https://github.com/fmidev/smartmet-server
• MIT Licence • Documentation in GitHub
• FMI will host the development • Small contributions with pull requests • In larger contributions, implementation plan is
recommended (in GitHub wiki) • CLA (Contributor Licence Agreement) will be
required
22/3/2017 Open Source Software @ Finnish Meteorological Institute | Roope Tervo 5
Open Source
Architecture
22/3/2017 SmartMet Server | Roope Tervo, Mika Heiskanen 6
• Frontend • Load balancer • Knowledge about
backend services • Backend
• Different backends may contain different services
• Plugin-based architecture • Engines provide shared
access to the data • Plugins provide services
(APIs) built upon engines
Most Important Components
22/3/2017 SmartMet Server | Roope Tervo, Mika Heiskanen 7
• Frontend • Provides HTTP 1.1 server • Monitors status of backend services and provides
load balancing • Provides LRU product cache
• Data Engines (providing C++ API) • Querydata engine provides access to the grid data • Observation engine provide access to the point data
in database • Geonames and gis engines provide geolocation
information
Most Important Components
22/3/2017 SmartMet Server | Roope Tervo, Mika Heiskanen 8
• Plugins (providing HTTP API) • WMS: Generates SVG images from grid
data on-demand, which are rendered to requested raster format
• WFS: Point data output for grid data and observations
• Timeseries: Custom point data interface with support for aggregate values over time and area
• Download / WCS: Grid data output
Post-Processing Capabilities
22/3/2017 SmartMet Server | Roope Tervo, Mika Heiskanen 9
• Corrects the data based on accurate DEM (up to 30 meter resolution) and land/water information
• Calculates derivative parameters
• Support for aggregate values over time and area
FMI Setup In 2016
22/3/2017 SmartMet Server | Roope Tervo, Mika Heiskanen 10
• 2 frontends • RAM: 256G
• CPU: 24x 2.10GHz
• OS: RHEL7 • 5 backends
• RAM: 12G
• CPU: 24x2.50GHz
• OS: RHEL7 • Load Balancer
• F5 BIG IP 11
WFS 140 ms/req WMS 130 ms/req Timeseries 30 ms/req Autocomplete 4 ms/req
Performance Production (FMI Setup)
22/3/2017 SmartMet Server | Roope Tervo, Mika Heiskanen 11
• > 30 000 000 req/day
• Baseline 200 req/s
• Peaks over 650 req/s
Average response times Typical load
Performance Load Tests (Production Setup)
22/3/2017 SmartMet Server | Roope Tervo, Mika Heiskanen 12
• Scenario based on operative use at FMI • Peaks over 4300 req/s • Avg 173 ms, 95% of responses in 244 ms, median 54 ms • Possibly heavy data requests require QoS management
• Independent queues for slow and fast queries
Roadmap
22/3/2017 SmartMet Server | Roope Tervo, Mika Heiskanen 13
Native GRIB and NetCDF support for input data
Support for GRIB and NetCDF data as input data without converting data to internal data format
Clustering support over Internet
Possibility to provide data from it’s original source via single API (bring users to data)
WCS support
Implement WCS interface for download plugin
www.fmi.fi
https://github.com/fmidev https://en.ilmatieteenlaitos.fi/open-data
http://roopetervo.com
http://www.slideshare.net/tervo