DSD-INT 2015 - The future of Delft-FEWS - Simone van Schijndel, Deltares

29 October 2015

The future of Delft-FEWS

Simone van Schijndel and many others

Deltares

Delft-FEWS2020

The world is changing...

29 October 2015

...and it is

going faster

and faster!

1951

Von Neumann built the

first real computer for

meteorological

computations

29 October 2015

The cloud...

29 October 2015

every known hurricane and tropical storm since 1851,

developed by IDVSolutions. Data from NOAA, USA

New data sources and techniques

29 October 2015

Accountability becomes increasingly important

29 October 2015

Prevention and adaptation

29 October 2015

http://www.customerintelligence360.com/5-absolutely-stunning-big-data-visualizations/

Relations change...

and organisations with it

>20 years of Delft-FEWS Applications

Flood Forecasting Systems

• Running models

• Integration of (spatial) data

• Ensembles etc

Water Information Systems

• Data validation

• Data analysis

• Long records of data

Model Management platform

• Off-line scenario analysis

• Complex modelling systems

And...not to forget

Water quality

Ground water

Storm surge

Data science

Our starting points

Delft-FEWS is for free

Delft-FEWS is open software, but not open source

Delft-FEWS balances flexibility and stability

Delft-FEWS stimulates sharing of knowledge and data

Delft-FEWS connects organisations and stimulates collaboration

The community is essential!

29 October 2015

Some considerations…

Why do we have Delft-FEWS?

• Focus on good and innovative projects, but no ‘world domination’

• Sharing new knowledge and technologies

• Access to organisations with good ideas (and financial resources)

Growth is essential! We never say no, unless…

But...

Beware of the Microsoft cyclus (lack of focus)…

• Growth→increasing complexity

• Complexity → dependency of Deltares

• Dependency of Deltares → stops growth and innovation

29 October 2015

Delft-FEWS worldwide

Anno 2015...

Time for reflection

Time for consolidation for what we have achieved so far

Time to think about the future

Three aspects:

1. Market developments

2. Technical developments

3. Organisation of product management

And...attention for (more) active participation of our users

29 October 2015

29 October 2015

Official establishment of the Delft-FEWS Community Strategy

Board (CSB) per 27th October 2015

• Participate actively in Delft-FEWS2020 process

• Discuss support and maintenance and other services

• Funding mechanisms and business model

• Exchange of experiences

• Representation of the community

Community Strategy Board, who

Organisation Represented by

Bureau of Meteorology in Australia

(BoM) Jeff Perkins

Dutch Waterboards (DWB) Roger de Crook

Environment Agency in England (EA) Deborah Summerskill

Federal Office for the ENvironment in

Switzerland (FOEN) Therese Buergi

National Weather Service in the USA

(NWS) Jon Roe

Rijkswaterstaat in the Netherlands

(RWS) Marc Phillipart

29 October 2015

Community Strategy Board, how

IUD

Oct ‘15

NA UD

March ‘16

AU UD

April ‘16

NL UD

June ‘16

IUD

Oct ‘16

29 October 2015

1st formal

CSB meeting 2nd CSB

meeting

January CSB

meeting

Topic: S&M

and other

services

May CSB

meeting

Topic: TBD

Sep CSB

meeting

Topic: TBD

Challenges in the organisation of PM

Support & Maintenance

• Effective management of an increasing

number of applications

• ICT is increasingly complex

• Maintain a solid base for Delft-FEWS

Developments

• Prioritise following plans for

Delft-FEWS2020

• Generate sufficient financial funds

to continuously develop

Delft-FEWS

Continuity in the organisation is essential!

29 October 2015

Delft-FEWS architecture

Delft-FEWS architecture is mainly developed in 2003; technology has evolved:

So, there is a need for a structured review of the Delft-FEWS architecture

Where do we need to focus on:

• Keep it simple, less software components (Jboss, Tomcat, ..)

• Use of standard components introduces many times more complexity!

• Where possible share ideas and knowledge with other Deltares software

systems

Do not forget:

• Why have we made certain decisions in the past, there is an idea behind!

• Watch out for buzz-words; What is the role of FEWS?

Delft-FEWS: 2015-2020

Developments: interesting or hype?

What is our view?

Not everything is a hype!

See the changes!

Mostly project driven!


Focus for now

1. Big data and Large Datasets

2. Open FEWS Archive

3. Thick and Thin clients

4. Cloud computing and Simplification of FEWS roll-out

5. Webservices

Big Data

• Large amounts of unstructured data from many sources (i.e. Twitter)

• FEWS will not use this data directly

• We expect external services/software will use BIG data, structure it (find

correlation) and provide information that can be used in FEWS

• FEWS must be able to deal with large amounts of structured data

• Trends in meteorological forecasts, satellite data, ..


Delft-FEWS and Large Datasets

We need to find a solution how FEWS handles large data sets

• Large number of scalar data in Forecasting System seems no problem.

• Operational meteorological forecasts (ensembles) also no problem.

• Store data for longer period in operational database can be problematic

• There must be a solution for storage of large model data sets that do not fit in

FEWS operational database

• Model states of Swan or Delft-3D, 3Di (store on file system, NoSQL)

• Model results (first process, filter and only store important information)

• It is key we connect to open standards

• In 2014 Development of new Open Archive has started

• Will play a central role in storage of large datasets


Delft-FEWS and Open Archive

• Development of new FEWS Archive is important and essential

• Store validated historical data in the archive

• Store historical grids in archive (i.e. Radar grids)

• Users must have direct access to archive (Seamless integration in OC)

• Fits in Open Data Strategy of many FEWS users


Delft-FEWS: Thick Client and Thin Client

• Discussion on Delft-FEWS Client

• Keep the FEWS Thick client; maybe redesign but keep it for the experts

• For the users the look and feel of the application must not change too

much (large investments in training)

• CITRIX and Remote Desktop solutions are more and more popular

• Will we initiate the development of a Thin client (Web-application) from

Delft-FEWS

• Sounds nice, but is complex to develop and maintain

• Large dependency on popular browsers

• Deltares has developed a Delta Data Viewer; we need to have an open

discussion if this is a general solution

• Current focus on webservices and Open-API for the development of

Web viewers by others (Lizard, HydroNet, Delta Data Viewer, ..)


Cloud Computing

• Remote computing through the internet (Cluster is internal!)

• Make use of free capacity of external servers

• FEWS needs to have an answer on this development!

Examples:

• 3Di as cloud solution (FEWS-3Di, how to run models remotely)

• Bring your model to the data; reduce data traffic over the internet

Solutions:

• Simple: Start a model that runs in the cloud

• Complex: Send a complete FEWS system (FSS) with config to the data and

run on a cloud computer. Minimise the data traffic and maintain a good

performance

• Important in the discussion is performance, security, robustness


Cloud Computing: Pilot 2015

The Deltares ID-lab is used to promote new ideas and applications developed by

Deltares (also: initiate new developments)

One of the new developments is a Proof of Concept (PoC) with Microsoft to use

the Azure cloud for FEWS. The PoC for 2015 includes several steps:

1. Roll-out FEWS “as is” in the cloud (FSS and MC+Database)

2. Investigate with the ‘Experts’ what short term developments are needed to

simplify the roll-out of FEWS in the cloud

3. Investigate if larger developments are needed to make better use of the cloud

One of the Goals of this initiative is to simplify the current FEWS components for

easier rollout of Server and Client components and automated upgrades


Web Services

• A webservice makes it possible to request a service from a server. This can be

a data request, execution of a task or process or send data.

• Difference between provider and consumer webservices

• Current FEWS webservice functionality will be made consistent and extended

• FEWS-PI, UM-Aquo, WaterML, Json, …

• Will support running FEWS form external servers (start workflow, modifier,

etc..).


29 October 2015

Other items to consider

• Simplify Delft-FEWS configuration (partly done) and installation

• Run new model types under Delft-FEWS

• Model in the Cloud de Cloud (3Di)

• Model as a service

• Use of new data data types

• Ecologial data (already developed for Dutch waterboard Waternet)

• 3D data; new Deltares models (NetCDF Unstruc)

• Use of 2-D spectra

• Simplify error solving in Delft-FEWS

Share additional requests and ideas with us now or

by email to [email protected]!!

29 October 2015

Next steps

January 2015 – October 2015

• Internal discussions

• Discussions with clients and consultants in NL and abroad

• Discussion in CSB

Before March 2016:

• Design step-wise approach for review current architecture

• Further detail technical plans and prioritize resulting activities

• Explore opportunities for funding our plans

User Days next year present first results

En hoe doen we het samen?

Nog in te vullen

29 October 2015

To raise the water level in a fish pond

(Zhang Huan, 1997, Beijing)

Software

DSD-INT 2015 - The future of Delft-FEWS - Simone van Schijndel, Deltares