A Microservice Architecture for the Processing of Large Geospatial Data … · 2018. 12. 17. ·...

A Microservice Architecture for the Processing of Large Geospatial Data in the CloudMichel Krämer

Urban planning use case

Paparoditis et al. (2012)!2

Identify trees in 3D point clouds acquired by LMMS

Challenge:

Process data at least as fast as it is acquired, in order to be able to continuously monitor tree growth

Land monitoring use case

Highhydraulicenergy

Heavyrains

Smallandsteepbasins

Floodsandland slides

Abandonedagriculturalterraces

Analyse topography and orographic precipitation to prepare against hazardous events (floods and landslides)

Challenge:

Being able to continuously monitor evolution of terrain for the first time

Problem statement

• Similar challenges in other use cases • change detection in urban areas • traffic pattern analysis • monitoring of seabed and coastal changes • etc.

• These use cases require “Big Geo Data”

• Goals cannot be reached with state-of-the-art approachesYang et al. (2011), Kitchin, R., & McArdle, G. (2016)

Processing of large geospatial data

Cloud-based data processing Developers/researchers

Use platformContribute

processing algorithms

Problem statement – Users

Requirements:

A platform to processBig Geo Data

Automate recurring processing tasks

High-level interface for process automation

My approach:

Microservice architecture for Cloud-based processing

Workflow management

Domain-Specific Language

Requirements:

My approach:

Workflow management

Requirements:

My approach:

Workflow management

Problem statement – Developers/researchers

Requirements:

Execute existing algorithms in the Cloud without modifications

Focus on algorithms and not on the details of Cloud

Orchestrate algorithms to workflows

My approach:

Service integration based on lightweight metadata

Workflow management

Requirements:

My approach:

Workflow management

Requirements:

My approach:

Workflow management

Hypothesis

A microservice architecture and Domain-Specific Languages can be used to orchestrate existing geospatial processing algorithms, and to compose and execute geospatial workflows in a Cloud environment for efficient application development and enhanced stakeholder experience.

Hypothesis

A microservice architecture and Domain-Specific Languages can be used to orchestrate existing geospatial processing algorithms, and to compose and execute geospatial workflows in a Cloud environment for efficient application development and enhanced stakeholder experience.

Microservice architecture

Domain-Specific Languages

Enhanced stakeholder experience

Orchestrate existing geospatial processing algorithms

Compose and execute geospatial workflows

Efficient application development

Related work – Cloud for geospatial applications

Mostly for Smart Cities

e.g. Khan, Anjum, & Kiani (2013), Krylovskiy, Jahn, & Patti (2015)

Single experiments

Specific platform/processing model

e.g. Qazi, Smyth, & McCarthy (2013), Warren et al. (2015), Li et al. (2010)

e.g. Ajiy et al. (2013), Eldawy & Mokbel (2013)

No workflow management

Related work: My approach:

Comprehensive platform for user-defined workflows

Supports various use cases

Supports mix of multiple programming paradigms

Workflow management

Single experiments

Workflow management

Single experiments

Workflow management

Single experiments

Workflow management

System design

Architecture overview

Data accessservice

Distributed file system(Storage cloud)

Processing services(Processing cloud)

Catalogue serviceJobManager

Workflows Datacatalogue

Servicecatalogue

Notification

Workfloweditor

Workflow service

GIS expert

Data browser

File upload/download

Main user interface

Interpreter

Parser

Krämer & Senner (2015)

Data accessservice

Servicecatalogue

Notification

Workfloweditor

Workflow service

GIS expert

Data browser

Main user interface

Interpreter

Parser

Data accessservice

Servicecatalogue

Notification

Workfloweditor

Workflow service

GIS expert

Data browser

Main user interface

Interpreter

Parser

Data accessservice

Servicecatalogue

Notification

Workfloweditor

Workflow service

GIS expert

Data browser

Main user interface

Interpreter

Parser

Data accessservice

Servicecatalogue

Notification

Workfloweditor

Workflow service

GIS expert

Data browser

Main user interface

Interpreter

Parser

Data accessservice

Servicecatalogue

Notification

Workfloweditor

Workflow service

GIS expert

Data browser

Main user interface

Interpreter

Parser

Processingservices

Node A Node B Node C

Job manager

MapReduce job

Algorithm B (distributed)

Algorithm A (single core)

Algorithm B (distributed)

Algorithm C (multi core)

Distributed file system

Algorithm D (GPU)

Data accessservice

Servicecatalogue

Notification

Workfloweditor

Workflow service

GIS expert

Data browser

Main user interface

Interpreter

Parser

Benefits of microservice architecture

• Focus

• Independency

• Development distributability

see also Krämer & Senner (2015) !14

• Composability

Presentation tier

Logic tier

Data tier

Presentation

DataData Data

GatewayGateway Gateway

Shared data

a) A monolithic software with astandard three-tier architecture

b) A pure microservice architec-ture with a complex deployment

c) A microservice architecture de-composed along bounded contexts

Data accessservice

Servicecatalogue

Notification

Workfloweditor

Workflow service

GIS expert

Data browser

Main user interface

Interpreter

Parser

Example Workflow

Data accessservice

Servicecatalogue

Notification

Workfloweditor

Workflow service

GIS expert

Data browser

Main user interface

Interpreter

Parser

Data accessservice

Servicecatalogue

Notification

Workfloweditor

Workflow service

GIS expert

Data browser

Main user interface

Interpreter

Parser

Dynamic workflow execution

A priori design-time knowledge (one DAG for the whole workflow)

A priori runtime-time knowledge (iterative)

Existing work: My approach:

JobManager

JobManager architecture

JobManager

Rule System

Processing Cloud

Processing Connector 1 Processing Connector n

Process Chain Manager

Service metadata

Data metadata ControllerR

HTTP Server

Working Memory

ProcessChains

Workflows

Client

Data accessservice

Servicecatalogue

Notification

Workfloweditor

Workflow service

GIS expert

Data browser

Main user interface

Interpreter

Parser

JobManager

Rule System

Processing Cloud

Service metadata

HTTP Server

Working Memory

ProcessChains

Workflows

Client

Data accessservice

Servicecatalogue

Notification

Workfloweditor

Workflow service

GIS expert

Data browser

Main user interface

Interpreter

Parser

JobManager

Rule System

Processing Cloud

Service metadata

HTTP Server

Working Memory

ProcessChains

Workflows

Client

Data accessservice

Servicecatalogue

Notification

Workfloweditor

Workflow service

GIS expert

Data browser

Main user interface

Interpreter

Parser

JobManager

Rule System

Processing Cloud

Service metadata

HTTP Server

Working Memory

ProcessChains

Workflows

Client

Data accessservice

Servicecatalogue

Notification

Workfloweditor

Workflow service

GIS expert

Data browser

Main user interface

Interpreter

Parser

JobManager

Rule System

Processing Cloud

Service metadata

HTTP Server

Working Memory

ProcessChains

Workflows

Client

Data accessservice

Servicecatalogue

Notification

Workfloweditor

Workflow service

GIS expert

Data browser

Main user interface

Interpreter

Parser

Controller

Execute workflow

Controller Rule System

load metadata

start reasoning

registerprocess chains

lookup processchain results

registerworkflow results

fire rules

newprocess chains

no newprocess chains

all process chains succeeded

JobManager

Rule System

Processing Cloud

Service metadata

HTTP Server

Working Memory

ProcessChains

Workflows

Client

Controller

Execute workflow

load metadata

start reasoning

fire rules

newprocess chains

JobManager

Rule System

Processing Cloud

Service metadata

HTTP Server

Working Memory

ProcessChains

Workflows

Client

Controller

Execute workflow

load metadata

start reasoning

fire rules

newprocess chains

JobManager

Rule System

Processing Cloud

Service metadata

HTTP Server

Working Memory

ProcessChains

Workflows

Client

Controller

Execute workflow

load metadata

start reasoning

fire rules

newprocess chains

JobManager

Rule System

Processing Cloud

Service metadata

HTTP Server

Working Memory

ProcessChains

Workflows

Client

Controller

Execute workflow

load metadata

start reasoning

fire rules

newprocess chains

JobManager

Rule System

Processing Cloud

Service metadata

HTTP Server

Working Memory

ProcessChains

Workflows

Client

lookup nextprocess chain

select node request processchain status

new running

register processchain results

finished

executeprocess chain

no nodeavailable

stillrunning

JobManager

Rule System

Processing Cloud

Service metadata

HTTP Server

Working Memory

ProcessChains

Workflows

Client

new running

finished

no nodeavailable

stillrunning

JobManager

Rule System

Processing Cloud

Service metadata

HTTP Server

Working Memory

ProcessChains

Workflows

Client

new running

finished

no nodeavailable

stillrunning

JobManager

Rule System

Processing Cloud

Service metadata

HTTP Server

Working Memory

ProcessChains

Workflows

Client

new running

finished

no nodeavailable

stillrunning

JobManager

Rule System

Processing Cloud

Service metadata

HTTP Server

Working Memory

ProcessChains

Workflows

Client

new running

finished

no nodeavailable

stillrunning

JobManager

Rule System

Processing Cloud

Service metadata

HTTP Server

Working Memory

ProcessChains

Workflows

Client

new running

finished

no nodeavailable

stillrunning

JobManager

Rule System

Processing Cloud

Service metadata

HTTP Server

Working Memory

ProcessChains

Workflows

Client

Controller

Execute workflow

load metadata

start reasoning

fire rules

newprocess chains

JobManager

Rule System

Processing Cloud

Service metadata

HTTP Server

Working Memory

ProcessChains

Workflows

Client

Controller

Execute workflow

load metadata

start reasoning

fire rules

newprocess chains

JobManager

Rule System

Processing Cloud

Service metadata

HTTP Server

Working Memory

ProcessChains

Workflows

Client

Controller

Execute workflow

load metadata

start reasoning

fire rules

newprocess chains

JobManager

Rule System

Processing Cloud

Service metadata

HTTP Server

Working Memory

ProcessChains

Workflows

Client

Controller

Execute workflow

load metadata

start reasoning

fire rules

newprocess chains

JobManager

Rule System

Processing Cloud

Service metadata

HTTP Server

Working Memory

ProcessChains

Workflows

Client

Controller

Execute workflow

load metadata

start reasoning

fire rules

newprocess chains

JobManager

Rule System

Processing Cloud

Service metadata

HTTP Server

Working Memory

ProcessChains

Workflows

Client

Results

Dataset:

City of Toulouse, 120.63 GiB,1.58 billion points, 529 tiles Acquisition time: 1h 53m

Evaluation results:

Processing time: 1h 51m 18 compute nodes on Fraunhofer IGD Cloud Almost linear scalability

Result: Point cloud with labels for individual trees

Dataset:

City of Toulouse, 120.63 GiB,1.58 billion points, 529 tiles Acquisition time: 1h 53m

Evaluation results:

Processing time: 1h 51m 18 compute nodes on Fraunhofer IGD Cloud Almost linear scalability

Result: Point cloud with labels for individual trees

Land monitoring use case

Dataset:

3D point cloud of Liguria Region 451.16 GiB, 17.35 billion points, 684 strips

Evaluation results:

Processing time: 35m 49s Previously (without my work): several days (*)

Result: Point cloud allowing for fast extraction of basins in a certain level of detail

* according to GIS users from the Liguria region (personal communication)

Highhydraulicenergy

Heavyrains

Smallandsteepbasins

Floodsandland slides

Abandonedagriculturalterraces

Scientific contributions

Microservice Architecture

• Scalability

• Modifiability

• Availability

Processing

• Service integration

• Service orchestration

• Dynamic workflow management

• Rule-based workflow execution

Workflow Modelling

• Domain-Specific Language (DSL)

• Method for DSL modelling

Scientific contributions

Microservice Architecture

• Scalability

• Modifiability

• Availability

Processing

• Service integration

• Service orchestration

• Dynamic workflow management

• Rule-based workflow execution

Workflow Modelling

• Domain-Specific Language (DSL)

• Method for DSL modelling

Conclusions

Research hypothesis is supported by • Positive results from detailed evaluation • Successfully satisfied stakeholder

requirements

Conclusions

My thesis documents a major step in the paradigmshift from desktop GIS to the Cloud within the

geospatial community and geoinformatics

Research hypothesis is supported by • Positive results from detailed evaluation • Successfully satisfied stakeholder

requirements

(Future) work

JobManager has been put into productionat Deutsche Telekom AG

Several improvements:• Create VMs on demand (OpenStack) • Capability-based scheduling • Improved security

General features

‣Geospa'al feature store ‣Schema agnos'c ‣Format preserving ‣Cloud-based ‣Event-driven ‣Easy to use/integrate

Michel KrämerFraunhofer-Institut für Graphische Datenverarbeitung IGD

Fraunhoferstraße 5

64283 Darmstadt michel.kraemer@igd.fraunhofer.de

A Microservice Architecture for the Processing of Large Geospatial Data … · 2018. 12. 17. ·...

Documents

Pregel: A System for Large-Scale Graph Processing · Large-Scale Graph Processing I Large graphs needlarge-scale processing. I A large graph eithercannot t into memoryof single computer

Online GeoSpatial Processing (OLGP): An Experimentation With UMN-MapServer

PROCESSING GEOSPATIAL DATA @ SCALE GEO(MESA/WAVE

A microservice architecture for the processing of large ...tuprints.ulb.tu-darmstadt.de/6956/7/2018-01-19_Kraemer_A-microser… · 2.2.4 Cloud architectures for geospatial processing

Large-scale 3D geospatial processing made possible · Informix, Microsoft SQL Server and RethinkDB are examples in this realm. Relational databases supporting 3D geometrical entities

Distributed geospatial information processing: sharing ... · PDF fileDistributed geospatial information processing: sharing distributed geospatial resources to support Digital Earth

Development of an IDL-based geospatial data processing ...aqrp.ceer.utexas.edu/projectinfoFY12_13/12-TN2/12-TN2 Final Report.pdf · Development of an IDL-based geospatial data processing

RT Geospatial Processing with NVIDIA GPUs, Storm, HyperDex | …on-demand.gputechconf.com/.../S3305-RT-Geospatial-Processing-GP… · via Socket (C++/C#/Java) via Thrift Client via

A Hierarchical Vector Data Model for Distributed Geospatial Processing

Geospatial Stream Query Processing using Microsoft SQL Server

HGrid A Data Model for Large Geospatial Data Sets in HBase

Block Processing Large Images

Geospatial Image Processing Services

Use of XML for Web-Based Query Processing of Geospatial Data · Metadata is the foundation for success of web-based query processing of geospatial data. Metadata refers to data about

Performance Benchmarking of Geospatial Applications · Performance Benchmarking of Geospatial Applications Vartika Bajpai . ABSTRACT - Geospatial applications deal with large volume

New Offering 12/2019: Integrated Cloud-based Geospatial ......Integrated Cloud-based Geospatial Processing Services (with Web API, Web App and Web Mapping) Processing Service 1: Turn

Geospatial Data Processing with Stetl - GeoPython2016.geopython.net/talks/geopython-2016-1024-v1.pdf · Geospatial Data Processing with Stetl Just van den Broecke GeoPython 2016 Muttenz

Scaling Geospatial Searches in Large Spatial Databases

Distributed Geospatial Analysis through Web Processing ...Distributed Geospatial Analysis through Web Processing Service: A Case Study of Earthquake Disaster Assessment Xiaoliang Meng1

Event-based Dissemination and Processing of Geospatial ... · Event-based Dissemination and Processing of Geospatial Information ... Receive Feature-JSON from Kafka ... Kafka MQTT