Raster data in GeoServer and GeoTools: Achievements, issues and future developments

Raster data in GeoServer and GeoTools: Achievements, issues and future developments

Ing. Simone Giannecchini, GeoSolutions Ing. Daniele Romagnoli

FOSS4G 2011, Denver 12th-16th September 2011

Outline

Who we are

Raster Developments

JAITools

ImageIO-Ext

GeoTools

GeoServer

GeoBatch


GeoSolutions

Founded in Italy in 2006

Expertise

• Image Processing, GeoSpatial Data Fusion

• Java, Java Enterprise, C++, Python

• JPEG2000, JPIP, Advanced 2D visualization

Supporting/Developing FOSS4G projects

GeoTools, GeoServer

GeoBatch, GeoNetwork

Clients

Public Agencies

Private Companies

http://www.geo-solutions.it


http://www.geo-solutions.it/



JAITools

High performance raster processing

Extends, complements, replaces Oracle Java Advanced

Imaging (JAI) Library

Open Source, BSD license

Low Level Operations

e.g. RangeLookup

Higher Level Operations

Vectorize, Contour

Works mostly in pure raster space (Image Processing)

Stable release is 1.2.0


http://jaitools.org/

JAITools – Low Level Operations

Basic Extensions to JAI processing

RangeLookup

Raster (re)classification

Raster Statistics

ZonalStats

ClassifiedStats*

KernelStats

High Perf ColorMap*

ColorMap Extraction/Inversion

High Perf VectorBinarize

Vector to raster conversion


JAITools – High Level Operations

Vectorize

Polygon Extraction

Contour

Isolines extraction

Raster Algebra (Jiffle)

MapCalc like extension

Generic Raster Processing


http://code.google.com/p/jiffle/

ImageIO-Ext

High Performance raster I/O

Extends, complements, replaces Oracle JAI ImageIO

Library

Open Source, LGPL and BSD licenses

Plugins:

BigTiff*, netCDF-CF,GriB1,MatFile 5, HDF

JP2000 (Kakadu)

GDAL integration via ImageIO-ext

JPEG2000, MrSID, ECW

BigTIFF, ERDAS Image, HDF4

Complete list here


http://java.net/projects/imageio-ext



http://java.net/downloads/imageio-ext/Documentation/gdalsupportedformats.html

ImageIO-Ext – Improving Tiff


Improved standard Tiff reader and writer

Support for r/w double and float data type

JP2 compressor/decompressor*

Improved metadata Management

More aggressive caching

Based on soft references

BigTiff support

Read/write

Reduced resolution page support

Overviews!

ImageIO-Ext – JPEG2000

Kakadu based reader/writer

Wraps kakadu JNI APIs as ImageIO ImageReader/ImageWriter

Support for GeoJP2

Supporting additional JPEG2000 code params

Tiling, Corder, Tile Marker, Part Tile Marker, Sprofile , etc…

Some of them needed for NITF support

Fine grain control over JPEG2000 codestream creation

FOSS4G 2011, Denver

12th-16th September 2011

ImageIO-Ext – GDAL

Wraps GDAL behind ImageIO API

Mainly Read Capabilities

Write support experimental (JP2Kak, GeoTiff)

Geotiff, JP2Kak, ECW, MrSID, JP2MrSid, Erdas, IDRIS, Esri Binary Grid, Envisat, NITF, DTED, DOQ1, DOQ2, etc..

Expose Generic IIOMetadata for basic Dataset info

Georeferencing, Band layout, CRS, etc..

Expose Specific IIoMetadata for Driver specific info

E.g. MrSid metadata

1.1.0 supports GDAL 1.7.3 directly (no custom bindings!)


http://java.net/downloads/imageio-ext/Documentation/gdalsupportedformats.html

GeoTools

The Open Source Java GIS Toolkit

LGPL

Management of Geospatial data

CRUD Operations

Raster and vector

Rendering

GDAL Support

Based on best-of-breed Open Source projects

Java Advanced Imaging (JAI), ImageIO

Java Topology Suite (JTS)

FOSS4G 2011, Denver

12th-16th September 2011

http://geotools.org/

GeoTools - JPEG2K Improvements


Leverage on ImageIO API

Read/Write Capabilities

Fine grain support for writing with code params

GeoJP2 support

Other capabilities

Re-Tiling on the fly

ColorMasking

Immediate (Java2D) VS Deferred Mode (JAI)

Multithreaded loading (per tile)

GeoTools - Tiff & BigTiff


Leverages on ImageIO API

Read/Write Capabilities

Internal/External Overviews support

Improved Metadata management

Less memory

Wise caching – more speed

Heavy Refactor ongoing


ColorMasking

Immediate (Java2D) VS Deferred Mode (JAI)

GeoTools - Raster Processes


Basic Building Blocks for GeoServer WPS

Clip and Ship (raster and vector), buffering, vector intersection, vector union, etc…

JAITOOLs

Vectorize, Contour

Statistics

GDAL based*

gdal_translate wrapper

gdalwarp wrapper

org2ogr wrapper

GeoTools - ImageMosaic


Wraps multiple homogeneus raster as a single one

Improvements Support for footprint on single tiles

Support for external catalog of tiles

Autoindexing of directories

Support for TIME, ELEVATION

Support for OGC & CQL Filtering

Multithreaded loading

Support for heterogeneous overviews layout

Supports any format with an ImageIO reader


ColorMasking on the fly

Immediate VS Deferred Mode

GeoTools - ImagePyramid


Multiresolution/Multiscale tile based raster plugin

Take a large raster split in small chunks/tiles mosaic and downsample start over until happy

Upon request

Select right resolution level select needed chunks/tiles assemble do some magic and return

Improvements

Automatically recognize output of gdal_retile

Automatic tile indexing

Other improvements inherited from ImageMosaic

GeoTools – Optimized Reprojection


Standard Approach Accurate Transformation

Each pixel is back mapped using the full transformation

Grid to world on the target

Backwards projection towards the source

World to grid on the source

Slow!, cannot be optimized in hardware

Optimization Approximate Transformation

Local Linear Approximation using Grid Warps

Use a set of known positions and their transformations

Everything else is linearly interpolated between the known positions

Estimation through iterative process (speed vs quality)

http://demo.geo-solutions.it/share/OptimizingRasterReprojection.pdf



Linearly interpolate on the top and bottom lines Linearly interpolate between the two resulting points



GeoServer

GeoSpatial enterprise gateway

Java Enterprise

Management of raster and vector data

Standards compliant

OGC WCS 1.0, 1.1.1 (RI)

OGC WFS 1.0, 1.1 (RI), 2.0

OGC WMS 1.1.1, 1.3

OGC WPS 1.0.0

Google Earth/Maps support

KML, GeoSearch, etc..


http://geoserver.org/display/GEOS/Welcome

GeoServer - Rendering Transformations


In Place Transformations during rendering

Specified in SLD docs

Applied right before the rest of the

directives

Can be used to:

Extract and draw contours on

the fly

Extract and draw polygons

on the fly

Extract and draw symbols (e.g. arrows) on the fly Transform geometries on the fly

(Add your own!)

http://geo-solutions.blogspot.com/2011/01/developers-corner-have-your-sld.html




GeoServer - Rendering Transformations

Band Combine + Symbology

Raster to point vector

On-the-fly Contouring

On-the-fly Polygonalization


GeoServer – Improved Raster Rendering






Hit JAVA2D Bottleneck – Scalability Issue

http://bit.ly/qJcZBi

http://bit.ly/oe0CHo

Created direct raster rendering path with JAI

Drop-in replacement (1 raster at time)

2x speedup

2x/3x scalability improvement under heavy load

Enabled/Disabled via Java Switch





GeoServer – Improved Raster Chains Performance






Improved internal JAI Concurrency

Introduced and Improved JAI Tile Recycling

Reduced depth of JAI Processing Chains

More aggressive factory caching

Added a few specific high perf JAI operations

e.g. crop

Created Specific high perf ROI class

Improving Tile Caching

Work ongoing

Transparent File vs Memory Caching when writing

raster

GeoServer – More Raster WPS Processes






Relies on GeoTools Processes

Vectorize

Contouring

RasterClassification

ZonalStats

GDAL Processes

gdal_translate

gdalwarp

ogr2ogr

GeoServer – Support for TIME & ELEVATION dimensions + Filters






WMS TIME and ELEVATION dimensions

…&TIME=20110914T00:00:000Z&ELEVATION=10

Implemented thanks to ImageMosaic

Support for the Current keyword

Support for Default values

Support for

TimeInstant

Discrete Range Tb/Te/Res, Eb/Ee/Res

Continuous Range Tb/Te, Eb/Ee

Any combination of the above

Support for OGC and CQL filters

Filter Mosaic granules based on index attributes

http://www.youtube.com/watch?v=Xj-ftS7Di_g&feature=relmfu





GeoBatch

Real-time produced data

Remote Sensing, In situ, Models, etc..

Recognizable, isolate data streams

Flow

Incoming series of geospatial data

Specific format

Defined rules for preprocessing

Defined rules for ingestion and/or exploitation

Automatic Ingestion and preprocessing No manual intervention

Example, GeoTiff preprocessing

Producer GeoServer

GeoNetwork

Tile Overviews Publish


http://geobatch.geo-solutions.it/

GeoBatch – GeoTiff Preprocessing






GeoTiff serving optimizations Internal Tiling

Compression

Internal/External Overviews

Mosaicking

Pyramiding

GeoServer publishing

2 Options

Pure Java based on ImageIO and GeoTools

External Processes with calls to GDAL

GeoBatch – MetOc Data Preprocessing






Optimization of MetOc for WMS/WCS serving

Conversion from

GRiB 1

netCDF-CF

COAMPS

BUFr*

To

Multiple 2D geotiff (Internal Overviews)

Indexing with ImageMosaic

More preprocessing

Fixing No Data

Change UoM

Band Processing Wind(U,V) Wind ( mod, theta)

The End

Questions? [email protected]


GeoTools - ImageMosaic




A raster reprojection involves chaining togheter the standard reprojection with the grid to world transformations of source and target



Set of evaluation vectors used in the Recursive algorithm

Test distance between actual position of the reprojected point and the predicted by a linear interpolation



Technology

Raster data in GeoServer and GeoTools: Achievements, issues and future developments