Netlobs Manipulating Gridded Data in a

Relational World

Neil Stamps

Technical Architect

AgendaAgenda

Introduction to Lost Wax

The problem framed

Oracle 10g capabilities

NetLobs – NetCDF meets database

Introduction to Lost Wax

AdvancedSoftware

Engineering

Multi-agent SystemsContinual R&DRoles based business analysisAgent frameworkProducts (SPL)Innovation projects

Server-side SystemsJ2EE / MS.NETSOA and Web ServicesLegacy integration / wrappingGIS Mapping Solutions

MobileDistributed computingJava PDAs / phones

The problem framedThe problem framed

DTI DEWS research project partners:

Met Office, Reading University, BADC, BMT, IBM

Provides web services to multiple domains

GADS provides marine services

Oracle target platform

InternetWeb Service

InternetWeb Service                     

Forecast Data

SQL

GADS

Oracle – Blob supportOracle – Blob support

Oracle 10g deployment platform

Large object support Blob – max size (4GB –1)*block

e.g. 32k block = 128TB maxClob, nClob – max size as per Blob

Extension supportJava, C extensionsJava stored proceduresCustom data types (cartridges)

Remote symbolic debugging (JDeveloper)

Oracle - Custom data typesOracle - Custom data types

Provide encapsulation of attributes and methods

Introduce OO capabilities into relational world

Allow unstructured data to be queried

Extensions to indexes allow efficient queries

Nested tables provide collection capability

NetLobs – NetCDF ‘SmartLobs’NetLobs – NetCDF ‘SmartLobs’

Provides NetCDF file capability to Oracle

Encapsulates data and meta-data in single type

Physical implementation agnostic

Automatic extraction and storage of meta-data

Interrogate meta-data without blob enquiry

Extraction over single or multiple Netlobs

NetLobs – NetCDF ‘SmartLobs’NetLobs – NetCDF ‘SmartLobs’

NetCDF 2.2 open source Java libraries

NetLob wraps Oracle for NetCDF Files

Extraction interfaces based upon current GADS requirements:

SubsetReductionConcatenation

Higher level interfaces to be layered over basic functionality

Cartridge invocationCartridge invocation

PL/SQL interface maps to Java call (or C)

Oracle instantiates NetLob object

Object implements SQLData interface

Blob pointer passed to Java, Random access provided via ‘internal’ JDBC

NetLobs – Data ingestionNetLobs – Data ingestion

Upload using Oracle SQL*Loader

Upload in two–phase method

Validation at Netlob creation

System optimisation based upon once-only performance hit at extraction of meta-data

Meta-data ‘chunks’ will facilitate query by value

NetLobs – PerformanceNetLobs – Performance

Reference GADS system provides predictable, linear extraction performance

NetLob cartridge aims to achieve similar performance characteristic over large data extractions

Optimisation tailored to larger extractions

Moving forwardMoving forward

Storage and retrieval of rotated data

Pluggable interpolation framework

Offloading processing to GRID

Enhanced meta-data to meet community needs

Query by-value enhancements

Any Questions?Any Questions?

Neil.Stamps@Lostwax.comwww.LostWax.com