Embed Size (px)
Citation preview
DNV GL © 2014 7th November 2014 SAFER, SMARTER, GREENER DNV GL © 2014
7th November 2014
Karl John Pedersen
OIL & GAS
Analysing Big Data in ArcGIS
1
AIS based risk modelling
Esri European Petroleum GIS Conference 2014
DNV GL © 2014 7th November 2014
Introduction
Large amounts of data are produced from monitoring systems
Processing and analysing this data provides a challenge
How does DNV GL use ‘Big data’ – AIS vessel tracking
2
Background and challenges
Solution Implementation
examples Q & A
DNV GL © 2014 7th November 2014
Background
3
400 offices
100 countries
16,000 employees
150 years
Environmental Modelling and GIS unit within Oil & Gas, DNV GL, Oslo
– DNV GL - International company headquartered in Norway
– Global classification, certification, technical assurance and advisory company
GIS used in risk modelling and visualisation for customers
Many projects use AIS data
How have we adapted to increasing amounts of AIS data
Examples from Oil & Gas and Maritime sectors
DNV GL © 2014 7th November 2014
What is AIS
AIS - Automatic Identification System - vessel tracking data
Create lines – append lines spanning multiple days
Join to ship database to get attributes e.g Vessel type & size, engine & fuel type
4
DNV GL © 2014 7th November 2014
Analysis examples
Examples where AIS data are used in risk modelling
5
DNV GL © 2014 7th November 2014
View vessel traffic around an offshore installation
6
DNV GL © 2014 7th November 2014
Calculate vessel density around installation – collision risk
7
DNV GL © 2014 7th November 2014
Risk to pipelines from fishing activity
8
Installation
DNV GL © 2014 7th November 2014
Number of fishing vessels per grid cell – quantifies risk
9
DNV GL © 2014 7th November 2014
Fishing activity vs. depth – risk from vessel anchors
10
DNV GL © 2014 7th November 2014
Developing risk models in ArcGIS
AIS data used in further analysis
– AIS data from ArcGIS often used as input to DNV GL risk models
– Results visualised/mapped in ArcGIS
Risk models increasingly developed in ArcGIS:
11
AIS
Database
Process AIS data in ArcGIS
Run risk model
Map/
Visualisation
Data export for users
Excel/Access
ArcGIS
DNV GL © 2014 7th November 2014
ArcGIS modelling example
Input accident frequencies from Excel
Run risk model - Python
Arctic Shipping Risk
12
DNV GL © 2014 7th November 2014
How we previously handled AIS data
– Data in SQL Server accessed via IBM Cognos
– Imported text files from Cognos into ArcGIS
Challenges:
– Large text files, lacking data structure
– Many steps - possibility of errors
– Limited geographic selection
– Time consuming
– Increasing data quantities
– Historical data
– Global data
Need to improve access to AIS ‘Big data’
13
DNV GL © 2014 7th November 2014
What is Big data?
Big data – …a collection of data sets so large and complex …difficult to process
using … traditional data processing applications (Wikipedia)
Example from DNV GL:
Database of > 2,5 billion AIS points
14
DNV GL © 2014 7th November 2014
Solution for handling AIS data
IBM Netezza data warehouse implemented to improve data access
Implemented in existing Cognos environment
What is Netezza:
Data warehouse appliance from IBM
Provides rapid access and analysis of ‘Big data’
Includes a Spatial Esri package
AIS data readable directly in ArcGIS
15
DNV GL © 2014 7th November 2014
Making data available to ArcMap from Netezza
16
Netezza Spatial Esri
package
AIS data imported
from supplier
Spatial data
created in Netezza
Polygon data
loaded via WKT
format
DNV GL © 2014 7th November 2014
Accessing data in Netezza from ArcGIS
Install IBM Netezza ODBC driver
Add data as normal – geographic data has own icon
Data access via:
ArcToolbox Query Layer
Python
17
DNV GL © 2014 7th November 2014
ArcToolbox Query Layer
Processing carried out on server
Select geographical area
Points mapped directly in ArcGIS
Data downloadable to ArcGIS
Work reduced from days to minutes
18
DNV GL © 2014 7th November 2014
Allows analysis of global AIS data
19
DNV GL © 2014 7th November 2014
Serving maps to users - ArcGIS Server
ArcGIS Server service for AIS data created
Issues creating service for whole dataset
20
DNV GL © 2014 7th November 2014
Serving maps to users - Cognos
Cognos use spatial data from Netezza
Esri Maps for Cognos
Implemented simple mapping tool
21
DNV GL © 2014 7th November 2014
Limitations
– ArcGIS cannot write to Netezza
– Issues creating ArcGIS service
– No support for raster
– Limited user base
22
DNV GL © 2014 7th November 2014
Summary
Advantages of implementing solution:
– Enables significantly improved access to AIS data
– Allows analysis which was not previously possible
– Part of existing data warehouse
– Ability for existing non-GIS users to analyse data using familiar tools
23
DNV GL © 2014 7th November 2014
SAFER, SMARTER, GREENER
www.dnvgl.com
That’s all ! Questions?
24
Contact details:
Karl John Pedersen
Principal Specialist, Environmental Modelling and GIS,
Environmental Risk Management, Høvik, Norway
E-mail [email protected] Mobile +47 950 02 061
