© 2015 IBM Corporation

Geospatial analytics with dashDB in the cloudSession # 1823

Torsten Steinbach @torsstei

Please Note:• IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole discretion.

• Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision.

• The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract.

• The development, release, and timing of any future features or functionality described for our products remains at our sole discretion.

Performance is based on measurements and projections using standard IBM benchmarks in acontrolled environment. The actual throughput or performance that any user will experience will varydepending upon many factors, including considerations such as the amount of multiprogramming in theuser’s job stream, the I/O configuration, the storage configuration, and the workload processed.Therefore, no assurance can be given that an individual user will achieve results similar to those statedhere.

2

3

dashDB

area distance length perimeter

Multistring MultiPolygon

Polygon LineString Multipoint

Point Curve Surface Geometry

Collection

Geometry

Multicurve Multisurface

Geospatial Data

Loading Esri Shapefiles

Cloudant Warehousing with GeoJSON

{GeoJSON}

Other data sources

{GeoJSON}

GeoJSON data comes in 3 flavours:

{ "type": "LineString", "coordinates": [ [ 2.3200, 48.8657 ], [ 2.2951, 48.8738 ]]}

...as „Simple“ Geometry{ "type": "FeatureCollection", "features": [ { "type": "Feature", "properties": { "name": "Champs Elysées"}, "geometry": { "type": "LineString", "coordinates": [ [ 2.3200, 48.8657 ], [ 2.2951, 48.8738 ]] } }, { "type": "Feature", "properties": { "name" : "Notre-Dame"}, "geometry": { "type": "Point", "coordinates": [ 2.3497, 48.8528 ] } } ]}

...as Feature Collection

{ "type": "Feature", "properties": { "name": "Champs Elysées"}, "geometry": { "type": "LineString", "coordinates": [ [ 2.3200, 48.8657 ], [ 2.2951, 48.8738 ]] }}

...as Feature

{ "_id": "75000", "_rev": "08066f8ecd5f780646aa1573460852c", "type": "FeatureCollection", "features": [ { "type": "Feature", "properties": { "name": "Le Louvre"}, "geometry": { "type": "Point", "coordinates": [2.3382, 48.8605] } }, { "type": "Feature", "properties": { "name" : "Notre-Dame"}, "geometry": { "type": "Point", "coordinates": [2.3500, 48.8530] } } ]}

_id _rev type75000 08066f8... FeatureCollection

_id array_index type properties_name geometry75000 0 Feature Le Louvre POINT (2.34 48.86)75000 1 Feature Notre-Dame POINT (2.35 48.85)

Cloudant Database: SightsInParis

DashDB WarehouseTable: SIGHTSINPARIS

Table: SIGHTSINPARIS_FEATURES

GeoData & dashDB

{GeoJSON}

WKT((),())

WKB

GML

GeoSpatial Analytics In dashDB

• Implements OGC SFS & ISO SQL/MM part 3 standards for spatial See http://www.iso.org/iso/catalogue_detail.htm?csnumber=38651

• Spatial data type ST_GEOMETRY (hierarchy)• Enables spatial operations (e.g. joins) in database through spatial

operators available as user defined functions• Dedicated support in ESRI tools starting V 10.3• dashDB - R support through extension to ibmdbR package

Spatial Functions and Predicates in dashDB

SELECT a.name, a.type FROM highways a, floodzones b WHERE ST_Intersects(a.location,b.location) = 1 AND b.last_flood > 1950

SELECT a.road_id, a.time, i.id, ST_Distance(a.loc, i.loc,’METER’) as distance FROM accidents a, intersections i WHERE ST_Distance(a.loc,i.loc,’METER’) < 10000 AND a.weather = ‘RAIN’

- accidents near intersections

- highways in flood zones

ST_Distance(g1,g2)

?

ST_Intersects(g1,g2)

?

Spatial Constructor Functions

ST_Point(x, y, srs_id) – create point at this location

ST_Point(‘POINT (-121.5, 37.2)’, 1)

ST_Linestring(‘LINESTRING (-121.5 37.2,-121.7 37.1)’,1)

ST_Polygon(CAST (? AS CLOB(1M)),1)– For host variable containing well-known text, well-known binary, or shape representation

Spatial Predicates – WHERE Clause

ST_Distance(geom1, geom2) < distance_constant or var

ST_Contains(geom1, geom2) = 1

ST_Within(geom1,geom2) = 1

EnvelopesIntersect(geom1, geom2) = 1

EnvelopesIntersect(geom1, x1, y1, x2, y2, srs_id) = 1

ST_Area(geom) < some_value

Spatial Functions that Create New Spatial Values

ST_Buffer(geom, distance)

ST_Centroid(geom)

ST_Intersection(geom1, geom2)

ST_Union(geom1, geom2)

Functions that Return Information About a Spatial Value

ST_Area(geom), ST_Length(geom)

ST_MinX(geom, ST_MinY(geom), ST_MaxX(geom), ST_MaxY(geom)

ST_IsMeasured(geom)

ST_X(geom), ST_Y(geom)

ST_AsText(geom)

And Many More …

ST_AreaST_AsBinaryST_AsTextST_BoundaryST_BufferST_CentroidST_ContainsST_ConvexHullST_CoordDimST_CrossesST_DifferenceST_DimensionST_DisjointST_DistanceST_EndpointST_EnvelopeST_EqualsST_ExteriorRingST_GeomFromWKBST_GeometryFromTextST_GeometryN

ST_GeometryTypeST_InteriorRingNST_IntersectionST_IntersectsST_IsClosedST_IsEmptyST_IsRingST_IsSimpleST_IsValidST_LengthST_LineFromTextST_LineFromWKBST_MLineFromTextST_MLineFromWKBST_MPointFromTextST_MPointFromWKBST_MPolyFromTextST_MPolyFromWKBST_NumGeometriesST_NumInteriorRingST_NumPoints

ST_OrderingEqualsST_OverlapsST_PerimeterST_PointST_PointFromTextST_PointFromWKBST_PointNST_PointOnSurfaceST_PolyFromTextST_PolyFromWKBST_PolygonST_RelateST_SRIDST_StartPointST_SymmetricDiffST_TouchesST_TransformST_UnionST_WKBToSQLST_WKTToSQLST_WithinST_XST_Y

And more…

Simplified Constructors from x,y WKT WKB GML shapeLinear referencingSpatial aggregationST_AsGMLST_AsShape

Harness the Full Power of SQL Outer join Common table expressions Recursive queries, sub-queries Aggregate functions Order by, group by, having clauses OLAP, XML, and more ...

WITH sdStores AS (SELECT * FROM stores WHERE st_within(location, :sandiego) = 1)SELECT s.id, s.name, AVG(h.income) FROM houseHolds h, sdStores sWHERE st_intersects(s.zone, h.location) = 1GROUP BY s.id, s.nameORDER BY s.name

Example problem: Determine the average household income for the sales zone of each store in the San Diego Example problem: Determine the average household income for the sales zone of each store in the San Diego area.area.

Predictive Analytics With R In dashDB

Backup

dashDB: Key Use Cases

• Minimize capital expense of DR solutionDR in the Cloud

We Bring Netezza Compatible Analytic Platform to the Cloud

Analytic Extension FrameworkUDX C++ API

Canned Analytics

Application Integration

AE Framework In-DB R In-DB LUAIn-DB Python In-DB Perl

OLAP Functions

ROW_NUMBER

RANK

LAG LEAD

DENSE_RANK Linear Regression

Kmeans Clustering Decision Tree

Association Rules

Association Rules

Naive Bayes

Spatial Operators

Contains

Touches

Within

Intersects

Crosses

Overlaps

R Wrapper Watson Analytics ESRI ArcGIS Connector …

Analytics Applications of ISVs and Customers

STDDEV

COVAR

……

Analytic Code & Algorithms:

Analytic Data:

Data pulled out and processed in analytic application

Analytic Applications

This is where we start from: All analytic processing done on application side

Analytics of Warehouse Data

SQLs

Analytic Code & Algorithms:

Analytic Data:

Simple data lookup & massage operations pushed down as SQL operations

Analytic Applications

Benefit: Acceleration with no SQL skills required

SQLs

Push Down Step 1: BLU tables only logically represented in analytic application

Accelerate Analytics for Warehouse Data

SQLs

Analytic Code & Algorithms:

Analytic Data:

Call built-in functions via SQL to execute typical algorithms inside db

Cloud Tooling

Analytic Applications

Benefit: Bring Standard Analytics to the Data

SQLs

Canned Algorithms

Push Down Step 2: Typical and popular algorithms pushed down to canned UDFs in the db

Accelerate Analytics for Warehouse Data

Lang

uage

Fra

mew

ork

(UD

X &

AE

)

Analytic Code & Algorithms:

Analytic Data:

Deploy customer code and call via special SQL function interfaces

SQLsSQLs

Canned Algorithms

Analytic Applications

Benefit: Bring Custom Analytics to the Data

Push Down Step 3: Execute entire customer analytic programs inside the db

Accelerate Analytics for Warehouse Data

Don’t forget to submit your Insight session and speaker feedback! Your feedback is very important to us – we use it to continually improve the

conference.

Access your surveys at insight2015survey.com to quickly submit your surveys from your smartphone, laptop or conference kiosk.

We Value Your Feedback!

26

27

Notices and DisclaimersCopyright © 2015 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without written permission from IBM.

U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM.

Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of initial publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. THIS DOCUMENT IS DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED. IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISING FROM THE USE OF THIS INFORMATION, INCLUDING BUT NOT LIMITED TO, LOSS OF DATA, BUSINESS INTERRUPTION, LOSS OF PROFIT OR LOSS OF OPPORTUNITY. IBM products and services are warranted according to the terms and conditions of the agreements under which they are provided.

Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice.

Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary.

References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in which IBM operates or does business.

Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials and discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant or their specific situation.

It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and interpretation of any relevant laws and regulatory requirements that may affect the customer’s business and any actions the customer may need to take to comply with such laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law.

28

Notices and Disclaimers (con’t)

Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the ability of any such third-party products to interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.

The provision of the information contained herein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual property right.

•IBM, the IBM logo, ibm.com, Aspera®, Bluemix, Blueworks Live, CICS, Clearcase, Cognos®, DB2® , DOORS®, Emptoris®, Enterprise Document Management System™, FASP®, FileNet®, Global Business Services ®, Global Technology Services ®, IBM ExperienceOne™, IBM SmartCloud®, IBM Social Business®, IMS™, Information on Demand, ILOG, Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®, PureData®, PureExperience®, PureFlex®, pureQuery®, pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, Smarter Commerce®, SoDA, SPSS, Sterling Commerce®, StoredIQ, Tealeaf®, Tivoli®, Trusteer®, Unica®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.

© 2015 IBM Corporation

Thank You