The overall classification of this presentation is:

Classified by:

Derived from:

Declassify on:

Mike Jeffe, Contractor

May 4, 2016

Hootenanny: Creating an Open Source Conflation

Engine

Approved for Public Release, 16-374

“The age of free public and open source geospatial feature data has significantly increased the opportunity to conflate such

data to create enhanced products.”

Source: Proc. SPIE 8747, Geospatial InfoFusion III, 874703 (23 May 2013)

Approved for Public Release, 16-374

Lots of Disparate Data Sources

• Large amounts of overlapping data from various sources

Government

Commercial

Crowdsource

• Leverage best parts of multiple sources to build a best-of-breed conflated map that is superior to any one input source

Source: Vector Data Sources: OpenStreetMap (OSM) public data; NGA Feature Foundation Data (FFD)

Approved for Public Release, 16-374

Unifying two or more separate datasets, which share certain

characteristics, into one integrated all-encompassing result.

Target Dataset

Source Dataset

Output Model

Conflation

Approved for Public Release, 16-374

What makes conflation so challenging?

• Different database schemas

• Varying geometric fidelity (scale, extraction

spec)

• Tools typically limited to desktop solutions

• Manual techniques are very tedious

Approved for Public Release, 16-374

Hootenanny Conflation Suite Hootenanny is a suite of emerging non-proprietary tools to support fully-automated and

semi-automated conflation of geospatial vector datasets (roads, buildings, and POIs)

Key features: ► Feature matching and merging logic for both geometry and attribute information.

► Flexible key value schema and PostgreSQL database based on OpenStreetMap (OSM) schema

► Reference (Vertical), Cookie Cutter (Horizontal), and Average Conflation

► Service oriented architecture (SOA) that leverages both OGC Web Services (WMS, WFS, WPS), REST

and node.js

► Currently being integrated with the MapBox iD Editor

► Support various import/export physical formats (Shapefiles, or Esri FGDB), schemas (OSM, TDS,

MGCP)

► Open Source Software (OSS) and available for download at https://github.com/ngageoint/hootenanny

c1

c2*

a3

b2

a2 b3

b4

a1 b1

High Score

Low Score

Roads Buildings POIs Web Client

Sources: DigitalGlobe GBM (EnhancedView license); OSM data from openstreetmap.org (pink); MGCP data (pink) from GEOINT New Zealand Data Service (CC-BY-3.0-NZ)

Approved for Public Release, 16-374

Hootenanny Pair-wise Conflation Workflow

A B C D

A Evaluate, Extract, Translate, Load

B Conflate

C Review and Correct Conflicts

D Export to Final Schema

Approved for Public Release, 16-374

Conflation Feature Matching Evaluation

Peer Review

Source: Proc. SPIE 8747, Geospatial InfoFusion III, 874703 (23 May 2013)

Approved for Public Release, 16-374

Hootenanny – where to access

Source: DigitalGlobe GBM (EnhancedView license); OSM data from openstreetmap.org (pink); MGCP data (pink) from GEOINT New Zealand Data Service (CC-BY-3.0-NZ)

Approved for Public Release, 16-374

Tropical Cycle Evan – December 2012

Source: NPP satellite's VIIRS instrument around 1220Z on December 13, 2012 (NOAA/NASA)

Image over a destroyed bridge in Samoa's capital Apia, Friday, Dec. 14, 2012 (credit: AP Photo/Seti Afoa.) (Source: Jeff Master’s Wunderground Blog)

Approved for Public Release, 16-374

Apia, Western Samoa

Source: Samoa Wikipedia web page (CC BY-SA 3.0)

Source: OpenStreetmap.org (ODbL )

Approved for Public Release, 16-374

Data Sources

Source: OpenStreetmap.org (ODbL )

Source: GEOINT New Zealand Data Service , (CC-BY-3.0-NZ)

Source: Wikimapia data with Wikimapia Satellite basemap (CC-BY-SA)

Source: Developing a Google Earth Database for Spatial Data in Samoa, John Van Hoesen, Ph.D. (CC-BY-SA)

KML

Standardized Public Data Custom - Analyst Data

Approved for Public Release, 16-374

Use Case: Conflate POIs, Buildings, and Roads in

Apia, Samoa

Need:

Updated POIs (population centers, shelters, schools, hospitals, etc. )

Enrichment of transportation features (bridges, roads, tunnels)

Building footprint polygons

Available Data: ► OSM: Solid general coverage. Very good in urban areas that are part of an existing map

campaign.

► MGCP: Not as dense as OSM but very accurate (derived from high resolution imagery and contain good attribution

► Analyst: additional features, enriched POIs from Analysts and experts.

Workflow:

Two conflation operations:

1. MGCP - > OSM

2. Results from above -> Shoebox

Approved for Public Release, 16-374

MGCP vs. OSM: Roads, Buildings and POIs

Source: OpenStreetMap data (Orange) extracted via Overpass API; MGCP data (Green) downloaded from GEOINT New Zealand Data Service (CC-BY-3.0-NZ); OpenStreetMap basemap Service (ODbL)

Approved for Public Release, 16-374

MGCP vs. OSM: Roads and POIs

Source: OpenStreetMap data (Orange) extracted via Overpass API; MGCP data (Green) downloaded from GEOINT New Zealand Data Service (CC-BY-3.0-NZ);

Approved for Public Release, 16-374

OSM vs MGCP Road tag comparison chart

Approved for Public Release, 16-374

Hootenanny Data Ingest

Hoot KVS

OSM Shoe Box

MGCP

Approved for Public Release, 16-374

Hootenanny Conflation Workflow

OSM

MGCP

+

+

MERGED OUTPUT OF CONFLATED OSM AND MGCP

ANALYST DATA

FINAL CONFLATED DATASET

Approved for Public Release, 16-374

Conflating in Hootenanny UI

Sources: DigitalGlobe GBM (EnhancedView license); osm data from openstreetmap.org (orange); MGCP data (teal) from GEOINT New Zealand Data Service (CC-BY-3.0-NZ);

Approved for Public Release, 16-374

Viewing post conflation statistics

Approved for Public Release, 16-374

Interpreting Conflation Results

Blue – MGCP (Secondary Layer) Orange- OSM (Reference Layer) Green – Merged Red Highlight - Current Reference Review feature Blue Highlight – Current Secondary Review feature

Sources: osm data from openstreetmap.org (orange); MGCP data (teal) from GEOINT New Zealand Data Service (CC-BY-3.0-NZ);

Approved for Public Release, 16-374

What features automatically merged

Sources: DigitalGlobe GBM (EnhancedView license); osm data from openstreetmap.org (orange); MGCP data (teal) from GEOINT New Zealand Data Service (CC-BY-3.0-NZ);

Approved for Public Release, 16-374

POI Manual Review Example

Sources: DigitalGlobe GBM (EnhancedView license); osm data from openstreetmap.org (orange); MGCP data (teal) from GEOINT New Zealand Data Service (CC-BY-3.0-NZ);

Approved for Public Release, 16-374

Hootenanny – Exporting Conflated Results

Sources: Bing Aerial Imagery (Bing License) available on OpenLayers.org; Conflated MGCP/OSM data.

Approved for Public Release, 16-374

An Open Source Solution

Hootenanny was developed and released for open source in coordination with the National Geospatial-Intelligence Agency

Approved for Public Release, 16-374

Current Efforts

Automated, RPM, Deployment

R&D:

► Intersection matching

► Conflation change sets

Build and refine existing conflation workflows

Join us:

► Backlog: https://github.com/ngageoint/hootenanny/issues

► Fork and submit a pull request:

https://github.com/ngageoint/hootenanny/pulls

Approved for Public Release, 16-374

Summary

► Conflation provides the ability to leverage best parts, geometry and

attributes, of input sources to build a superior best-of-breed conflated map

► Hootenanny developed to conflate geospatial vector datasets, roads (polylines), buildings (polygons), POI’s (points)

► Built around OSM key value pair attribute schema and PostgreSQL database for robust attribute mapping to standardized schema

► Translation support for standard schemas, TDS, MGCP, and UTP. Other data sources can be supported with the Translation Assistant

► Built upon Mapbox iD Editor software to provide a streamlined, intuitive, web enabled, standards based approach to conflation

► Hootenanny’s open source model provides flexible licensing requirements, shareable, customizable, and ability to leverage user contributions. It is available at https://github.com/ngageoint/hootenanny/.

Approved for Public Release, 16-374

Questions?

Approved for Public Release, 16-374