Data Interoperability: An Introduction...Esri UC2013 . Technical Workshop . Technical Workshop 2013 Esri International User Conference July 8–12, 2013 | San Diego, California Data

Esri UC2013 . Technical Workshop .

Technical Workshop

2013 Esri International User Conference July 8–12, 2013 | San Diego, California

Data Interoperability An Introduction Bruce Harold – Esri

Dale Lutz – Safe Software


Why Data Interoperability?

Background


What is the Data Interoperability Extension? ESRI & SAFE Co-Development based on FME

• Format Support –Adds support for 200+ data formats –Plus Your Custom Formats –Direct use in ArcGIS Desktop

• Quick Data Translation –Data Interoperability Tools Toolbox –Convert between data formats

• Spatial ETL (Extract, Transform, Load) –Schema Mapping (may be Dynamic) –Transformation of Geometry –Logic Controls Data Flow – Per Feature


ArcGIS Data Interop Common Tasks • Data Integration

- Bring multiple formats together • Data Migration

- Process of moving data between systems • Data Cleanup

- Clean up errors in the data and streamline data preparation

• Data Distribution - Distribute data to different systems

• Data Merge - Merge changes from edited data

• Data Validation - Geometry and Attributes

ETL


Data Interoperability Differentiators Not Just About Schema Manipulation

• Change Detection • 3D PDF Writing • XML Documents as Features or Attributes • JSON Documents as Features or Attributes • Protocol Support (HTTP, RSS…) • Web and Compressed Inputs • Parallel Processing • Vertical Coordinate Systems • BLOB Attributes


Basics: OGC Support What you get just by installing


OGC GML Simple Feature Format Support Support for GML-SF in ArcGIS Desktop

• GML – SF – simple features • WFS – optionally non-cached for auto refresh • WFS – cache timeout

• Note: Installation of the extension is required


Bruce Harold

OGC Formats: GML & WFS

Demo


Demo Summary: GML & WFS

• Comes Free - But you have to install - Simple Features Only

• Better with the extension licensed

- Non-simple parameters supported

• Direct Read of GML, consume WFS

• Important in many jurisdictions


Using Formats Direct Read of any format


Direct Read of Formats

• Formats recognized by Data Interop. - File-based, by extension, like .tab, .mid, .skp, .accdb - Visible in Catalog window browse tree - May be extended with Custom Formats

• Directly usable in ArcGIS - Save time converting data

• Consume external data - ArcPad AXF - Other application formats - Web feeds


Easy Format Selection • Browse or Search


When to Browse and when to Connect Browse & Direct Read uses format defaults

• Many formats have optional parameters - Select subsets, set conditions, passwords, filters…

• Connections give you access to format parameters • Connections let you set Coordinate Systems


Interoperability Connections Persist between application sessions

• Convenient to access in browse tree • Retain your settings • Off-disk data

- Web Feeds - Google Spreadsheets

• Share with colleagues - .FDL file created - C:\Users\<user>\AppData\Roaming\Safe

Software\Interoperability


Bruce Harold

Direct Read

Demo


Demo Summary: Direct Read

• Direct Read Summary - Directly read, inspect and browse within ArcGIS - Works like native ESRI formats (no GDB behavior)

- Access directly in Desktop Apps, Geoprocessing

- No development or customization required


Quick Translation Quick Import & Quick Export GP Tools


Quick Data Import/Export Enabling Extension Provides ‘Quick’ Tools

• Quick Export - To any format

• Quick Import - To staging Geodatabase

• Not Configurable - Is this OK?


Automation and Quick Translations • Use Quick Tools in ModelBuilder

- Get any format into a temporary File GDB - Handle Multiple Formats In/Out


Bruce Harold

Quick Tools

Demo


Demo Summary: Quick Import/Export

• “Quick” GP Tools - Useful in ModelBuilder - Also Scriptable in Python

- Easy interaction with others

- Departments - Contractors - 3rd Party Software Users


Leveraging Formats Custom Formats


What if you need your own format? Save time by “canning” a custom format workflow

• Commonly handled text-based sources - CSV - XML

• Build format recognition workflow once • Re-use the persisted workflow any time • Lets you develop a file format for sharing


Bruce Harold

Custom Formats

Demo


Demo Summary: Custom Formats

• Based on an existing format

• May be shared - .FDS file in profile directory for each format - C:\Users\<user>\My Documents\FME\Formats

• Editable later

Get the custom Quake format by downloading the slides!


The Power of Transformation Spatial ETL


The Full Power of Spatial ETL

• Formats are only part of the story • Often the format does not change *

- But the schema does • Transformation is key

- Many categories of transformer - Feature by Feature or Dataset by Dataset - Geometry or Attribute transformations

• Many Inputs Many Outputs is OK

*like shapefile to shapefile


Workbench Application • Graphically Design Data Flows

READ TRANSFORM WRITE


It’s a Pipeline • Readers send features into the workspace

- Geometry – X:Y:Z:M - Attributes – with a schema

• Transformers “do something’ - May operate “per feature” and pass it

immediately - May work on sets or all features together - Some offer parallel processing

• Writers send features to the destination

DATA


Some Transformers Are All You Need • This example derives Adds, Deletes and NoChanges in one step - Esri Community Maps contributor data is

processed this way


The Power of Transformation • Using Spatial ETL

• 100’s of Transformers - Categorized

• Browse • Search • Popup in Workbench • Navigate to in Workbench • Don’t see what you need?

- Ask for it!


Authoring Spatial ETL Tools Readers, Transformers, Visualizers, Writers

• Add Spatial ETL tools into a ToolBox - Right Click context menu

• Wizard to stub out your Workbench • Refine the workspace • Publish Parameters

- Inputs are automatic • Run the workspace

- In Workbench - As a GP Tool


Bruce Harold

ETL

Demo


Demo Summary: Spatial ETL

• Complements Core Functionality

• Analogous to ModelBuilder, except… - Feature by Feature granularity - Feature geometry manipulable - Non-Esri & Web data sources are OK


Wrapping Up More on use cases…


A Word on XML XML is a Document, not a Format

• XMLTemplater - Easily use an XML file to stub out XML - Populate attributes with XQuery expressions

• XQuery Extract, Update, Explode <feederObject objectID="{fme:get-attribute("objectID")}"> <feederName>{fme:get-attribute("feederName")}</feederName> <feederNo>{fme:get-attribute("feederNo")}</feederNo> <eaLoc>{fme:get-attribute("eaLoc")}</eaLoc> </feederObject>


Organizational Interaction

State

Federal

Regional / County

ETL

National GIS


ArcGIS Server and Data Interoperability 64bit

• Non-Native formats in Map Services • Publish ETL as GP services • Esri WFS services more flexible

- Non-cached streaming option - Setting in Interop Connection


New in 10.2 See the Help too

• Based on FME 2013 SP1 + - Improved performance - Improved FME Workbench user experience

• New Formats – highlights only - Salesforce - OData - RIEGL Laser Scan - SpatiaLite* - ZIP Files

• New or Updated Transformers - Many condensed (e.g. AttributeCreator)


Resources: Learn More

• How do I try it? - Request software evaluation (60 days): - http://www.esri.com/software/arcgis/extensio

ns/datainteroperability/evaluate • How do I learn it?

- New course (another coming): - http://training.esri.com/gateway/index.cfm?fa

=catalog.webCourseDetail&courseid=2571 • Who do I talk to?

- Visit the Analysis Island in the Esri Showcase or the SAFE Software booth (#1608) in the Expo.

http://www.esri.com/software/arcgis/extensions/datainteroperability/evaluate

http://www.esri.com/software/arcgis/extensions/datainteroperability/evaluate

http://training.esri.com/gateway/index.cfm?fa=catalog.webCourseDetail&courseid=2571

http://training.esri.com/gateway/index.cfm?fa=catalog.webCourseDetail&courseid=2571


Please fill out the session evaluation

Wednesday Offering ID: 1343 Thursday Offering ID: 1430

Online – www.esri.com/ucsessionsurveys Paper – pick up and put in drop box

Thank you…

Esri UC2013 . Technical Workshop . Data Interoperability - An Introduction