Workflow Automation Templates for Processing Mosaic ... · Workflow Automation Templates for Processing Mosaic Dataset Imagery . ... based on image type ... • Checkboxes are used

Workflow Automation Templates for Processing Mosaic Dataset Imagery

Cody Benkelman & Jamie Drisdelle

Tuesday, March 11, 2014 5:30 pm OFFERING ID: 3019

Outline

• Mosaic Datasets • Automation with ModelBuilder & Python • Mosaic Dataset Configuration Script (MDCS)

- Configuration files - Source / Master / Derived design - Documentation and other resources

Mosaic Dataset Optimum Model for Image Data Management

• Catalog all types of raster data • Image data remains in original files • Define – in Geodatabase:

- Metadata - Processing to be applied - Default viewing rules

• Access – In all ArcGIS applications - As Image

- Dynamic Mosaic, Processed on the fly

- As Catalog - Footprints, Detailed metadata

Mosaic datasets: powerful, flexible… with many details

• GP Tools - Many have multiple options

Mosaic datasets: powerful, flexible… with many details

• GP Tools - Many have multiple options

• MD properties - Configuration depends on data

type and also use case

Automation Requirements

• Simplicity • Productivity

- Repeatability, Scalability, Maintainability - Documentation ‡ Facilitate QA & QC, Design Review

• Training/Examples - Encapsulate best practices - Reusable templates

Options for building Mosaic Datasets

• Discrete GP tools (manual) • ModelBuilder • Python (added at 10.1)

Create Simple Mosaic Dataset - ModelBuilder

Comparing Models

Python in the GP Framework

• Calling Geoprocessing tools in Python: - Import arcpy - arcpy.CreateMosaicDataset_management(“C:\\temp\test.gdb”,”abc”… - HELP LINK Using tools in Python - HELP LINK Create Mosaic Dataset

http://resources.arcgis.com/en/help/main/10.2/index.htmlhttp://resources.arcgis.com/en/help/main/10.2/index.html

Python Example - Mosaic Dataset Configuration Script (MDCS)

• Calling standard Geoprocessing tools from a single script • Input configuration file contains complete information to:

- Create, - Populate, and - Configure one mosaic dataset

- Also generates detailed log files

Advantages of MDCS

• Configuration file encapsulates “Best practices” (mosaic dataset properties) based on image type

• “Self Documenting” – - Template is reusable for different image types, or multiple mosaic datasets

within a more complex system - Compare versions (difficult with ModelBuilder)

• Automated Log files – Simple Review • Standard Geoprocessing tools have been enhanced/improved • Compatibility with upcoming ArcGIS Pro App

Other features to note within MDCS

• Can run subsets of full configuration via command line options • Built in version compatibility checks • Extensible: additional commands can be added

Configuration file contents

Input Data Paths


Input Data Paths

Raster Types & Raster Functions


Input Data Paths


Mosaic Dataset properties


Input Data Paths


GP tools necessary for the workflow



Input Data Paths


GP tools necessary for the workflow


“BEST PRACTICES”

Configuration XML

Correlating XML to MD Properties

• Geoprocessing tool interface for end users.

• Allows users to define output GDB, mosaic dataset name and input directory.

• Checkboxes are used to trigger elements of MDCS that create products.

• Allows users to output mosaic dataset product as well as referenced mosaic dataset products.

• Customizable

MDCS GP Tools

Python toolbox code

• Controls toolbox/tool labelling

• Validates parameters

• Formats parameters

• Verifies license level

• Passes parameters into MDCS

Multi-resolution scanned maps

Demo - MDCS

Advanced Image Data Management Objectives/Tasks

• Multiresolution, Multi-date image collections • Updating existing Mosaic Datasets

Managing large image collections

• Best practices implicit in the design of MDCS • Key field in configuration file “MD Type”, with allowable values:

- Source - Derived - Referenced

• For simple systems, use “Source” and stop here!...

Source Imagery

Source Mosaic

Datasets

2007

2004

2010

Source Mosaic Datasets

Source Imagery

Source Mosaic

Datasets

NED 1/3 arcsec

SRTM

LiDAR Project #N

Source Mosaic Datasets

Source Imagery

Source Mosaic

Datasets Derived

Mosaic Dataset

Combine into Derived Mosaic Dataset

Use TABLE Raster Type

Advantage: All image data available in a single location

Derived Mosaic Dataset

f

f

f

Referenced Mosaic Datasets

• Read-only copies of the parent MD • Referenced MDs are live copies, so

any changes in the parent are immediately available in the Ref MDs

• Functions applied on-the-fly; no need to write new raster to disk

Limited extent e.g. “California”

Pan sharpening

NDVI


Source Imagery

Source Mosaic

Datasets Derived

Mosaic Dataset

…many other functions

Hillshade

Ellipsoidal Height

f

f

f

Orthometric Height

Slope

f


Example – ArcGIS World Elevation

Source Imagery

Source Mosaic

Datasets Derived

Mosaic Dataset

…many other functions

Aspect Ellipsoidal

Height

Orthometric Height

Slope

f

Single image service with multiple server functions

Alternative architecture added at 10.1

Example – ArcGIS World Elevation – Server Raster Functions

Source Imagery

Source Mosaic

Datasets Derived

Mosaic Dataset

Hillshade

Ellipsoidal Height

f

f

f

Orthometric Height

Slope

f


When Derived parent is updated, Referenced Mosaic Datasets synch automatically

Example – ArcGIS World Elevation

Worldwide elevation data, with on-the-fly derived products Adding Lidar

Demo - MDCS

MDCS_UC.py class UserCode: def A(self, data): workspace = data['workspace'] md = data['mosaicdataset'] log = data['log'] log.Message('%s\\%s' % (workspace, md), 0) return True def B(self, data): log = data['log'] log.Message('hello world', 0) return True

User function (A)

User function (B)

MDCS

C:\> python.exe MDCS.py -c:CM+A+B+AR

MDCS extensibility: User Code module

Finding MDCS and related resources

• Landing page on Resource Center

Finding MDCS and related resources

• Image Management Guidebook in ArcGIS Help System

ArcGIS Online (AGOL) group

• Downloadable examples • More workflows/templates

to be added over time

Resources: “Imagery Management Workflows” • Imagery Resource Center : http://esriurl.com/6005 • Image Management Workflows: http://esriurl.com/6550 • Guidebook in ArcGIS Help: http://esriurl.com/6007 • ArcGIS Online Group: http://esriurl.com/6539 • Available soon on Github

- Cody Benkelman [email protected] - Jamie Drisdelle [email protected]

PLEASE fill out a SURVEY! Offering ID: 3019 http://esriurl.com/survey

http://esriurl.com/6005http://esriurl.com/6550http://esriurl.com/6007http://esriurl.com/6539mailto:[email protected]:[email protected]

Workflow Automation Templates for Processing Mosaic Dataset ImageryOutlineMosaic DatasetMosaic datasets: powerful, flexible… with many detailsMosaic datasets: powerful, flexible… with many detailsAutomation RequirementsOptions for building Mosaic DatasetsCreate Simple Mosaic Dataset - ModelBuilderComparing ModelsPython in the GP FrameworkPython Example - Mosaic Dataset Configuration Script (MDCS)Advantages of MDCSOther features to note within MDCSConfiguration file contentsConfiguration file contentsConfiguration file contentsConfiguration file contentsConfiguration file contentsConfiguration XMLCorrelating XML to MD PropertiesSlide Number 21Slide Number 22Demo - MDCSAdvanced Image Data Management Objectives/TasksManaging large image collectionsSource Mosaic DatasetsSource Mosaic DatasetsSlide Number 28Referenced Mosaic DatasetsExample – ArcGIS World Elevation Example – ArcGIS World Elevation – Server Raster Functions Example – ArcGIS World Elevation Demo - MDCSMDCS extensibility: User Code moduleFinding MDCS and related resourcesFinding MDCS and related resourcesArcGIS Online (AGOL) groupArcGIS Online (AGOL) groupResources: “Imagery Management Workflows”Slide Number 40

Documents

Workflow Automation Templates for Processing Mosaic ... · Workflow Automation Templates for Processing Mosaic Dataset Imagery . ... based on image type ... • Checkboxes are used