Data Basin ArcGIS Geoprocessing Tools … Basin ArcGIS Geoprocessing Tools Documentation, Release 0.2 These geoprocessing tools were created to help support spatial analysis and other

Data Basin ArcGIS GeoprocessingTools Documentation

Release 0.2

Conservation Biology Institute

July 22, 2014

Contents

1 Tabulate Tool 31.1 Available Summary Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.2 Inputs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.3 Outputs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51.4 Error Handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2 Messaging 112.1 messaging.py . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

3 Utilities 133.1 PathUtils.py . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133.2 ProjectionUtilities.py . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133.3 FeatureSetConverter.py . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

4 Requirements 17

5 Installation - ArcGIS 10.2.x 195.1 Pre-packaged versions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195.2 Packaging and deploying yourself . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

6 Testing 21

7 Known Limitations 23

8 Authors 25

9 License 27

Python Module Index 29

i

ii

Data Basin ArcGIS Geoprocessing Tools Documentation, Release 0.2

These geoprocessing tools were created to help support spatial analysis and other operations in Data Basin.

Source code available in Bitbucket.

These tools are intended to be deployed as geoprocessing services on an ArcGIS Server that also hosts map services(their data must be hosted locally). Within Data Basin, these services are accessed via server middleware to route toand control the number of jobs per ArcGIS server. Custom-built client code (HTML/JS/CSS) is used to provide theuser interface to these tools, and manages generation of input parameters, and display of output results.

However, most of the internal functions only rely on arcpy and access to the input data, and thus could be executedfrom other environments.

Refer to the specific tools below for more information:

Contents 1

http://databasin.org/

https://bitbucket.org/databasin/databasin_arcgis_geoprocessing_tools/


2 Contents

CHAPTER 1

Tabulate Tool

This tool tabulates various summary values for feature or raster datasets within an area of interest. The area of interestcan be represented as one or more points, lines, or polygons (limited to one type of geometry per analysis).

This tool creates a custom Albers Equal Area (WGS84 datum) projection centered over the area of interest to use asthe standard throughout processing; however, the native projection of the target raster dataset will be used if it is avalid projection for calculating areas, such as Albers Equal-Area, Lambert Azimuthal Equal-Area, or UTM.

For raster analysis, the tool uses one of two methods:

1) approximate: the area of interest is converted to a raster dataset with the same resolution as the target raster (pixelcalculations are not based on partial pixels); thus it is necessary to compare the area of interest in pixels against thesummary area returned for the target raster. This method is used when the area of interest is represented by points orthe number of pixels in the extent of the area of interest is higher than optimal for precise method (>50,000 pixels).

2) precise: the raster is extracted to the extent of the area of interest in its native projection, and then a fishnet featureclass is created that matches it. This fishnet is then intersected with the area of interest, and proportional areas ofoverlap area calculated as weights for each pixel. These weights can then used for either area weighted statistics.

1.1 Available Summary Methods

Feature layers

• area or length and count of features inside area of interest

• area or length of features that intersected area of interest (the total area or length of the original feature bothinside and outside area of interest)

• area or length and count of features by unique attribute values

• area or length and count of features by classes of a continuous attribute

• statistics of a continuous attribute inside area of interest: MIN, MAX, SUM, MEAN

Note: MEAN is always weighted by the polygon area, line length, or point count of the target featureswithin the area of interest.

Raster layers

• area and pixel count of area of interest in resolution of target raster

• area and pixel count of raster inside area of interest

• area and pixel count of unique values of a raster or raster attribute inside area of interest

• area and pixel count of classes of a continuous raster or raster attribute inside area of interest

3


• statistics of raster or continuous attribute inside area of interest. Valid statistics are: MIN, MAX, SUM, MEAN,STD (standard deviation).

Note: If the precise method is used, MEAN is weighted by the proportion of each pixel occupied bythe area of interest if it is a polygon, or by the proportional length if area of interset is a line.

1.2 Inputs

featureSetJSON: Area of interest represented as an ArcGIS FeatureSet in JSON format:

{"fields": [{"alias": "OBJECTID", "type": "esriFieldTypeOID", "name": "OBJECTID"}],"geometryType": "esriGeometryPolygon","features": [

{"geometry": {

"rings": [[

[-12510743.8804,3962356.0276999995],[-12500772.095800001,3955536.6137000024],[-12509264.1962,3945822.1655000001],[-12510936.8827,3944921.4880999997],[-12513381.578299999,3946015.1677000001],[-12517112.955699999,3957466.636500001],[-12514925.5965,3960040.0002999976],[-12510743.8804,3962356.0276999995]

]]

},"attributes": {"OBJECTID": 3}

}],"spatialReference": {"wkid": 102100,"latestWkid": 3857}

}

configJSON: The list of map services, layers, and summary methods:

{"services":[{"serviceID":"test","layers":[

{"layerID":0},{"layerID":0,"attributes":[{"attribute":"NAME"}]},{"layerID":0,"attributes":[{"attribute":"POP2000", "statistics":["MIN","MAX"]}]},{"layerID":2,"attributes":[{"attribute":"POP2000","classes":[[0,1000],[1000,10000],[10000,1000000]]}]},{"layerID":3},{"layerID":5},{"layerID":5,"classes":[[0,300],[300,310],[310,400]]},{"layerID":5,"statistics":["MIN","MAX","MEAN","SUM"]}

]}]}

For each map service, provide the serviceID (from the map service URL, this is /ar-cgis/rest/services/<serviceID>/MapServer), and the layer configuration.

For each layer, provide the layerID (this can be determined from looking at the list of layers for the map servicein ArcGIS REST API). If no other parameters are given for layer, only the total area or length and count offeatures inside area of interest, and total area or length and count of features intersecting the area of interest willbe returned.

4 Chapter 1. Tabulate Tool


Feature layers:

• To summarize by unique values of an attribute, simply include that attribute in the list of attributes:

{"layerID":0, "attributes":[{"attribute":"NAME"}]}

• To summarize by classes of an attribute, include the attribute and list of class value ranges (greater than orequal to first value, and less than second value):

{"layerID":0, "attributes":[{"attribute":"NAME", "classes":[ [0,10], [10,20], [20,30] ]}]}

• To return summary statistics of an attribute, list the desired statistics:

{"layerID":0,"attributes":[{"attribute":"POP2000","statistics":["MIN","MAX","MEAN","SUM"] }]}

Note: statistics option is mutually exclusive of above options

Raster layers:

• Categorical rasters will be summarized by unique value if no additional parameters are provided, continu-ous ones will not:

{"layerID":3}

• To summarize by classes of the raster, simply include class ranges at layer level:

{"layerID":5, "classes":[ [0,300],[300,310],[310,400] ]}

• To return summary statistics of raster, simply include statistics at layer level:

{"layerID":5, "statistics":["MIN","MAX","MEAN","SUM","STD"]}

• Attribute-level summaries are same as above

1.3 Outputs

During execution, the tool will add a progress message for each completed layer and service. The format is:PROGRESS [PERCENT_COMPLETE]

resultsJSON: JSON results follow similar format as configJSON above.

Key concepts:

• Very little is returned if no intersection is found. Generally only count properties will be returned in thiscase.

• Areas and lengths are returned using the general “intersectionQuantity” properties. Use the geometryTypeproperties to determine what units these represent. Quantities will not be returned for points.

• An important distinction is made between intersected and intersection results for features:

Intersection: the portion of the features WITHIN the area of interest. This will be in the units ofthe intersection.

Intersected: the original features that intersected the area of interest, INCLUDING the area oflength inside and outside the area of interest. This will be in the units of the original intersectedfeatures. This is useful for calculating the percentage of the original features that are within thearea of interest.

Results for examples above:

1.3. Outputs 5


{"area_units": "hectares", #area values are always in hectares"linear_units": "kilometers", #linear values are always in kilometers"sourceGeometryType": "polygon", #point, line, or polygon"services": [{"serviceID": "test",

"layers": [{

#a point feature layer"layerID": 0,"intersectionGeometryType": "point", #will be point, line, polygon, or pixel (raster)"intersectedCount": 2, #number of features that INTERSECTED area of interest"intersectedGeometryType": "point","intersectionCount": 2 #number of featues WITHIN area of interest

},{

"layerID": 0,"intersectedGeometryType": "point","intersectedCount": 2,"attributes": [

{#a categorical attribute"attribute": "NAME","values": [

{"intersectedCount": 1,"intersectionCount": 1,"value": "Avondale"},{"intersectedCount": 1,"intersectionCount": 1,"value": "Goodyear"}

]}

],"intersectionGeometryType": "point","intersectionCount": 2

},{

"layerID": 0,"intersectedGeometryType": "point","intersectedCount": 2,"attributes": [

{#a continuous attribute"attribute": "POP2000","statistics": {

"MAX": 35883,"MIN": 18911

}}

],"intersectionGeometryType": "point","intersectionCount": 2

},{

#a polygon feature layer"layerID": 2,"intersectionGeometryType": "polygon","intersectedGeometryType": "polygon",#quantities are hectares for polygon geometry type, kilometers for line, and not present for point"intersectionQuantity": 3774.3558016523793,"intersectedQuantity": 7670.2729527175416,"intersectedCount": 1,"attributes": [



{#a continuous attribute"attribute": "POP2000","classes": [

{"class": [0,1000],"intersectedQuantity": 0,"intersectedCount": 0,"intersectionQuantity": 0,"intersectionCount": 0

},{

"class": [1000,10000],"intersectedQuantity": 0,"intersectedCount": 0,"intersectionQuantity": 0,"intersectionCount": 0

},{

"class": [10000,1000000],"intersectedQuantity": 7670.2729527175416,"intersectedCount": 1,"intersectionQuantity": 3774.3558016523793,"intersectionCount": 1

}]

}],"intersectionCount": 1

},{

#a categorical raster, will be summarized on unique values"layerID": 3,"method": "approximate",#approximate: area of interest represented as a grid, no area weighting. precise: area of#interest is a polygon representation of grid, with area weighting."intersectionCount": 124796,"sourcePixelCount": 124796,"intersectionQuantity": 11231.639999999999,"pixelArea": 0.089999999999999997,"geometryType": "pixel","values": [

{"value": 1,"intersectionCount": 24090,"intersectionQuantity": 2168.0999999999999

},{

"value": 2,"intersectionCount": 38736,"intersectionQuantity": 3486.2399999999998

},{


},{

1.3. Outputs 7



},{


}]

},{

#a continuous raster, will only be summarized for intersection area"layerID": 5,"pixelArea": 0.089999999999999997,"geometryType": "pixel","method": "approximate","sourcePixelCount": 124796,"intersectionQuantity": 11231.820000000002,"intersectionCount": 124798

},{

"layerID": 5,"pixelArea": 0.089999999999999997, #area in hectares"classes": [

{"class": [0,300],"intersectionCount": 67863,"intersectionQuantity": 6107.6700000000001

},{

"class": [300,310],"intersectionCount": 38677,"intersectionQuantity": 3480.9299999999998

},{

"class": [310,400],"intersectionCount": 18256,"intersectionQuantity": 1643.04

}],"geometryType": "pixel","method": "approximate","sourcePixelCount": 124796,"intersectionQuantity": 11231.820000000002,"intersectionCount": 124798

},{

"layerID": 5,"pixelArea": 0.089999999999999997,"statistics": {

"STD": 11.514897346496582,"MAX": 378.656494140625,"SUM": 37146864.0,"MIN": 271.205322265625,"MEAN": 297.65594482421875

},"geometryType": "pixel",



"sourcePixelCount": 124798,"intersectionQuantity": 11231.820000000002,"method": "approximate"

}]

}],"sourceFeatureQuantity": 11231.81217300969, #area or length of area interest, if polygon or line"sourceFeatureCount": 1

}

1.4 Error Handling

This tool will almost always return successfully, because it is trapping and returning errors if encountered for eachservice and layer. These will be include the python stacktrace of the error to assist debugging, unless the error is aninput or data error that the tool is specifically trying to handle. Additional information may be present in the logs toindicate the problem.

Example error:

{’sourceFeatureQuantity’: 11231.925988334813,’sourceGeometryType’: ’polygon’,’sourceFeatureCount’: 1,’services’: [

{’layers’: [{’error’: ’GCS_NOT_SUPPORTED: Geographic Transformation to WGS84 not found for projection with GCS: GCS_North_American_1927’,’layerID’: 9

}],’serviceID’: u’arcgis_geoprocessing_tools_test_data’}

],’linear_units’: ’kilometers’,’area_units’: ’hectares’

}

Supporting modules:

1.4. Error Handling 9



CHAPTER 2

Messaging

2.1 messaging.py

class messaging.MessageHandler(messages, logger=None)Convenience class for handling messaging.

Includes a concept of progress, which is a measure of number of completed major and minor steps.

For example, when processing a series of map services and their layers, the number of map services is thenumber of major steps, and the number of layers is the number of minor steps (represents loop-within-loophierarchy).

setMajorSteps(major_steps)Set the number of major steps to measure progress against.

Parameters major_steps – number of major steps of operation

setMinorSteps(minor_steps)OPTIONAL: Set the number of minor steps to measure progress against.

Parameters minor_steps – number of major steps of operation

incrementMajorStep()Increment the current major step by one, and emit a new progress message.

incrementMinorStep()Increment the current minor step by one, and emit a new progress message.

setProgress(progress)Emit a new progress message: PROGRESS [PERCENT_COMPLETE]

Parameters progress – the current progress, on a percent scale.

addMessage(message)Emit a new message to both messages and logger (if available)

Parameters message – the message to emit

11


12 Chapter 2. Messaging

CHAPTER 3

Utilities

A collection of utility functions and

3.1 PathUtils.py

General path utilities for workspaces and routing to map documents behind map services

utilities.PathUtils.extractLayerPathFromMSDLayerXML(msd, xmlPath)Extracts layer data source from layer XML files stored in MSD.

Parameters

• msd – MSD file opened via ZipFile

• xmlPath – path to XML file with layerInfo

Returns list of layer paths, or None for each group layer; index in this list = layerID

utilities.PathUtils.getDataPathsForService(serviceID)Extract paths for data layers in map service. Returns cached lookup if possible.

Parameters serviceID –

Returns return list of layers paths (or None for group layers); order in this list = layerID

3.2 ProjectionUtilities.py

General utilities for helping deal with projection related information

utilities.ProjectionUtilities.getGCS(spatialReference)Return the geographic coordinate system name for the spatial reference (e.g., GCS_North_American_1983).

Parameters spatialReference – ArcGIS spatial reference object

utilities.ProjectionUtilities.getProjUnitFactors(spatialReference)Return the conversion factors for projection length to kilometers, projection area to hectares, and meters perprojection unit for the spatial reference. Used to multiply the length and area calculated for features.

Parameters spatialReference – ArcGIS spatial reference object

Note: Limited to projections based on Meter and Foot_US

13


utilities.ProjectionUtilities.getWGS84GeoTransform(gcs)Find a general geographic transformation from one coordinate system to WGS 1984. These transformations arebased on the continent scale transformations listed by ESRI, and are not exact for local calculations. Only thefollowing source geographic coordinate systems are currently supported:

•NAD 1983

•NAD 1983 HARN

•NAD 1983 CRS

•SAD 1969

•ED 1950

Parameters gcs – the input geographic coordinate system name extracted from the source projectionwell-known text

Note: Target geographic coordinate system is always WGS 1984

utilities.ProjectionUtilities.getGeoTransform(srcSR, targetSR)Return the geographic transformation required to project between two projections (passing through WGS 1984,if required), or empty string if not required.

Parameters

• srcSR – source ArcGIS spatial reference object

• targetSR – target ArcGIS spatial reference object

Note: limited to the geographic coordinate systems supported by getWGS84GeoTransform

utilities.ProjectionUtilities.projectExtent(extent, srcSR, targetSR)Project the extent to the target spatial reference, and return the projected extent. Creates a temporary featureclass based on bounding box of source.

Parameters

• extent – source extent

• srcSR – source ArcGIS spatial reference object

• targetSR – target ArcGIS spatial reference object

utilities.ProjectionUtilities.createCustomAlbers(extent)Given an extent in geographic coordinates, create a custom Albers projection centered over the extent thatminimizes area distortions. Uses 1/6 inset from YMin and YMax to define latitude bounds, and centerlinebetween XMin and XMax to define central meridian.

Parameters extent – extent in geographic coordinates

Returns custom Albers spatial reference

utilities.ProjectionUtilities.getSpatialReferenceFromWKID(WKID)Returns a spatial reference object for WKID

Parameters WKID – ESRI Well Known ID

Returns spatial reference object

utilities.ProjectionUtilities.isValidAreaProjection(spatialReference)Determines if projection is valid for area calculations.

14 Chapter 3. Utilities


Parameters spatialReference – spatial reference object

Returns True if valid for area projections, False otherwise

3.3 FeatureSetConverter.py

Converts a featureset in JSON into a feature class in memory

utilities.FeatureSetConverter.getFeatureGeometry(geomType, geometry)Extract geometry from featureset JSON, and convert into geometry representation required for feature class.

Parameters

• geomType – ArcGIS JSON geometry type: esriGeometryPoint, esriGeometryMultipoint,esriGeometryPolyline, esriGeometryPolygon

• geometry – the geometry object extracted from JSON

utilities.FeatureSetConverter.createFeatureClass(featureSet, name=’drawingFC’)Create an in-memory feature class from a featureset JSON.

Parameters

• featureSet – the featureset JSON string.

• name – name of output feature class (always in memory)

Note: the original feature IDs (FID / OBJECTID) are not preserved in feature class, as they are built up freshduring construction of feature class.

3.3. FeatureSetConverter.py 15


16 Chapter 3. Utilities

CHAPTER 4

Requirements

• ArcGIS Server 10.2.x, including spatial analyst extension

• Published map services containing data to be operated against

17


18 Chapter 4. Requirements

CHAPTER 5

Installation - ArcGIS 10.2.x

Note: This tool must be deployed to the root folder, with the name “databasin_arcgis_geoprocessing_tools” toproperly work with Data Basin.

5.1 Pre-packaged versions

These easiest way to install this tool is to deploy the latest service definition file from the downloads page

Simply download, and then upload to your ArcGIS server. This version comes with a pointer back to the sourceMercurial repository - develop branch (see below).

5.2 Packaging and deploying yourself

Use the ArcGIS server command line tool ags_tool_deploy to help manage the deployment process. Install per theinstructions in that repository.

Download the latest development snapshot from develop branch or the latest stable version from master branch

Then, from a command within the directory containing tools.pyt:

<python_packages_dir>/arcgis_tool_deploy/deploy.py publish tools.pyt databasin_arcgis_geoprocessing_tools <hostname> <admin_username> --password=<admin_password>

use --overwrite if you want to delete and replace and existing service of the same name.

Managing with Mercurial:

Note: This assumes Mercurial is installed on your server.

Given the active development and bugfixes on this tool, and the challenges in deploying to ArcGIS server, you canalso clone this repository to your local machine using mercurial, and include basic repository information when youdeploy the tool to ArcGIS server. This allows you to pull new updates directly to the ArcGIS server instead of havingto redeploy the tool.

Use the --hg option above to include Mercurial repository information.

Then from within the installed location on the ArcGIS server reported using the publish command above, simply runhg pull --update to update to the latest changes in the branch you used above (make sure you are on developfor the latest changes or master for the latest stable changes.

Once you have pulled and updated to the latest changes, simply restart the geoprocessing service.

19

https://bitbucket.org/databasin/databasin_arcgis_geoprocessing_tools/downloads/

https://bitbucket.org/databasin/ags_tool_deploy/

https://bitbucket.org/databasin/databasin_arcgis_geoprocessing_tools/get/develop.zip/

https://bitbucket.org/databasin/databasin_arcgis_geoprocessing_tools/get/master.zip/


20 Chapter 5. Installation - ArcGIS 10.2.x

CHAPTER 6

Testing

Because this set of tools is built to run on ArcGIS server against running map services, it is necessary to execute thetests in the same environment.

First, deploy this tool and make sure it is properly installed.

Second, deploy the test data from the downloads page as a map service calledarcgis_geoprocessing_tools_test_data

Next, execute the tool at http://<server_hostname>/arcgis/rest/services/databasin_arcgis_geoprocessing_tools/GPServer/test_tabulate

This will run the test suite as if it were a stand-alone geoprocessing tool. It will run through a variety of tests.

21

https://bitbucket.org/databasin/databasin_arcgis_geoprocessing_tools/downloads/


22 Chapter 6. Testing

CHAPTER 7

Known Limitations

• Tabulate tool is very slow due primarily to the implementation of the projection function in ArcGIS (arcpy).Work is underway to refactor out as many steps where tool is reprojecting data as is possible.

• A limited range of spatial projections are supported for target calculations and source map services, due to theArcGIS requirement of including a geographic transformation to project between many different projections.Currently only continent-scale geographic transformations are included with these tools. Additional transfor-mations can be added to utilities/ProjectionUtilities.py

• Tool execution times vary with the number of layers, complexity of geometries, and extent of analysis. Expectanalysis of several complex layers over larger areas to be slower.

• Path routing to ArcGIS layer data sources will not work for enterprise geodatabases.

23


24 Chapter 7. Known Limitations

CHAPTER 8

Authors

Conservation Biology Institute developers:

• Mike Gough

• Brendan Ward

Contact: databasinadmin at consbio dot org

25

http://consbio.org/

http://consbio.org/people/staff/mike-gough

http://consbio.org/people/staff/brendan-ward


26 Chapter 8. Authors

CHAPTER 9

License

Copyright (c) 2014, Conservation Biology Institute

All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that thefollowing conditions are met:

• Redistributions of source code must retain the above copyright notice, this list of conditions and the followingdisclaimer.

• Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the follow-ing disclaimer in the documentation and/or other materials provided with the distribution.

• Neither the name of the Conservation Biology Institute nor the names of its contributors may be used to endorseor promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANYEXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIESOF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENTSHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, IN-CIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITEDTO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSI-NESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CON-TRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANYWAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAM-AGE.

27


28 Chapter 9. License

Python Module Index

mmessaging, 11

uutilities.FeatureSetConverter, 15utilities.PathUtils, 13utilities.ProjectionUtilities, 13

29

Documents

Data Basin ArcGIS Geoprocessing Tools … Basin ArcGIS Geoprocessing Tools Documentation, Release 0.2 These geoprocessing tools were created to help support spatial analysis and other