ArcPy Overview – Extending and Automating ArcGIS with Python

Clinton Dow – Geoprocessing Product Engineer @ Esri

http://www.slideshare.net/ClintonDow/arcpy-overview-slides

Before we begin…Free stuff!

• Free ArcGIS Pro- http://bit.ly/ArcPyProTrial

• Free ArcGIS Online Account- http://bit.ly/PythonAGOLTrial

• ArcPy Documentation- http://bit.ly/ArcPyDocs

• ArcPy Resources- http://bit.ly/ArcPyResources

About MeClinton Dow – Calgary, AB > Los Angeles, CA

• Product Engineer @ Esri- ArcPy- Conda- Charts and Graphs

• GIS Developer @ Matrix Solutions Inc.- GeoProcessing Tool Development

- Oil & Gas Applications- Environmental Applications- Civil Engineering Projects

esriEnvironmental Systems Research Institute, Inc.

• Founded in 1969• HQ in Redlands California

- 41 Offices Globally- Employees from 67 Countries

• Supports Government, Military and Civilian organizations• Currently hiring Experienced Python Developers/Product Engineers - http://bit.ly/ArcPyJobs

ArcGISProduct Information

• Industry-leading Commercial GIS Software Suite• ArcGIS for Desktop

- Initial release in 1999- Current version of ArcMap 10.4- ArcGIS Pro included since 10.3 – January 2015

- Current version 1.2

• ArcGIS for Server- Current version 10.4- Portal for ArcGIS included since 10.2

• ArcGIS Online

ArcPyArcGIS Python site-package

• Introduced in ArcGIS 10.0- June 29, 2010

• Successor to arcgisscripting module• Create tools leveraging ArcGIS

- In-Application Geoprocessing Tools and Scripting- REST Web Services- Standalone Python Programs

• 1,000+ ArcPy-based Tools Included in ArcGIS

Python at EsriOther Uses of Python

• Test Framework- Proprietary test-runner and reporting

• Queries in ArcGIS- Tools not based on ArcPY

- Calculate Field Tool- Map Label Expressions

• CityEngine - Python Scripting- Built-in script editor

Agenda

• Python in ArcGIS: Moving Forward- Python 2 vs. 3- ArcGIS 10.x vs ArcGIS Pro

• Tip #1 –ArcPy in an IDE• Automating ArcGIS with Python

- Scripting Complex Workflows- Service Endpoint Automation

Agenda• Tip #2 – There must be a better way!• Extending ArcGIS with Python

- Python Standard Library- Tour of Complementary Python Packages

• Tip #3 – Comprehensions with Cursors• The future of Python in ArcGIS

- Managing Packages and Environments with Conda- Jupyter Notebook

ArcPy: Moving Forward‘Python is the language of GIS.’ – Bill Moreland, Esri Dev Summit 2016

• Python 2 vs. 3 in ArcGIS- Python 2 End of Support Countdown Clock!- ArcGIS 10.4 Python 2.7.10- ArcGIS for Server Python 2.7.10

- ArcGIS Pro 1.2 Python 3.4.3- ArcGIS Pro 1.3+ Decoupled Python!

ArcPy: Moving Forward‘Python is the language of GIS.’ – Bill Moreland, Esri Dev Summit 2016

•ArcGIS 10.x to ArcGIS Pro- ArcObjects SDK will not move to Pro- Great time to be a Python GIS Developer!- ArcObjects high cost of maintenance and very tightly

coupled to COM- Rewrite code in Python 3 in Pro

- Platform agnostic vs. .NET code- Web Services- Vast library of Packages- Improved features in Pro- … it’s Python and not .NET

Tip #1 – ArcPy in an IDEGIS from the comfort of your development environment

• Integrated Development Environments- Debuggers- Test Frameworks- Coverage- PyLint/pycodestyle- Version Control Integration- Refactoring Tools- Virtual Environment Support- AutoComplete

Tip #1 – ArcPy in an IDEGIS from the comfort of your development environment

• Recommended IDEs- PyCharm- Python Tools for Visual

Studio- Spyder- Eclipse with PyDev- Wingware

Automating ArcGIS with PythonToolboxes, Python Toolboxes, Interpreters, Command Line, Services…

• Multiple ways to create tools- ArcGIS Toolbox (.tbx)- Python Toolbox (.pyt)- Stand-alone Scripts (.py)- Interactive Python Window

• Decouple logic code from Toolbox- Testing!!

• Distutils to create packaged codeSharingInstall into tools as needed

Automating ArcGIS with PythonModularized Code Demo

Automating ArcGIS with PythonDistutils demo

Extending ArcGIS with Python ModulesPython is a ‘Batteries Included’ Programming Language

• Python Standard Library- Distutils- Collections- Itertools- XML- JSON- UnitTest- -asyncio

Tip #2 – There Must Be A Better Way!Never Underestimate The Python Standard Library!

• Raymond Hettinger- Transforming Code Into Beautiful, Idiomatic Python

- https://www.youtube.com/watch?v=OSGv2VnC0go- Beyond PEP 8 – Best Practices for Beautiful, Intelligible Code

- https://www.youtube.com/watch?v=wf-BqAjZb8M

Extending ArcGIS with Python ModulesUtilize the Extended Python Ecosystem in your ArcGIS Tools

• Scientific- Jupyter- IPython- NumPy- SciPy- Pandas- PySAL- SciKit Learn

• Utility- Requests- Arrow- Xlrd/xlwt- Chardet

• Visualization- Bokeh- Plotly

• Web-Scraping- BeautifulSoup- Scrapy- Tweepy

Tip #3 – Filter & Map with ComprehensionsUsing Pure Python Data Types to Work with Feature Classes

import arcpy

data_path = r"C:\Path\to\feature\class"

def cursor_comprehension_map(path): # Map function # m = map(lambda x: x[0] + 1.0, [row for row in arcpy.da.SearchCursor(path, ['OID@'])])

m = [row[0] + 1 for row in arcpy.da.SearchCursor(path, ['OID@'])] return m

def cursor_comprehension_filter(path): # Filter function # f = filter(lambda x: x ==1, [row for row in arcpy.da.SearchCursor(path, ['OID@'])])

f = [row for row in arcpy.da.SearchCursor(path, ['OID@']) if row[0] == 1] return f

Tip #3 – Comprehensions with CursorsUsing Pure Python Data Types to Work with Feature Classes

import arcpy

data_path = r"C:\Path\to\feature\class"

def cursor_list_comprehension(path): l = [row for row in arcpy.da.SearchCursor(path, ["SHAPE@", "Language"])] return l

def cursor_dictionary_comprehension(path): d = {row[0]: row[1:] for row in arcpy.da.SearchCursor(path, ["OID@", "MsgText", "Language"])} return d

def cursor_generator_comprehension(path): g = (row for row in arcpy.da.SearchCursor(path, ["SHAPE@", "Language"])) return g

def cursor_generator_statement(path): with arcpy.da.SearchCursor(path, ["OID@", "MsgText"]) as cursor: for row in cursor: yield str(row[0])

Tip #3 – Collections ExampleUsing Pure Python Data Types to Work with Feature Classes

from collections import Counterimport arcpy

data_path = r"C:\Users\clin8331\Desktop\Temp\tw.gdb\tw_lang_europe"

def count_field_occurances(data_path): c = Counter() with arcpy.da.SearchCursor(data_path, ["OID@", "Language"]) as cursor: for row in cursor: c[row[1]] += 1 return c

The Future of Python in ArcGISContinuum Analytics’ conda VirtualEnv & Package Manager

• Conda integration into ArcGIS- Advanced Python Package Management

- Curated channels of scientific Python packages- Includes C and Fortran compiler - Create and host your own conda packages- Easily share your code with the GIS community

- Built-In Virtual Environment Support- Manage multiple environments with access to ArcPy- Easily share environments- Trivializes dependency issues

The Future of Python in ArcGISScreenshots of conda UI

The Future of Python in ArcGISJupyter Notebook/IPython Integration

• Interactive Scientific Computing with ArcGIS Support- Interact with ArcGIS for Server and ArcGIS Online via Jupyter Notebooks- ArcPy Integration coming- Embed maps with simple, pythonic API- Access scalable, web-based GeoProcessing Services

- GeoEnrichment Services- GeoAnalytics Servers

- All the regular functionality of notebooks- Pandas/matplotlib magic methods- Github integration- Easily Sharable

Help us make ArcPy better!

• GeoNet- http://geonet.esri.com- https://geonet.esri.com/community/developers/gis-developers/python

• ArcGIS Ideas- http://ideas.arcgis.com

• Conda Support Email- conda@esri.com

• ArcPy Café- https://arcpy.wordpress.com/

Conclusion

• ArcGIS- Industry-leading GIS Platform- Big push to move towards Python- Really big customers (NASA, ESA, Governments, Fortune 500 companies…)- Lots of opportunity for Python programmers

• ArcPy- Moving from Python 2 to 3- Conda support- Start in ArcGIS Pro if you’re learning

• We are hiring! Talk to me if you are interested in learning more.

Thank You!Clinton Dow - Contact Info

cdow@esri.comhttp://lnked.in/cdow

http://www.slideshare.net/ClintonDow/arcpy-overview-slides