Introduction to arcpy Debugging Using Arcpy" (70 characters)

arcpy Dr Andy Evans

This lecture Introduction to arcpy Debugging Using arcpy

Application programming One use of Python is as an extension language; that is, one to script other software. Others include: Visual Basic for Applications (VBA); Lua. Generally extension languages work in two ways: as scripts or as GUI elements that have event listening functions overridden. Usually access is provided to an application object, representing the running code, and then this is drilled into using the dot operator: application.getDocument().getActiveView().getLayers()[0]

arcpy Access to the ESRI applications is through the arcpy library, which needs importing. Inline with Python's "It just works" philosophy, arcpy does some of the drilling down for you: Layer a = application.getDocument().getActiveView().getLayers().getLayer(0) is: with arcpy.mapping as m: a = m.listLayers(MapDocument("CURRENT"))[0]

arcpy Contains several modules depending on the extensions installed: GUI element support (pythonaddins) Geoprocessing etc. (arcpy) Data access module (arcpy.da) Mapping module (arcpy.mapping) (arcpy.mpmodule in ArcGIS Pro) ArcGIS Spatial Analyst extension module (arcpy.sa) ArcGIS Network Analyst extension module (arcpy.na) Also comes with a suitable numpy (though it only uses a specific numpy format called structured arrays, where columns have names and a memory structure).

Python 2.7 ArcDesktop/ArcMap uses Python 2.7 ArcGIS Pro uses 3.5 (including conda libraries). If you use Anaconda in the labs, there is a 2.7 (27) version of Spyder, but it won't work as it doesn't know about the arcpy libraries. We'll see how to use 2.7 in the practicals.

Python 2.7 Changes were mainly to make Python 3 more restrictive, so reasonably simple to go from 3 to 2 carry on writing 3-style for the majority. Standard libraries may have slight changes. External libraries may not exist. Major issues: except NameError as err: except NameError, err: # 2 # 3 "/" gives integer division in 2 (i.e. results in an integer if both numbers are integers). Use "//" if you want this in Python 3, which also works in 2. String formatting only with % and print("{}, {}".format(x, y)) * iterable unpacking operator doesn't work in 2.

Python 2.7 Try not to pick up any bad habits you might see in Python 2: print 'Hello World' xrange() raise Error, "message" generator.next() <> (use parentheses) (use range, which now does this job, unless efficiency key) (use raise Error("message") instead) (use next(generator) instead) (use != instead; it is the same from 2.6 onwards) Watch also: That input()in 2 reads numbers (and other types) directly, not strings (dangerous for security). That kwarg argument positioning is more flexible in 2. Loose use of variables: the scoping rules were much more flexible in 2, especially with loops. There's a maximum size for ints in 2 (found with sys.maxint); for very large ints, use long in 2.

ArcGIS Pro Uses Python 3. arcpy.mapping --> arcpy.mpmodule Uses conda for library installs. Can have scheduled jobs in Arc. Introduction: https://pro.arcgis.com/en/pro-app/arcpy/get-started/installing-python- for-arcgis-pro.htm

Help Starting point for ArcMap programming is: http://desktop.arcgis.com/en/arcmap/latest/analyze/python/ You can change the documentation depending on which version you have. NB: don't confuse this with the ArcGIS Pro documentation, at: https://pro.arcgis.com/en/pro-app/arcpy/

API Complete lists of classes, functions: http://desktop.arcgis.com/en/arcmap/latest/analyze/arcpy/what-is-arcpy-.htm Complete list of geoprocessing tools: http://desktop.arcgis.com/en/arcmap/latest/tools/main/a-quick-tour-of- geoprocessing-tool-references.htm Help docs for tools now come with Python examples, e.g.: http://desktop.arcgis.com/en/arcmap/10.3/tools/analysis-toolbox/buffer.htm

Running Python Python Window: single commands Script tools: Python Toolbox: Python addins: External scripts: multiple commands with option for a parameter GUI ArcToolbox built from Python with special access GUI elements with special access standard Python using arcpy As we'll see in the practicals, it is sometimes useful to have these working together.

Writing Python Generally write in notepad++. You can use Spyder, but you won't be able to run it, and the debugging messages appear in Arc. Set editor for scripts (right-click -> edit) in Geoprocessing menu -> Geoprocessing Options. Documenting scripts: right-click tool/box -> Item Description -> Edit button.

Scheduling a task http://desktop.arcgis.com/en/arcmap/latest/analyze/python/scheduling-a- python-script-to-run-at-prescribed-times.htm As it happens, this is useful for all Python.

Start import arcpy arcpy.env.workspace = "c:/data/myDirectory" arcpy.env.workspace = "c:/data/myGeoDatabase.gdb" Then don't need to give paths for data in that directory/database. By default this is %USER%/ArcGIS/Default.gdb And stays this even for saved maps.

Paths Decide on relative or absolute paths when adding scripts. If you put relative, Arc will adjust some paths when Arc opens: The script path Default paths Files referenced in tool metadata and help Layer files (.lyr) used for the symbology property Compiled help files (.chm) Style sheets However, you cannot use relative paths yourself inside scripts that use ".." (going up directories is fine, just not down) http://desktop.arcgis.com/en/arcmap/latest/tools/supplement/pathname s-explained-absolute-relative-unc-and-url.htm

Finding the current directory For external files, if you want to find the current directory (so you can do stuff relative to that, or set the workspace) the __file__ hidden variable is set for all Python files as the current script running. So: filename = os.path.abspath(__file__) directory_name = os.path.dirname(__file__) new_file_path = os.path.join(os.path.dirname(__file__), 'new.txt') new_file_path = os.path.join(os.path.dirname(__file__), r'data\new.txt') Note use of "r" to use raw string rather than interpreting "\" as escape.

Start Any issues importing libraries you think are installed, see: http://desktop.arcgis.com/en/arcmap/latest/analyze/python/importing- arcpy.htm Note that Arc has many classes etc. with the same names, so don't use: from someLibrary import *

Stuff in Arc ESRI distinguish between: Built in functions Classes and objects Object functions Geoprocessing tools Though the latter are just functions within modules and most stuff is accessed through arcpy, e.g. a = arcpy.Exists("c:/data/roads.shp") Note that ESRI generally don't follow community practice in having functions start with lowercase letters and using snake_case (using PascalCase instead).

Getting at stuff Either drill down to it through ArcObjects. (later in course) Or use a tool that accesses it directly. arcpy.AddField_management("c:/data/myGeodatabase.gdb/roads ", "ROAD_NUMBER", "TEXT")

Distribution Scripts can be distributed with toolboxes in the same directory using relative paths. Addins are distributed as zip files with associated resources, as we'll see. Python toolboxes have a complex distribution setup, but are probably the best way to tie tools and toolboxes: http://desktop.arcgis.com/en/arcmap/latest/analyze/python/extending- geoprocessing-through-python-modules.htm the same setup process can be used with standard custom toolboxes. Internationalisation: http://desktop.arcgis.com/en/arcmap/latest/analyze/python/international- language-support.htm

This lecture Introduction to arcpy Debugging Using arcpy

Debugging Scripts with an issue will be marked with a broken script icon. Addins with syntax errors will appear as a red stop sign. Debugging messages are often sent to the Python Window (so keep open). print() also usually writes to the Python Window. Debugging messages for geoprocessing tools will go to their dialog. Debugging information will also go to the results window (usually hidden).

Errors When something goes wrong with an arcpy component, it generates arcpy.ExecuteError exceptions. When something goes wrong with a geoprocessing tool, it generates messages - indeed, they also generate messages when working. Geoprocessing messages are subdivided into the following levels of severity: 0: general messages 1: non-critical warnings 2: critical errors In Arc, geoprocessing messages get diverted to the screen, so it is rarely necessary to deal with them separately, but in external scripts this kind of thing helps: except arcpy.ExecuteError: print(arcpy.GetMessages()) # or print(arcpy.GetMessages(2))

Messages The geoprocessing dialog appears whenever tools run, including script tools. You can write to this with: arcpy.AddMessage(messageString) arcpy.AddWarning(messageString) arcpy.AddError(messageString) arcpy.AddIDMessage(message_severity, message_ID, argument1, argument2)

Pattern for raising your own errors A common pattern used by ESRI in the docs (and quite nice) is this quick and easy way of checking your own stuff is working ok with an empty exception class: class MyException(Exception): pass try { # stuff if (condition): raise MyException("message") except MyException: # do stuff

Python traceback As arcpy error messages don't always include information about the Python itself (such as the erroneous linenumber) it can be useful to explicitly get the stacktrace using the sys and traceback libraries when something goes wrong: tb = sys.exc_info()[2] tbinfo = traceback.format_tb(tb)[0] print(tbinfo) # or arcpy.AddError(tbinfo)

Emergency treatment In the (unlikely) scenario you destroy Arc to the point it won't reboot, there are various options for resetting it. The easiest is to delete the new map template at: %APPDATA%\Roaming\ESRI\Desktop10.x\ArcMap\Templates\Normal.mxt Rename it, and Arc will rebuild it. Other options at: https://my.usgs.gov/confluence/display/EGIS/Resetting+your+ArcGIS+applicat ion+profile

Using arcpy

This lecture Introduction to arcpy Debugging Using arcpy

Arc env variables arcpy.env contains a set of variables that control overall behaviour in Arc. arcpy.env.workspace = "c:/data/myGeodatabase.gdb" From then on, this is default location for output. List all: environments = arcpy.ListEnvironments() for environment in environments: env_value = getattr(arcpy.env, environment) Reset all env settings: arcpy.ResetEnvironments() Reset an env setting: arcpy.ClearEnvironment("workspace")

Scratch space arcpy.env.scratchGDB and scratchFolder used for temp files. Set through the scratchWorkspace: arcpy.env.scratchWorkspace = 'c:/LandUse/scratch.gdb' arcpy.env.scratchWorkspace = 'c:/LandUse' If workspace is set to a GDB or folder, the other adjusts appropriately, with the default database being scratch.gdb. Can generate temp paths for use in tools, thus: temp_path = arcpy.CreateScratchName(workspace=arcpy.env.scratchGDB)

Other useful env variables http://desktop.arcgis.com/en/arcmap/latest/analyze/arcpy-classes/env.htm arcpy.env.addOutputsToMap arcpy.env.autoCommit == True | False arcpy.env.overwriteOutput == True | False arcpy.env.extent = arcpy.Extent(-107.0, 38.0, -104.0, 40.0) Some tools will only process features within the current extent. == True | False

Built in functions: Exists Checks whether something exists. a = arcpy.Exists("c:/data/buildings.shp") Note that the advantage of doing this in Arc is that it unifies multiple files (for example the various files that make up a full shapefile with data) to a single entity, and allows path-based exploration of geodatabases (so called catalog paths as opposed to system paths): a = arcpy.Exists("c:/data/myGeodatabase.gdb/roads") http://desktop.arcgis.com/en/arcmap/10.4/analyze/arcpy-functions/exists.htm

Checking inputs For scripts, the system should check input parameters exist. Otherwise you can check with: input = arcpy.GetParameterAsText(0) if arcpy.Exists(input):

Built in functions: Walk Walk(top, topdown, onerror, followlinks, datatype, type) allows scanning of a directory tree in Arc file space including geodatabases, e.g.: c:\data\MyGeoDataBase.gdb\myfeaturedataset\myfeatureclass for dirpath, dirnames, filenames in arcpy.da.Walk( workspace, topdown=True, datatype="RasterDataset"): if "back_up" in dirnames: dirnames.remove('back_up') for filename in filenames: rasters.append(os.path.join(dirpath, filename))

Built in functions: Describe Function returns a Describe object particular to the data type passed in. A bit like a uber-"type()". http://desktop.arcgis.com/en/arcmap/latest/analyze/python/describing-data.htm http://desktop.arcgis.com/en/arcmap/latest/analyze/arcpy-functions/describe.htm Can contain "Property sets" which can be drilled into further. d = arcpy.Describe("c:/data/buildings") f = d.FieldInfo print(f.count) (for this example, see also ListFields and ListIndexes)

Geoprocessing tools Essentially the tools presented in ArcToolbox. Need to find the proper name of the toolbox and tool (we'll see how in practicals). You can then do: arcpy.toolname_toolboxname(params) or arcpy.toolboxname.toolname(params) In which case it will quietly run. If you want associated parameter dialogs in an addin: pythonaddins.GPToolDialog(toolboxname, toolname) But, as we'll see in the practicals, there are some issues with this.

Geoprocessing tools Inputs in [] indicate a list should be used. Will also generally take semicolon separated strings and a ValueTable: http://desktop.arcgis.com/en/arcmap/latest/analyze/arcpy-classes/valuetable.htm

Geoprocessing tools For a custom toolbox in an external script, you need to load the toolbox to use it (as you'd have to as a user): arcpy.ImportToolbox(toolbox_path, alias_for_toolbox) The alias is a short (singleword) name for the toolbox to use in, e.g. arcpy.toolname_toolboxname(params) as the toolboxname. The alias_for_toolbox is optional if the toolbox has an alias set manually.

Using extensions If tools are in Arc extension packs (not application extension addins), you need to deal with the licensing: For example: import arcpy.sa arcpy.CheckOutExtension("spatial") # do stuff arcpy.CheckInExtension("spatial") Here we're assuming the license is ok. To check licences: if arcpy.CheckExtension("3D") == "Available": arcpy.CheckOutExtension("3D") List of extension names at: http://desktop.arcgis.com/en/arcmap/latest/analyze/arcpy-functions/checkextension.htm See also: http://desktop.arcgis.com/en/arcmap/latest/analyze/python/access-to-licensing-and- extensions.htm

Finding tools Search window in Arc (we'll see how to add tools to this in the practicals). tools = arcpy.ListTools("*") tools = arcpy.ListTools("*_conversion") Note optional wildcard * for tool in tools: print(arcpy.Usage(tool)) for toolbox in arcpy.ListToolboxes(): print(toolbox)

Optional tool parameters 1) Fill in spaces with "", "#" (including quotes), or None. 2) Use kwargs. For tools that demand an output path, missing out the output/setting it to "#" or None will usually make the tool result a system-created temp file. This saves having to determine this beforehand, but requires write access to default locations.

GUI options We've seen that scripts and models can have parameter GUIs. Addins are explicitly GUI elements, but can spawn others (filedialogs and messages, for example). You can add a tool to buttons and menus from a custom toolbox using the customisation options. The tool should be listed under "Geoprocessing". If not, you can add it manually: http://desktop.arcgis.com/en/arcmap/latest/analyze/finding-tools/adding-and- removing-tools-on-menus-and-toolbars.htm

Tool results Come back as a "Result" object. Usually path to output, or single data value or a list of data, including lists of lists for multivalue parameters. result = arcpy.Buffer_analysis("bomb", "buffer", "100 METERS") print (result) For multiples: result.outputCount result.getOutput(i) # Returns strings, recordsets or layers. result[i] # The same. result.saveToFile("c:\temp\file.rlt") http://desktop.arcgis.com/en/arcmap/latest/analyze/arcpy-classes/result.htm

Result result.status gives info: 0 : New 1 : Submitted 2 : Waiting 3 : Executing 4 : Succeeded 5 : Failed 6 : Timed out 7 : Cancelling 8 : Cancelled 9 : Deleting 10 : Deleted for i in range(result.messageCount): print (result.getMessage(i)) Last is often the most useful: print(result.GetMessage(result.GetMessageCount - 1)) Alternatively: print(result.GetMessages()) However, because there are cases where tool failure doesn't create a results object, you can also do the same with arcpy directly: print(arcpy.GetMessages())

Online services Some tools can run online services. IsSynchronous can be used to see if the tool is asynchronous (i.e. results may not be immediate and the code can continue to run). results = arcpy.Buffer_analysis(bomb, output, "100 METERS", "FULL", "ROUND", "NONE") if not arcpy.IsSynchronous("Buffer"): while results.status < 4: time.sleep(0.1) Note just toolname used.