Chapter 8
Spatial analysis and geoprocessing
Instructor
Chapter objectives
• Understand the web-based analysis tools provided in ArcGIS
• Know the collection of variables available from ArcGIS Online
• Understand the basics of ArcGIS Insights
• Create web apps that use ArcGIS Online analysis
• Author and share web tools with ArcGIS Enterprise
• Use web tools in web apps
• Understand the workflow for big data analysis
• Understand ArcGIS Notebooks
Getting to Know Web GIS, fourth edition
Data sources Desktop tools ArcGIS Online / ArcGIS Enterprise Client apps
CSV files Feature layers Ready-to-use apps and
configurable templates
Shapefiles Raster tile layers
Web Configurable app templates
KMLs Vector tile layers maps/
web ArcGIS StoryMaps
File
File geodatabases
geodatabases Scene layers
scenes Experience
Experience Builder/Web
Builder/WebAppBuilder
AppBuilder
Enterprise geodatabases ArcMap Map image layers
ArcGIS Dashboards
Nonspatial data ArcGIS Pro Image layers ArcGIS Insights
Tools ArcGIS Feeds ArcGIS Collector, Survey123,
Photos/Imagery Drone2Map Stream layers Explorer, QuickCapture, Tracker,
Navigator, and Workforce …
Raster function templates ArcGIS Custom web
web tools
tools
ArcGIS AppStudio
Sensor and real-time data CityEngine Standard
Standard tools
tools ArcGIS Earth
Big
Big data
data GeoAnalytics
GeoAnalytics tools/big
tools/big ArcGIS Indoors viewer
data analytic
analytics
… ArcGIS VR 360
Real-time analytics
AuGeo
360 VR Experience
…
Geometry services
Custom apps
Living Atlas
Atlas of
of the
theWorld
World
ArcGIS API for JavaScript
… ArcGIS Runtime SDKs
ArcGIS
ArcGIS API
API for
forPython
Python
… Getting to Know Web GIS, fourth edition
Web GIS is more than mapping
Understanding and Determining how places Finding the best locations
measuring places are related
Detecting and quantifying Making predictions Directions, routing,
patterns and logistics
Getting to Know Web GIS, fourth edition
Spatial analysis and spatial data science
• Spatial analysis holds great practical value for our personal life and for enterprise operations
• Spatial data science allows analysts to extract deeper insight from data using a comprehensive
set of analytical methods and spatial algorithms, including machine learning and deep learning
techniques
• Web-based spatial analysis allows more users to discover relationships, patterns, and trends in
geospatial data
Getting to Know Web GIS, fourth edition
ArcGIS provides web tools and clients for spatial analysis
Web tools Clients
For standard-sized ArcGIS Map Viewer
Standard tools
data ArcGIS Web AppBuilder
Feature ArcGIS Experience Builder
analysis Insights
For big data in vector
Geoanalytics tools
format ArcGIS Pro
Ready-to-use ArcGIS Notebooks
web tools
Raster ArcGIS API for Python
For raster data
analysis ArcPy
ArcGIS API for JavaScript
…
Custom Publish toolboxes or Python to ArcGIS
web tools Enterprise as geoprocessing services
Custom analysis
ArcGIS Your python code run in an isolated
Notebooks environment
Getting to Know Web GIS, fourth edition
Compare ArcGIS Enterprise and ArcGIS Online analysis
ArcGIS Enterprise
ArcGIS Online
analysis tools
Ready-to-use tools Custom web tools
Created and hosted by Esri. Comes with software install. Hosted on your To be created and hosted by organizations
Source of tools Ready to use infrastructure. Ready to use
Low. Easy to use Relatively high. Requires knowledge of
Difficulty level ModelBuilder, Python, or other scripts to
create services
Both non-professionals and professionals Professionals who provide web tools or
Target users services
Yes (you can use your own No (you can use the data from ArcGIS Online)
Data provided? data too)
Yes No, unless you are referencing ArcGIS Online No
Cost credits tools and premium content
Costs organizations to purchase, install and
host
Publishers, administrators, and custom accounts with certain privileges Administrators, and publishers (if enabled to
System requirements publish geoprocessing tools)
Getting to Know Web GIS, fourth edition
Standard analysis tools provided by ArcGIS Online and ArcGIS Enterprise
Getting to Know Web GIS, fourth edition
Living Atlas of the World supports ArcGIS Online analysis
Getting to Know Web GIS, fourth edition
Workflow to use ArcGIS Online analysis
Review and
Prepare data Add to map Perform analysis
interpret results
The first two steps are optional if you use only the data provided by ArcGIS Online
Getting to Know Web GIS, fourth edition
Big data
• Datasets that are so large and complex that
traditional data-processing application
software is inadequate to deal with them
Large scale Different forms
of data of data
Volume Variety
Big
data
Velocity Veracity
High speed Uncertainty
of data flow of data http://www.washingtonpost.com/wp-dyn/content/article/2011/02/10
/AR2011021004916.html
Getting to Know Web GIS, fourth edition
Geospatial big data
• Smart sensors
- Electrical meters (AMI), SCADA, UAVs
• GPS telemetry
- Vehicle tracking, smartphone data collectors, workforce tracking, geofencing
• Internet data
- Social media streams, web log files, customer sentiment
• Sensor data
- Weather sensors, stream gauge measurements, heavy equipment monitors
• Imagery
- Satellites, frame cameras, drones
Getting to Know Web GIS, fourth edition
ArcGIS GeoAnalytics Server: For big data (vector) analysis
Too big to fit in one computer and to be processed by one
computer
Big
data ArcGIS GeoAnalytics Server leverages distributed analytics and storage
Getting to Know Web GIS, fourth edition
Big data analysis tools provided by ArcGIS Enterprise
Getting to Know Web GIS, fourth edition
Use GeoAnalytics Server to analyze big data
Register and edit
Prepare data Perform analysis Review results
big data file share
Getting to Know Web GIS, fourth edition
Perform big data analysis
For browsing and using
custom web tools
For standard analysis For big data analysis
Getting to Know Web GIS, fourth edition
New York City taxi pickup locations (170 million) aggregation
1 3 5
Bin time series
Now
Time step
Bin
Then
2 4 6 Getting to Know Web GIS, fourth edition
Custom web tools and geoprocessing services
• Geoprocessing services expose the analytic capability of ArcGIS to the web
Toolbox GP service
Results
Publish
Tools
(Models, GP tasks
Python)
Run inside Run inside
ArcMap or ArcGIS Pro ArcGIS Server
Getting to Know Web GIS, fourth edition
Steps to create custom web tools
Author your tool
Run your tool in Share the result
using ModelBuilder
ArcGIS Pro (execution plan)
or Python
• Python & ArcPy
Execution mode
• ModelBuilder • Synchronous
• Asynchronous
Getting to Know Web GIS, fourth edition
ModelBuilder
• ModelBuilder
- Available in ArcMap and ArcGIS Pro
- A visual programming language for building GP workflows
- Can create, edit, and manage models
- Easier than Python and other scripting languages
• Model
- Created with ModelBuilder
- A diagram that chains together sequences of GP tools and connects them with inputs
and outputs
Getting to Know Web GIS, fourth edition
Python and ArcPy
• Another way to create GP tools
• Python is a free, powerful, cross-platform, and open-source scripting language
• ArcGIS extends Python by providing ArcPy, a module that facilitates GIS data analysis, data
conversion, data management, and map automation
• Python and ArcPy are included in ArcMap, ArcGIS Pro, and ArcGIS Enterprise install
Getting to Know Web GIS, fourth edition
Synchronous and asynchronous execution modes
Request
Synchronous
Server
•
Client
- Client waits for the server to finish executing and then gets the result Result
- Typically for fast services
Request
Job id
• Asynchronous Status?
- Client submits a request, and checks server for status repeatedly processing
Server
Client
- Client gets the result once the server finishes the job Status?
- Typically for services that tale longer time to run finished
Request result
Result
• .
Getting to Know Web GIS, fourth edition
Share custom web tool from ArcGIS Pro to ArcGIS Enterprise
Portal for
ArcGIS
to ol
Web
Geopr
ocessi
ng ser
ArcGIS Pro v ice ArcGIS
Server
renced
Data refe
or
Data copie
d
ArcGIS
Data Store
Getting to Know Web GIS, fourth edition
For browsing and
Access ArcGIS web tools using custom web
tools
For standard analysis For big data analysis
• ArcGIS Map Viewer
• ArcGIS Web AppBuilder
• ArcGIS Experience Builder
• Insights
• ArcGIS Pro
• ArcGIS Notebooks
• ArcGIS API for Python
• ArcPy
• ArcGIS API for JavaScript
• … ArcGIS
Web AppBuilder
Analysis Widget
Getting to Know Web GIS, fourth edition
Perform analysis in ArcGIS Insights
• An analytics workbench that allows you to perform iterative and exploratory data analysis
• Drag-and-drop driven
• Visualize and analyze data in maps, tables, and charts
o Bubble, histogram, scatter, time-series graph, and treemap
Spatial analytics o Bar, line, donut, stacked, and combo charts
ArcGIS Insights
Business intelligence • Analysis and visualization at the same time
• Stimulates questions and rapid iterations of analysis
• Interact with more than one map or chart at a time
• Record steps of analysis
www.esri.com/en-us/arcgis/products/insights-for-arcgis/overview
Getting to Know Web GIS, fourth edition
ArcGIS Insights
ArcGIS Insights Gallery
Getting to Know Web GIS, fourth edition
Perform analysis in ArcGIS Notebooks
• Create and share notebooks that contain live Python code and visualizations
• Provides a Jupyter notebook experience with which data scientists can combine ArcGIS spatial analysis
algorithms with open-source Python libraries to build precise spatial data science models
ArcGIS
ArcGIS Notebooks • Jupyter
• pandas
• ArcGIS API for Python Open Source • Conda
• ArcPy Python & Data Science • Keras
• Analytics • Scipy
• Geoprocessing • PyTorch
• Web GIS analysis • NumPy
• GeoAnalytics • SymPy
• Raster Analytics
Getting to Know Web GIS, fourth edition
Perform analysis in ArcGIS Notebooks
• Include hundreds of Python libraries
- ArcGIS Python API: a Python library for performing GIS visualization and analysis, spatial data management, and GIS
system administration tasks in Web GIS (https://developers.arcgis.com/python/guide/)
- ArcPy
- Common open source data science libraries
• Conveys analysis results with intuitive, interactive maps and apps for data storytelling that drives insight and
action
• Increases cross-team collaboration. You can save your notebooks in ArcGIS Online and ArcGIS Enterprise and
can share them with other users the same way you share other items. Other users can open and run the
analysis or workflow implemented in your notebook
• Accessible anywhere with a desktop or a mobile web browser
Getting to Know Web GIS, fourth edition
Artificial intelligence, machine learning, and deep learning
Artificial Intelligence Popular research areas:
• Computer vision
• Natural language processing
• Speech recognition
• Video game behavioral AI
Machine • ...
Learning
Deep
Learning
Convolutional
Neural Networks
Getting to Know Web GIS, fourth edition
Machine learning in ArcGIS
Classification Prediction
Deciding to which category Using the known to
an object should be assigned estimate the unknown
Integration with
Clustering
Grouping observations based on ArcGIS external frameworks
similarities of values or locations
Getting to Know Web GIS, fourth edition
Richness of Imperiled Species in the United States
Created using
ArcGIS machine
learning analysis
Getting to Know Web GIS, fourth edition
Deep learning in ArcGIS
ArcGIS Python API provides arcgis.learn model for deep learning
Case study: ArcGIS Python API provides deep-learning tools that can
effectively identify road cracks from dashcam videos
• Much cheaper than visual inspection or inspection using sophisticated
instruments
• ArcGIS Python API arcgis.learn module was trained with photos that have
various types of cracks labeled
• The trained tool then detected road cracks on images and in video feeds
• The analysis results included the locations, types, and severities of road
cracks and helped authorities to plan and prioritize timely repairs
Getting to Know Web GIS, fourth edition
Deep learning in ArcGIS
Integration with third-party libraries
Case study: Georgia Power combined ArcGIS Python API with
IBM Watson machine-learning API to automatically detect
damaged insulators and locate them on the map
• Human manual inspection would be a huge and tedious task
• George Power flew drones to take photos along the transmission lines
• Trained IBM Watson machine-learning with sample photos that had known
damaged insulators
• Used the trained tools to automatically detect damaged insulators from the
hundreds of thousands of photos
• ArcGIS Python API saved the locations of the photos to a feature layer for
visualization to guide field repairs
Getting to Know Web GIS, fourth edition
Tutorial demos
8.1-8.2: Restaurant site selection using ArcGIS Online analysis
8.3-8.5: Factory site selection by publishing and using custom web tools
8.6: New York taxicab drop-off and pick-up big data analysis using
ArcGIS GeoAnalytics Server
8.7: Crime hot spot analysis using ArcGIS Notebooks
Getting to Know Web GIS, fourth edition
Summary
• Web-based spatial analysis holds great practical value for our personal life and for enterprise
operations
• ArcGIS provides standard analysis tools, big data analysis tools, raster data analysis tools, and the
capability for users to publish custom web tools
• You can author tools using ModelBuilder and Python in ArcMap or ArcGIS Pro, run the tools, and
then publish the results (the execution plan) to ArcGIS Enterprise to create geoprocessing services
or custom web tools
• ArcGIS provides tools for machine learning, including tools for classification, clustering, predication,
and the capability to integrate other data science packages
• ArcGIS Python API arcgis.learn module can perform deep learning analysis, including computer
vision
• ArcGIS Notebooks provides an insolated environment with rich pre-installed python libraries for
users to create analysis notebooks, which can be saved, shared, and rerun
Getting to Know Web GIS, fourth edition
Assignment 7.1: Use ArcGIS Online analysis to select your new home
Select one from 7.1 and 7.2
You are in the market to look for a new home. You are considering many factors, such as the distance
between your home and work or your children’s schools, neighborhood income, median house value,
median family income, unemployment rate, and crime rate
Create an app to help you narrow the area in which you would like to buy your new home
What to submit:
- The URL of the web app you created using Web AppBuilder
- Screenshots of your analysis processes and your final choice of neighborhood areas
Getting to Know Web GIS, fourth edition
Assignment 7.2: Create a web app to clip, zip, and ship GIS data
Select one from 7.1 and 7.2
A survey and mapping bureau would like to improve its data-sharing workflow. Instead of extracting and
copying data manually to serve its customers, the bureau wants to automate the workflow with a web
app, which should allow its users to:
- Select the layers they need
- Draw the area of interest
- Select the desired data format
- Have the data clipped and zipped for them to download
• What to submit:
- The URL or REST endpoint of your web tool
- The URL to your web app
Getting to Know Web GIS, fourth edition
Discussion
Getting to Know Web GIS, fourth edition
