Traditional StatisticsTraditional Statistics

• Mean, StDev Mean, StDev (Normal Curve)(Normal Curve)

• Central TendencyCentral Tendency

• Typical Response (scalar) Typical Response (scalar)

Minimum= 5.4 ppmMinimum= 5.4 ppmMaximum= 103.0 ppmMaximum= 103.0 ppm

Mean= 22.4 ppmMean= 22.4 ppmStDEV= 15.5StDEV= 15.5

Spatial StatisticsSpatial Statistics

• Map of the Variance Map of the Variance (gradient)(gradient)

• Spatial DistributionSpatial Distribution

• NumericalNumerical Spatial Relationships Spatial Relationships

Spatial Spatial DistributionDistribution(Surface)(Surface)

Traditional GISTraditional GIS

• Points, Lines, PolygonsPoints, Lines, Polygons

• Discrete ObjectsDiscrete Objects

• Mapping and Geo-queryMapping and Geo-query

Forest Inventory Forest Inventory MapMap

Grid-based Map Analysis Grid-based Map Analysis (Spatial Analysis/Statistics)(Spatial Analysis/Statistics)

Spatial AnalysisSpatial Analysis

• Cells, Surfaces Cells, Surfaces

• Continuous Geographic SpaceContinuous Geographic Space

• ContextualContextual Spatial Relationships Spatial Relationships

Erosion Erosion PotentialPotential(Surface)(Surface)

Grid-Based Map AnalysisGrid-Based Map Analysis

Spatial analysisSpatial analysis investigates the “investigates the “contextualcontextual” relationships in mapped data…” relationships in mapped data…

ReclassifyReclassify— — reassigning map values (position; value; size, shape; contiguity) reassigning map values (position; value; size, shape; contiguity) OverlayOverlay— — map overlay (point-by-point; region-wide; map-wide)map overlay (point-by-point; region-wide; map-wide) DistanceDistance— proximity and connectivity (movement; optimal paths; visibility)— proximity and connectivity (movement; optimal paths; visibility) NeighborsNeighbors— — ”roving windows” (slope/aspect; diversity; anomaly)”roving windows” (slope/aspect; diversity; anomaly)

(Berry)

Data MiningData Mining investigates the “investigates the “numericalnumerical” relationships in mapped data…” relationships in mapped data…

DescriptiveDescriptive— — aggregate statistics (e.g., average/stdev, similarity, clustering)aggregate statistics (e.g., average/stdev, similarity, clustering) PredictivePredictive— — relationships among maps (e.g., regression)relationships among maps (e.g., regression) PrescriptivePrescriptive— — appropriate actions (e.g., optimization) appropriate actions (e.g., optimization)

Surface modelingSurface modeling maps the “maps the “spatial distributionspatial distribution” and pattern of point data…” and pattern of point data…

Map GeneralizationMap Generalization— characterizes spatial trends (e.g., titled plane)— characterizes spatial trends (e.g., titled plane) Spatial InterpolationSpatial Interpolation— deriving spatial distributions (e.g., IDW, Krig)— deriving spatial distributions (e.g., IDW, Krig) OtherOther— roving window/facets (e.g., density surface; tessellation) — roving window/facets (e.g., density surface; tessellation)

Spatial StatisticsSpatial Statistics

Point Density AnalysisPoint Density AnalysisPoint Density analysis identifies the number of customers with Point Density analysis identifies the number of customers with

a specified distance of each grid locationa specified distance of each grid location

Roving Window (count)

(Berry)

Identifying Unusually High DensityIdentifying Unusually High DensityPockets of unusually high customer density are identified as more Pockets of unusually high customer density are identified as more

than one standard deviation above the meanthan one standard deviation above the mean

(Berry)

Surface Modeling Surface Modeling (Density Surface)(Density Surface)

(Berry)(Berry)

DiscreteMap Surface

2 Hugags every 30 min for 30 days

HugagHugagCountsCounts

Hugag

Hugag Activity draped over ElevationHugag Activity draped over Elevation

ContinuousMap Surface

Most of the activity is in the NE

Hugag Density SurfaceHugag Density Surface

Avg- 17.5 StDev= 15.0

Roving WindowRoving WindowTotal number of counts within 6-cell radiusTotal number of counts within 6-cell radius

Density Surface ModelingDensity Surface Modeling

““Counts” the number of occurrences within a Counts” the number of occurrences within a specified “roving window” reach— higher values specified “roving window” reach— higher values indicate concentrations of occurrenceindicate concentrations of occurrence

……from from discretediscrete observations to observations to continuouscontinuous spatial distribution spatial distribution

(Short Exercise #6)(Short Exercise #6)

Spatial InterpolationSpatial Interpolation (Smoothing the Variability)(Smoothing the Variability)

The “The “iterative smoothingiterative smoothing” process is similar to slapping a big chunk of ” process is similar to slapping a big chunk of modeler’s clay over the “data spikes,” then taking a knife and cutting away modeler’s clay over the “data spikes,” then taking a knife and cutting away

the excess to leave a the excess to leave a continuous surfacecontinuous surface that encapsulates the peaks and that encapsulates the peaks and valleys implied in the original field samplesvalleys implied in the original field samples

……repeated repeated smoothing smoothing slowly “erodes” slowly “erodes” the data surface the data surface to a flat planeto a flat plane= = AVERAGEAVERAGE

(Berry)(Berry)(digital slide show SSTAT2)(digital slide show SSTAT2)

Inverse Distance Weighted ApproachInverse Distance Weighted Approach

(Berry)

Tobler’s First Law of GeographyTobler’s First Law of Geography— — nearby things are more alike than distant thingsnearby things are more alike than distant things

1/DPower

Spatial Autocorrelation Spatial Autocorrelation (Kriging)(Kriging)

Tobler’s First Law of GeographyTobler’s First Law of Geography— — nearby things are more alike than distant thingsnearby things are more alike than distant things

VariogramVariogram— — plot of sample data similarity as a function of distance between samplesplot of sample data similarity as a function of distance between samples

(Berry)

……Kriging uses regional variable theory based on an underlying variogram to develop Kriging uses regional variable theory based on an underlying variogram to develop custom weightscustom weights based on trends in the sample data (proximity and direction) based on trends in the sample data (proximity and direction)

……uses uses Variogram EquationVariogram Equation instead of a fixed instead of a fixed 1/D1/DPowerPower Geometric Equation Geometric Equation

Surface Modeling Methods Surface Modeling Methods (Surfer)(Surfer)

Inverse DistanceInverse Distance to a Power to a Power— weighted average of samples in the summary — weighted average of samples in the summary window such that the influence of a sample point declines with “simple” distancewindow such that the influence of a sample point declines with “simple” distance

Modified Shepard’s MethodModified Shepard’s Method— uses an inverse distance “least squares” method — uses an inverse distance “least squares” method that reduces the “bull’s-eye” effect around sample pointsthat reduces the “bull’s-eye” effect around sample points

Radial Basis FunctionRadial Basis Function— uses non-linear functions of “simple” distance to — uses non-linear functions of “simple” distance to determine summary weightsdetermine summary weights

KrigingKriging— summary of samples based on distance and angular trends in the data— summary of samples based on distance and angular trends in the data

Natural NeighborNatural Neighbor—weighted average of neighboring samples where the weights —weighted average of neighboring samples where the weights are proportional to the “borrowed area” from the surrounding points (based on are proportional to the “borrowed area” from the surrounding points (based on differences in Thiessen polygon sets)differences in Thiessen polygon sets)

Minimum CurvatureMinimum Curvature— — analogous to fitting a thin, elastic plate through each analogous to fitting a thin, elastic plate through each sample point using a minimum amount of bending sample point using a minimum amount of bending ((Spatial InterpolationSpatial Interpolation) )

Nearest NeighborNearest Neighbor— assigns the value of the nearest sample point— assigns the value of the nearest sample point

TriangulationTriangulation— identifies the “optimal” set of triangles connecting — identifies the “optimal” set of triangles connecting all of the sample points all of the sample points ((Geometric FacetsGeometric Facets) )

Polynomial RegressionPolynomial Regression— — fits an equation to the entire set fits an equation to the entire set of sample points of sample points ((Map GeneralizationMap Generalization)) Thiessen PolygonsThiessen Polygons

(Berry)(Berry)

Geometric facetsGeometric facets

Map GeneralizationMap Generalization

Spatial InterpolationSpatial Interpolation

Spatial InterpolationSpatial Interpolation

……all interpolation algorithms assume that… all interpolation algorithms assume that…

1) “1) “nearby things are more alike than distant thingsnearby things are more alike than distant things” (spatial autocorrelation), ” (spatial autocorrelation), 2) 2) appropriate sampling intensity appropriate sampling intensity (ample number of samples), and a (ample number of samples), and a 3) 3) suitable sampling patternsuitable sampling pattern

… …the interpolated surfaces “the interpolated surfaces “map the spatial variationmap the spatial variation” in the data samples” in the data samples

Spatial Spatial Interpolation is Interpolation is similar to similar to throwing a throwing a blanket over the blanket over the “data spikes” to “data spikes” to conforming to the conforming to the geographic geographic pattern of the pattern of the data.data.

(Berry)(Berry)

Comparing Spatial Interpolation ResultsComparing Spatial Interpolation Results

Comparison of the IDW Comparison of the IDW interpolated surface to the interpolated surface to the whole field average shows whole field average shows LARGE differencesLARGE differences in in localized estimates localized estimates

(Berry)(Berry)

Comparison of the IDW Comparison of the IDW and Krig interpolated and Krig interpolated surfaces shows surfaces shows small small differencesdifferences in in localized in in localized estimatesestimates

Surface Modeling Surface Modeling (Full Exercise #6)(Full Exercise #6)

(Berry)(Berry)

Use Use SurferSurfer to interpolate a continuous surface… to interpolate a continuous surface…

……and generate contour and solid surface plots and generate contour and solid surface plots

Spatial InterpolationSpatial Interpolation

Use Use MapCalcMapCalc to create a density surface (total count) to create a density surface (total count)

Density Surface Derivation Density Surface Derivation (Use(Use MapCalc MapCalc to derive a customer density surface) to derive a customer density surface)

SCANSCAN Total_Customers TOTAL WITHIN 6 FOR Customer_density6 Total_Customers TOTAL WITHIN 6 FOR Customer_density6 RENUMBERRENUMBER Customer_density6 ASSIGNING 0 TO 0 THRU 33.7 Customer_density6 ASSIGNING 0 TO 0 THRU 33.7 ASSIGNING 1 TO 33.7 THRU 1000 FOR Customer_highDensity ASSIGNING 1 TO 33.7 THRU 1000 FOR Customer_highDensity

Spatial Interpolation TechniquesSpatial Interpolation Techniques

((BerryBerry))

Characterizes the spatial distribution by fitting a mathematical Characterizes the spatial distribution by fitting a mathematical equation to localized portions of the data (roving window)equation to localized portions of the data (roving window)

AVG= 23 everywhere

Spatial Interpolation techniques use “roving windows” to summarize sample values within a specified reach of each map location. Window shape/size and summary technique result in different interpolation surfaces for a given set of field data

…no single techniques is best for all data.

Inverse Distance Weighted (IDW) technique weights the samples such that values farther away contribute less to the average

…1/Distance Power

AVG= 23

Spatial InterpolationSpatial Interpolation (Evaluating performance)(Evaluating performance)

(Berry)(Berry)Assessing Interpolation Results (Residual Analysis)Assessing Interpolation Results (Residual Analysis)

……the best map is the the best map is the one that has the “one that has the “bestbestguessesguesses””

Spatial Interpolation Spatial Interpolation (Spatially characterizing error)(Spatially characterizing error)

A Map of Error (Residual Map)A Map of Error (Residual Map)……shows you shows you wherewhere your estimates are likely good/bad your estimates are likely good/bad

(Berry)

Spatial Dependency Spatial Dependency (Spatial Autocorrelation & Correlation)(Spatial Autocorrelation & Correlation)

• the conditions of that variable at nearby locations, termed the conditions of that variable at nearby locations, termed Spatial AutocorrelationSpatial Autocorrelation (intra-variable dependence for (intra-variable dependence for Surface ModelingSurface Modeling))

(Berry)(Berry)

Spatial Variable DependenceSpatial Variable Dependence — what occurs at a location — what occurs at a location in geographic space is related to:in geographic space is related to:

• the conditions of other variables at the conditions of other variables at that location, termed that location, termed Spatial CorrelationSpatial Correlation (inter-variable dependence for (inter-variable dependence for Spatial Data MiningSpatial Data Mining))

……understanding relationships understanding relationships withinwithin a single map layer a single map layer

Basis for… Basis for…

Surface Surface ModelingModeling

……understanding relationships understanding relationships amongamong map layers map layers

Basis for… Basis for…

Spatial DataSpatial DataMiningMining

Grid-Based Map AnalysisGrid-Based Map Analysis

Data MiningData Mining investigates the “investigates the “numericalnumerical” relationships in mapped data…” relationships in mapped data…

DescriptiveDescriptive— — aggregate statistics (e.g., average/stdev, similarity, clustering)aggregate statistics (e.g., average/stdev, similarity, clustering) PredictivePredictive— — relationships among maps (e.g., regression)relationships among maps (e.g., regression) PrescriptivePrescriptive— — appropriate actions (e.g., optimization) appropriate actions (e.g., optimization)

(Berry)

Spatial analysisSpatial analysis investigates the “investigates the “contextualcontextual” relationships in mapped data…” relationships in mapped data…

ReclassifyReclassify— — reassigning map values (position; value; size, shape; contiguity) reassigning map values (position; value; size, shape; contiguity) OverlayOverlay— — map overlay (point-by-point; region-wide; map-wide)map overlay (point-by-point; region-wide; map-wide) DistanceDistance— proximity and connectivity (movement; optimal paths; visibility)— proximity and connectivity (movement; optimal paths; visibility) NeighborsNeighbors— — ”roving windows” (slope/aspect; diversity; anomaly)”roving windows” (slope/aspect; diversity; anomaly)

Surface modelingSurface modeling maps the “maps the “spatial distributionspatial distribution” and pattern of point data…” and pattern of point data…

Map GeneralizationMap Generalization— characterizes spatial trends (e.g., titled plane)— characterizes spatial trends (e.g., titled plane) Spatial InterpolationSpatial Interpolation— deriving spatial distributions (e.g., IDW, Krig)— deriving spatial distributions (e.g., IDW, Krig) OtherOther— roving window/facets (e.g., density surface; tessellation) — roving window/facets (e.g., density surface; tessellation)

Spatial StatisticsSpatial Statistics

Visualizing Spatial RelationshipsVisualizing Spatial Relationships

(Berry)(Berry)

What spatial What spatial relationships do you relationships do you see?see?

Interpolated Spatial DistributionInterpolated Spatial Distribution

Phosphorous (P)

……do relatively high levels do relatively high levels of P often occur with high of P often occur with high levels of K and N?levels of K and N?

……how often?how often?

……where?where?

Identifying Unusually High MeasurementsIdentifying Unusually High Measurements……isolate areas with mean + 1 StDev (tail of normal curve)isolate areas with mean + 1 StDev (tail of normal curve)

(Berry)(Berry)

Level SlicingLevel Slicing……simply multiply the two maps to identify joint coincidence simply multiply the two maps to identify joint coincidence

1*1=1 coincidence (any 0 results in zero)1*1=1 coincidence (any 0 results in zero)

(Berry)(Berry)

Multivariate Data SpaceMultivariate Data Space……sum of a binary progression (1, 2 ,4 8, 16, etc.) provides sum of a binary progression (1, 2 ,4 8, 16, etc.) provides

level slice solutions for many map layerslevel slice solutions for many map layers

(Berry)(Berry)

Calculating Data DistanceCalculating Data Distance……an n-dimensional plot depicts the multivariate distribution—an n-dimensional plot depicts the multivariate distribution—

the the distance between pointsdistance between points determines the relative similarity in data patterns determines the relative similarity in data patterns

……the closest floating ball is the the closest floating ball is the least similarleast similar ( (largest data distancelargest data distance) from the comparison point) from the comparison point(Berry)(Berry)

Identifying Map SimilarityIdentifying Map Similarity

(Berry)(Berry)

……the green tones indicate field locations with fairly similar P, K and N levels; red tones indicate dissimilar areas the green tones indicate field locations with fairly similar P, K and N levels; red tones indicate dissimilar areas

……the the relative data distancerelative data distance between the comparison point’s data pattern between the comparison point’s data pattern and those of all other map locations form a and those of all other map locations form a Similarity IndexSimilarity Index

Clustering Maps for Data ZonesClustering Maps for Data Zones

……groups of “floating balls” in data space groups of “floating balls” in data space identify locations in the field with similar data identify locations in the field with similar data patterns– patterns– data zonesdata zones

…a map stack is a spatially organized set of numbers

(Berry)

……fertilization rates vary for the fertilization rates vary for the different clusters “different clusters “on-the-flyon-the-fly””

(Cyber-Farmer, Circa 1990)(Cyber-Farmer, Circa 1990)

Variable Rate ApplicationVariable Rate Application

Evaluating Clustering ResultsEvaluating Clustering Results

(Berry)(Berry)

……graphical and statistics procedures graphical and statistics procedures assess how “distinct” clusters areassess how “distinct” clusters are——

Clustering PerformanceClustering Performance

……distinct in distinct in K and N K and N

(less), but (less), but not distinct not distinct

in Pin P

Spatial Data Mining Spatial Data Mining (Full Exercise #7)(Full Exercise #7)

(Berry)(Berry)

Similarity MapSimilarity Map

Cluster MapCluster Map

Regional AverageRegional Average

CompositeComposite

Scatter Plot Scatter Plot Univariate RegressionUnivariate Regression

Multivariate RegressionMultivariate Regression

Spatial statistics …Spatial statistics …use use MapCalcMapCalc to implement derive to implement derive

relationships among P, K, N and relationships among P, K, N and Yield in a farmer’s field Yield in a farmer’s field

(Short Exercise #7)(Short Exercise #7)

DescriptiveDescriptive

PrescriptivePrescriptive

The Precision Ag ProcessThe Precision Ag Process (Fertility example)(Fertility example)

As a combine moves through a field it As a combine moves through a field it 1)1) uses GPS to check its location then uses GPS to check its location then 2)2) checks the yield at that location to checks the yield at that location to 3)3) create a continuous map of the create a continuous map of the yield variation every few feet. This map isyield variation every few feet. This map is 4)4) combined with soil, terrain and other maps to combined with soil, terrain and other maps to derive derive 5)5) a “Prescription Map” that is used to a “Prescription Map” that is used to 6)6) adjust fertilization levels every few feet adjust fertilization levels every few feet in the field (variable rate application).in the field (variable rate application).

Farm dBFarm dBStep 4)Step 4)

Map AnalysisMap Analysis

On-the-Fly On-the-Fly Yield MapYield Map

Steps 1) – 3)Steps 1) – 3)

Prescription MapPrescription Map

Step 5)Step 5)

Zone 1

Zone 3

Zone 2

Step 6)Step 6)

Variable Rate ApplicationVariable Rate Application

Cyber-Farmer, Circa 1992Cyber-Farmer, Circa 1992

(Berry)(Berry)

Spatial Data MiningSpatial Data MiningPrecision Farming is just one example of applying spatial

statistics and data mining techniques

(Berry)(Berry)

Mapped data that Mapped data that exhibits high exhibits high spatial spatial dependencydependency create create strong prediction strong prediction functions. As in functions. As in traditional statistical traditional statistical analysis, spatial analysis, spatial relationships can be relationships can be used to predict used to predict outcomes…outcomes…

……the difference is the difference is that spatial statisticsthat spatial statisticspredicts predicts wherewhere responses will be responses will be high or lowhigh or lowGeo-business SDM

ContinuousSpatial Distribution

(Detailed)

Map AnalysisMap Analysis

Spatially Spatially Interpolated data Interpolated data

((Geographic Space — Geographic Space — Spatial Statistics)Spatial Statistics)

Data Analysis Data Analysis PerspectivesPerspectives (Review)(Review)

Identifies the Identifies the Central TendencyCentral Tendency Maps the Maps the VarianceVariance

Central TendencyCentral Tendency

Average = 22.0Average = 22.0

StDev = 18.7StDev = 18.7TypicalTypical

How TypicalHow Typical

DiscreteDiscreteSpatial ObjectSpatial Object

(Generalized)(Generalized)22.0 28.2

Traditional AnalysisTraditional Analysis

Field DataField DataStandard Normal Curve Standard Normal Curve

fit to the data fit to the data

((Data Space Data Space — Non-spatial Statistics)— Non-spatial Statistics)

(Berry)(Berry)

Grid-Based Map Analysis Grid-Based Map Analysis (Review)(Review)

Data MiningData Mining investigates the “investigates the “numericalnumerical” relationships in mapped data…” relationships in mapped data…

DescriptiveDescriptive— — aggregate statistics (e.g., average/stdev, similarity)aggregate statistics (e.g., average/stdev, similarity) PredictivePredictive— — relationships among maps (e.g., regression)relationships among maps (e.g., regression) PrescriptivePrescriptive— — appropriate actions (e.g., optimization) appropriate actions (e.g., optimization)

Surface ModelingSurface Modeling maps the “maps the “spatial distributionspatial distribution” and pattern of point data…” and pattern of point data…

Map GeneralizationMap Generalization— — characterizes spatial trends (e.g., titled plane)characterizes spatial trends (e.g., titled plane) Spatial InterpolationSpatial Interpolation— — deriving spatial distribution (e.g., IDW, Krig)deriving spatial distribution (e.g., IDW, Krig) OtherOther— — roving window (e.g., density surface; tessellation) roving window (e.g., density surface; tessellation)

Spatial AnalysisSpatial Analysis investigates the “investigates the “contextualcontextual” relationships in mapped data…” relationships in mapped data…

ReclassifyReclassify— — reassigning map values (position; value; size, shape; contiguity) reassigning map values (position; value; size, shape; contiguity) OverlayOverlay— — map overlay (point-by-point; region-wide; map-wide)map overlay (point-by-point; region-wide; map-wide) DistanceDistance— proximity and connectivity (movement; optimal paths; visibility)— proximity and connectivity (movement; optimal paths; visibility) NeighborsNeighbors— — ”roving windows” (slope/aspect; diversity; anomaly)”roving windows” (slope/aspect; diversity; anomaly)

(Berry)(Berry)

Spatial AnalysisSpatial Analysis

Spatial StatisticsSpatial Statistics

More Map Analysis ExperienceMore Map Analysis Experience ((MapCalc & Surfer)MapCalc & Surfer)

(Berry)(Berry)

NEW BOOKNEW BOOK — — seesee the description of the the description of the Map AnalysisMap Analysis book (Berry, 2007; book (Berry, 2007;

GeoTec Media)GeoTec Media) at…at… www.innovativegis.com/basis www.innovativegis.com/basis

…develops a structured view of the important concepts, considerations and procedures involved in grid-based map analysis.

…the companion CD contains further readings and software for hands-on experience with the material presented.

Complete ExperienceComplete Experience

See See Default.htmDefault.htm

Workshop CDWorkshop CDSurfer Tutorial

MapCalc Tutorial

Tutorial ExercisesTutorial Exercises Workshop ExercisesWorkshop Exercises

Short & FullShort & FullExercise SetsExercise Sets

……but before we leave but before we leave Spatial Statistics Spatial Statistics (Surface Modeling and Spatial (Surface Modeling and Spatial Data Mining—descriptive, predictive and prescriptive)Data Mining—descriptive, predictive and prescriptive) to tackle to tackle GIS GIS

ModelingModeling, any…, any…

Questions?Questions?

Questions?Questions?

(Berry)(Berry)