TerraStream: From Elevation Data to Watershed Hierarchies

Thursday, 08 November 2007

Andrew Danner (Swarthmore), T. Moelhave (Aarhus),

K. Yi (HKUST), P. K. Agarwal (Duke), L. Arge (Aarhus),

H. Mitasova (NCSU)

Andrew Danner 2

Current Problem: Large Point Data Sets• LIDAR

– NC Coastline: 200 million points – over 7 GB

– Neuse River basin (NC): 500 million points – over 17 GB

• Grid elevation models

– Neuse River basin:

* 20ft – 2.5 GB

* 10ft – 10GB

* 5ft – 40 GB

• Data too big for RAM

– Must reside on disk

– Disk is slow

Andrew Danner 3

• Traditional algorithms optimize CPU computation

– Not aware of performance penalty of disk access

– Virtual memory, swap space can’t predict disk access

• I/O model

– Memory is finite

– Data is transferred in blocks

– Complexity measured in disk blocks transferred

–

–

I/O-efficient Algorithms [AV88]

Disk

RAM

CPU

M

B

Andrew Danner 4

TerraStream: Terrain processing pipeline

Andrew Danner 5

TerraStream Goals

•Scalable – All stages must work for 100+ million points/cells

•General – Stages should work with either TIN or grid data

•Automated – No need for manual intervention/preprocessing

•Modular – Users only need to run the stages they want

•Adaptable – Allow each stage to support multiple models

Andrew Danner 6

TerraStream: Terrain processing pipeline

Andrew Danner 7

Points to DEM• Grid DEM

– Interpolation

– Use quad tree to automatically tile terrain

– Use quad tree neighbors for smooth boundary transitions

• TIN

– I/O efficient Delaunay triangulation

– Constrained Delaunay also possible if constraints (breaklines) fit in memory

• Height graph

− View both grids and TINs as a height graph.

− Nodes, neighbors, and edges between neighboring nodes

− Definition of node, neighbor different in TIN/Grid

− Design algorithms to work on height graphs

Andrew Danner 8

Flow Modeling

Andrew Danner 9

• Identifying minima due to noise

• Removing noise from terrains

• Modeling flow directions, extracting river networks

Flow Modeling

Andrew Danner 10

• Identifying minima likely due to noise

– Topological persistence – Computed in Sort(N) I/Os

– Assign a significance score to each minima (low score likely noise)

– Provide mechanism for removing low scoring sinks

– User can select score threshold

Coping with Noisy Data

Andrew Danner

PhD Defense

11

Noise Removal

Flooding in Sort(N) I/Os

Other Mechanisms? Carving?

Noisy terrain

After noise removal

Andrew Danner 12

From Elevation to River Networks• Where does water go?

• From higher elevation to lower elevation

• Single flow directions form a tree

• Support for multiple flow directions

Sea

80

90

10095

85

110

Andrew Danner 13

Drainage Area• How much area is upstream of each

node?

• Each node has initial drainage area (1)

• Drainage area of internal nodes depends on drainage area of children

1

1

1

1

1

1

1

1 1

1

3

3

2

2

2

3

2

5

7

17

25

14

11

9

7

Andrew Danner 14

Computing Flow Directions/Drainage• Terraflow

– Sort(N) I/Os on grids

• Modified to work on height graphs

– Same I/O bound

– Now works on TINs

• New implementation

– More robust, portable

– Incorporate new sink removal

– Better handling of flat areas…

Andrew Danner 15

Flow Modeling Improvements• Detection of flat areas

– Improved method on grids if O(1) rows fit in memory

• Routing on flat areas

– Soille extension of Garbrecht & Martz

• Flat areas usually result of hydrological conditioning with flooding

Andrew Danner 16

Hierarchical Watershed Decomposition

Andrew Danner 17

Watershed Hierarchies• Decompose a river network into a hierarchy of hydrological units

• All water in HU flows to a common outlet

• Hierarchy provides tunable level of detail

• Method used: Pfafstetter

• Want a solution scalable to large modern hi-res terrains

Andrew Danner 18

Pfafstetter• Find main river

• Find four largest tributaries

• Label basins/interbasins

• Recurse until single path1

1

1

1

1

1

1

1

1

1

2

2

2

3

2

7

17

25

14119

7

3

3

5

8

6

4

2

7

53

1

9

Andrew Danner 19

Example Watershed Boundaries

1

2

3

4

5

6

7

8

9

91

92

93

94

95

9697

98

99

991

992993

994

995

996

997

998999

All levels computed in one run. User selects

level of detail with map algebra

Andrew Danner 20

A Complete Pipeline

Andrew Danner 21

Implementation• TPIE: C++ primitives for I/O-efficient algorithms

• Standalone command line apps with GDAL

• GRASS: Open Source GIS Plugins

• ArcGIS Plugins (soon)

• Test Data:

– North Carolina LIDAR

* Neuse river basin: 400 million points (NC Floodmaps)

* Outer banks coastal data : 128 million points (NOAA CSC)

– USGS 30m NED

Andrew Danner 22

Our Results• Experimental Results

– Scales to over 400 million points

– Other software tools crash at 25 million points

– Keeps memory usage low using I/O efficient methods

Format 20ft grid 10ft grid TINHG vertices (millions) 397 1590 469Pipeline stage Dem Construction 19h 56m 27h 12m 4h 20m Building height graph 0h 07m 0h 30m 11h 42mHydro. Conditioning 1h 17m 7h 25m 10h 03mFlow Modeling Routing 1h 26m 6h 34m 15h 08m Accumulation 1h 40m 7h 35m 2h 05mWatershed Delineation 2h 28m 14h 39m 6h 26mTotal 25h 54m 63h 34m 49h 44m

Andrew Danner 23

Future Directions – Grid Construction• Interpolate leaves in parallel

• Test other interpolation methods

• Test with more data sources

• Finding the “ideal” resolution

Andrew Danner 24

Future Directions – Noise Removal• Bridge detection/removal

• Hydrological conditioning with carving

• Scoring of sinks based on volume

• Other flow routing methods

• Further flat routing improvements

Andrew Danner 25

Flow Routing and Bridges

Use flooded terrain for connectivity but

Use original terrain for routing

Andrew Danner 26

Questions?