RPI GeoStar Group 1

Parallel ODETLAP for Parallel ODETLAP for Terrain Compression Terrain Compression and Reconstructionand Reconstruction

Jared StookeyJared Stookey

Zhongyi XieZhongyi Xie

W. Randolph FranklinW. Randolph Franklin

Dan TracyDan Tracy

Barbara CutlerBarbara Cutler

Marcus V. A. AndradeMarcus V. A. Andrade

2RPI GeoStar Group

OutlineOutline

Quick Overview of our researchQuick Overview of our research ODETLAP (Non-Patch)ODETLAP (Non-Patch) Motivation for ParallelizationMotivation for Parallelization Our ApproachOur Approach MPI ImplementationMPI Implementation ResultsResults Current and Future WorkCurrent and Future Work

3RPI GeoStar Group

Quick OverviewQuick Overview

Our researchOur research Terrain compressionTerrain compression Compress terrain by selecting subset of pointsCompress terrain by selecting subset of points Reconstruct the terrain by solving a system of Reconstruct the terrain by solving a system of

equations to fill in missing pointsequations to fill in missing points The method we use to reconstruct the terrain The method we use to reconstruct the terrain

is slow for large datasetsis slow for large datasets We came up with a method for reconstructing We came up with a method for reconstructing

very large datasets quickly using MPIvery large datasets quickly using MPI

4RPI GeoStar Group

ODETLAPODETLAP

Over-Determined LaplacianOver-Determined Laplacian Two Equations:Two Equations:

4z4zijij = z = zi-1,ji-1,j + z + zi+1,ji+1,j + z + zi,j-1i,j-1 + z + zi,j+1i,j+1

zzijij = h = hijij

Multiple values for some pointsMultiple values for some points Require a smooth parameter R to interpolate Require a smooth parameter R to interpolate

when multiple values existwhen multiple values exist Reconstruct an approximated surface from Reconstruct an approximated surface from

{h{hijij} (Red points)} (Red points)

5RPI GeoStar Group

ExampleExample

ODETLAP can be used to fill in the unknown ODETLAP can be used to fill in the unknown data from a from a sparse point cloud:data from a from a sparse point cloud:

13,000,000 points 10,000*10,000

6RPI GeoStar Group

ODETLAP CompressionODETLAP Compression

Lossily compress image by selecting subset of pointsLossily compress image by selecting subset of points ODETLAP reconstruction solves for the whole terrainODETLAP reconstruction solves for the whole terrain

1) Compress

3) Reconstruct (ODETLAP)

2) Store

7RPI GeoStar Group

ODETLAP Performance IssueODETLAP Performance Issue

8RPI GeoStar Group

Motivation for ParallelizationMotivation for Parallelization

ODETLAP prohibitively slow for large datasetsODETLAP prohibitively slow for large datasets We need a scalable implementationWe need a scalable implementation Only a small neighborhood of points will affect a Only a small neighborhood of points will affect a

particular elevation.particular elevation. 1 pixel only affected1 pixel only affected

an area of 62x62an area of 62x62

9RPI GeoStar Group

Our ApproachOur Approach

Divide the terrain into individual patchesDivide the terrain into individual patches Run ODETLAP on each patch separatelyRun ODETLAP on each patch separately

3) Reconstruct each patch

1) Compressed terrain

2) Divide it into patches

4) Merge the patches

10RPI GeoStar Group

There is a problem!There is a problem!

Points near the edges of patches have Points near the edges of patches have incomplete data which causes errorsincomplete data which causes errors

Pixels in blue show correct results

Pixels in red show erroneous results

11RPI GeoStar Group

There is a problem! There is a problem! (continued)(continued)

We get discontinuity if we naively merge the patchesWe get discontinuity if we naively merge the patches

Naively reconstructed terrain: Errors:

12RPI GeoStar Group

SolutionSolution

Use overlapping layers of patchesUse overlapping layers of patches

13RPI GeoStar Group

SolutionSolution

Use overlapping layers of patchesUse overlapping layers of patches

14RPI GeoStar Group

SolutionSolution

Use overlapping layers of patchesUse overlapping layers of patches

15RPI GeoStar Group

SolutionSolution

Use overlapping layers of patchesUse overlapping layers of patches

16RPI GeoStar Group

SolutionSolution

Use overlapping layers of patchesUse overlapping layers of patches

17RPI GeoStar Group

SolutionSolution

Use overlapping layers of patchesUse overlapping layers of patches

18RPI GeoStar Group

SolutionSolution

Use overlapping layers of patchesUse overlapping layers of patches

19RPI GeoStar Group

SolutionSolution

Use overlapping layers of patchesUse overlapping layers of patches

20RPI GeoStar Group

SolutionSolution

Use overlapping layers of patchesUse overlapping layers of patches

21RPI GeoStar Group

SolutionSolution

Use overlapping layers of patchesUse overlapping layers of patches Then merge the resultsThen merge the results

22RPI GeoStar Group

SolutionSolution

Use overlapping layers of patchesUse overlapping layers of patches Then merge the resultsThen merge the results

23RPI GeoStar Group

Problem: Averaging the Problem: Averaging the patchespatches

A simple averaging of the patches incorporates A simple averaging of the patches incorporates the border error into the reconstructed terrain:the border error into the reconstructed terrain:

Terrain reconstructed using averaged patches Errors:

24RPI GeoStar Group

Solution: Bilinear Solution: Bilinear InterpolationInterpolation

Use bilinear interpolation to do a weighted Use bilinear interpolation to do a weighted average such that border values fall off to zero:average such that border values fall off to zero:

Bilinear interpolation results Error (avg: 0.1m, max: 2m):Naively averaging results

Elevation Range of the Original: 1105m..1610mElevation Range of the Original: 1105m..1610mUsing DTED Level 2 (30m spacing)Using DTED Level 2 (30m spacing)

Weighting Pattern for Bilinear Weighting Pattern for Bilinear Interpolation vs. Simple AveragingInterpolation vs. Simple Averaging

25RPI GeoStar Group

Bilinear InterpolationBilinear Interpolation Simple AveragingSimple Averaging

26RPI GeoStar Group

MPI ImplementationMPI Implementation

1) Each processor (except central process) is pre-1) Each processor (except central process) is pre-assigned one or more patchesassigned one or more patches

2) Every MPI process does the following for each 2) Every MPI process does the following for each patch assigned to it:patch assigned to it: Load patchLoad patch Run ODETLAP on the patchRun ODETLAP on the patch MPI_send the patch to the central processMPI_send the patch to the central process

3) When all of the patches have been received by 3) When all of the patches have been received by the central process, merge them using bilinear the central process, merge them using bilinear interpolation.interpolation.

27RPI GeoStar Group

ResultsResults 16,000*16,000 Central USA terrain data16,000*16,000 Central USA terrain data Use 128 2.6 GHz processors on RPI CCNI clusterUse 128 2.6 GHz processors on RPI CCNI cluster Divide into 101,761 patches of 100x100 sizeDivide into 101,761 patches of 100x100 size Completed in 28 minutes and 32 secondsCompleted in 28 minutes and 32 seconds Non-patch ODETLAP would have taken 179 daysNon-patch ODETLAP would have taken 179 days

28RPI GeoStar Group

Results(cont.)Results(cont.)

Size: 16K*16KSize: 16K*16K STD: 217STD: 217 Range: 1013Range: 1013 Mean Error: 1.96 Mean Error: 1.96 Max Error: 50 Max Error: 50 RMS Error: 2.76RMS Error: 2.76

The terrain was compressed by a factor of 100, The terrain was compressed by a factor of 100, with a mean error within 0.2% of the range.with a mean error within 0.2% of the range.

29RPI GeoStar Group

Original and reconstructed TerrainOriginal and reconstructed Terrain

Original Terrain (1000 * 1000) Reconstruction Result (1000 * 1000)

30RPI GeoStar Group

Patch Size vs. Time & ErrorPatch Size vs. Time & ErrorTotal sizeTotal size #points #points

usedusedPatch Patch sizesize

Running Running timetime

MeanMean

absolute absolute ErrorError

Max Max ErrorError

RMS RMS ErrorError

2000*20002000*2000 3989439894 50*5050*50 0m 38s0m 38s 0.66400.6640 1313 0.96170.9617

2000*20002000*2000 3989439894 100*100100*100 0m 55s0m 55s 0.65980.6598 1313 0.9530 0.9530

2000*20002000*2000 3989439894 200*200200*200 5m 25s5m 25s 0.65980.6598 1313 0.9527 0.9527

2000*20002000*2000 3989439894 400*400400*400 18m 49s18m 49s 0.65980.6598 1313 0.9527 0.9527

These results come from an 8-processor machine

31RPI GeoStar Group

Serialized vs. ParallelSerialized vs. Parallel Serialized: A single worker processor runs each patch Serialized: A single worker processor runs each patch

sequentially (speedup of 9.5 in the test)sequentially (speedup of 9.5 in the test) Parallel: Several processors run on many patches in Parallel: Several processors run on many patches in

parallel (additional speedup of 5.6 in the test)parallel (additional speedup of 5.6 in the test)

Test data: 800 x 800 size with mean elevation of 107

32RPI GeoStar Group

Running Time ComparisonRunning Time ComparisonMethodMethod Running Running

timetimeMean Mean ErrorError

Max ErrorMax Error RMSRMS

ErrorError

Original Original ODETLAPODETLAP

549s549s 0.61500.6150 77 0.88350.8835

Serial Serial ODETLAPODETLAP

34s34s 0.61560.6156 77 0.88460.8846

Parallel Parallel ODETLAPODETLAP

9s9s 0.61560.6156 77 0.88460.8846

Test data: 800 x 800 size with mean elevation of 107, run on 8 processors. Parallel ODETLAP is 50 times faster, while introducing only 0.1% additional error.

33RPI GeoStar Group

Number of Points 1 Large Patch 100x100 Patches

10000 0s 0s

40000 4s 2s

160000 41s 15s

640000 543s 58s

34RPI GeoStar Group

Number of Processors Time (seconds)

1 476

3 161

7 70

15 33

31 17

63 10

127 8

255 8

35RPI GeoStar Group

Current and Future WorkCurrent and Future Work

Improvements to our implementationImprovements to our implementation Reduce data size – regular grid can be more compactReduce data size – regular grid can be more compact Each process should grab the next available patchEach process should grab the next available patch Optimize for the Blue Gene/L system (see next slide)Optimize for the Blue Gene/L system (see next slide)

Reduce errors from the patch methodReduce errors from the patch method Improve the method for merging patchesImprove the method for merging patches

36RPI GeoStar Group

Blue Gene/L SystemBlue Gene/L System Computational Center for Nanotechnology Computational Center for Nanotechnology

Innovations (CCNI) at RPIInnovations (CCNI) at RPI 32,768 CPU’s @ 700 Mhz32,768 CPU’s @ 700 Mhz 512-1024MB memory/CPU (non-shared)512-1024MB memory/CPU (non-shared) Opportunity to run very large data sets quicklyOpportunity to run very large data sets quickly New methodNew method

Source, Sink, Workers, and CoordinatorSource, Sink, Workers, and Coordinator DEM size is not limited by process memory sizeDEM size is not limited by process memory size Use processors as cache instead of the diskUse processors as cache instead of the disk

On the BG, disk is slow, network and memory is very fastOn the BG, disk is slow, network and memory is very fast We must reduce the overhead to take advantage of all CPU’sWe must reduce the overhead to take advantage of all CPU’s

37RPI GeoStar Group

SummarySummaryOriginal Reconstructed

The terrain is divided into overlapping patches and ODETLAP is solved separately on individual patches, which will be merged into a whole patch using bilinear interpolation at the end

We experiment on a 16K*16K test data. We show two 1K*1K pieces from the original and reconstructed terrain to illustrate the accuracy of the reconstruction