Real-time Visualization of Massive Imagery and Volumetric Datasets Thesis Defense by Ian Roth

Real-time Visualization of Real-time Visualization of Massive Imagery and Massive Imagery and Volumetric DatasetsVolumetric Datasets

Thesis DefenseThesis Defense

byby

Ian RothIan Roth

OverviewOverview

• IntroductionIntroduction• Out-of-core software architectureOut-of-core software architecture

– Asynchronous data I/O engineAsynchronous data I/O engine– Hierarchical cachingHierarchical caching

• Interactive visualization techniquesInteractive visualization techniques– Viewing modes for image generationViewing modes for image generation– Per-tile rendering methodsPer-tile rendering methods– User interface and navigation modesUser interface and navigation modes

• Performance and benchmarkingPerformance and benchmarking• Features and extensionsFeatures and extensions• Conclusions and future workConclusions and future work

Kolam Software PackageKolam Software Package

• Developed at University of Missouri, ColumbiaDeveloped at University of Missouri, Columbia• Version 1.0 by Joshua Fraser (2000 – ‘04)Version 1.0 by Joshua Fraser (2000 – ‘04)

– Written in C, OpenGL and GLUTWritten in C, OpenGL and GLUT• Version 2.0 by Ian Roth (2001 – ‘04)Version 2.0 by Ian Roth (2001 – ‘04)

– Written in C++, OpenGL and QtWritten in C++, OpenGL and Qt• Additional code by Jared Hoberock (2001 – Additional code by Jared Hoberock (2001 –

‘02)‘02)– Dataset I/O utilitiesDataset I/O utilities

• Other contributorsOther contributors– Dr. K. Palaniappan, Ian Scott, Dave Metts, Dr. K. Palaniappan, Ian Scott, Dave Metts,

Vidyasagar Chada, Mike Sullivan, Ryan CalhounVidyasagar Chada, Mike Sullivan, Ryan Calhoun

MotivationMotivation

• Massive datasetsMassive datasets– Too large to fit entirely in memoryToo large to fit entirely in memory– Examples:Examples:

• IKONOS/MODIS panchromatic/multispectral satellite IKONOS/MODIS panchromatic/multispectral satellite imageryimagery

• CT/MRI/PET scans, Visible Human ProjectCT/MRI/PET scans, Visible Human Project

• Out-of-core approachOut-of-core approach– Require segmentation (tiling) for efficient roam Require segmentation (tiling) for efficient roam

operationsoperations– Require multiple resolutions (LODs) for efficient Require multiple resolutions (LODs) for efficient

zoom operationszoom operations

DefinitionsDefinitions

• Image or DatasetImage or Dataset– Encapsulation of a pyramid file (disk cache)Encapsulation of a pyramid file (disk cache)– IIrr((xx)) : image at resolution : image at resolution rr, pixel coordinate , pixel coordinate xx

• LayerLayer– A dataset with associated viewing parametersA dataset with associated viewing parameters

• Associated colormaps, heightmaps, etc.Associated colormaps, heightmaps, etc.• Stores x/y offset and scale for embedded datasetsStores x/y offset and scale for embedded datasets

• ColormapColormap– Maps dataset values to RGB color valuesMaps dataset values to RGB color values

• HeightmapHeightmap– Maps image pixels to height valuesMaps image pixels to height values

Kolam Software Kolam Software ArchitectureArchitecture• EngineEngine

– Based on the workpile design patternBased on the workpile design pattern– Multiple worker threads process requests from Multiple worker threads process requests from

application and perform data I/O operationsapplication and perform data I/O operations• ApplicationApplication

– Displays data and GUI componentsDisplays data and GUI components– Determines which tiles are visible and at what resolutionDetermines which tiles are visible and at what resolution– Posts request which are handled by engine’s worker Posts request which are handled by engine’s worker

threadsthreads• Application ExtensionsApplication Extensions

– Plugins: dynamically linked librariesPlugins: dynamically linked libraries– Provide additional GUI components, dataset I/O routines, Provide additional GUI components, dataset I/O routines,

image processing methods, image viewing and image processing methods, image viewing and projection modes, etc.projection modes, etc.

Kolam EngineKolam Engine

• Asynchronously processes Asynchronously processes requests from applicationrequests from application

• Proceedure:Proceedure:1.1. Worker threads are initially idleWorker threads are initially idle2.2. For each frame, the application For each frame, the application

determines visible tiles and determines visible tiles and posts requestsposts requests

3.3. Each posted request awakens Each posted request awakens a worker threada worker thread

4.4. Repeat forever for each awakened threadRepeat forever for each awakened thread1.1. Worker thread attempts to pop request from queue; if no requests are on Worker thread attempts to pop request from queue; if no requests are on

the queue, then exit loop.the queue, then exit loop.2.2. Worker thread reserves a free slot in cacheWorker thread reserves a free slot in cache3.3. Worker thread performs tile I/OWorker thread performs tile I/O4.4. Application is notified of new data in cacheApplication is notified of new data in cache

5.5. Worker thread enters idle stateWorker thread enters idle state

Kolam Engine VersionsKolam Engine Versions

• Version 1.0Version 1.0– Each layer has its own Each layer has its own

resourcesresources• Per layer: cache space, Per layer: cache space,

threads, tile lookup table, threads, tile lookup table, requestsrequests

– Fine for non-embedded Fine for non-embedded datasetsdatasets

• Version 2.0Version 2.0– Engine has global resources, Engine has global resources,

shared by layersshared by layers• Engine: cache space, threads, Engine: cache space, threads,

requestsrequests• Per layer: tile lookup tablePer layer: tile lookup table

– Allows active layers to seize Allows active layers to seize resources from inactive layersresources from inactive layers

Hierarchical CachingHierarchical Caching

• Three levels:Three levels:– Disk cacheDisk cache

• Stores tiles on diskStores tiles on disk• Original source may be Original source may be

row-major order image file row-major order image file on disk or tiles streamed on disk or tiles streamed over networkover network

– Main memory cacheMain memory cache• Stores fixed number of tiles in memoryStores fixed number of tiles in memory

– Texture cacheTexture cache• Stores fixed number of tiles on graphics hardwareStores fixed number of tiles on graphics hardware• Passed directly to real-time display componentPassed directly to real-time display component

Disk CacheDisk Cache

• Pyramid file formatPyramid file format– Additional resolutions appended to end of image fileAdditional resolutions appended to end of image file– Requires additional space (no more than 2 Requires additional space (no more than 2 original) original)

• 1D image takes 2 1D image takes 2 space of original space of original• 2D image takes 4/3 2D image takes 4/3 space of original space of original• 3D image takes 8/7 3D image takes 8/7 space of original space of original

• Wavelet encoded file formatWavelet encoded file format– Additional resolutions embedded in imageAdditional resolutions embedded in image– Requires no additional spaceRequires no additional space– Complex to encode/decode, especially in Complex to encode/decode, especially in nn-D-D– Examples:Examples:

• JPEG 2000JPEG 2000• MrSID (LizardTech)MrSID (LizardTech)

Main Memory CacheMain Memory Cache

• Asynchronous I/O Asynchronous I/O – Multiple read/write threadsMultiple read/write threads– Allows display thread to continue working while data is loadingAllows display thread to continue working while data is loading– Spreads out I/O lag time over shorter intervalsSpreads out I/O lag time over shorter intervals

• Caching: allocates/deallocates memory for tile dataCaching: allocates/deallocates memory for tile data– Cache methods available to application developersCache methods available to application developers

• checkout tile, checkin tile, add request, query tile/request, invalidate checkout tile, checkin tile, add request, query tile/request, invalidate tile/requesttile/request

– Variations of the checkout methodVariations of the checkout method• basic checkout, checkout first available, wait for tilebasic checkout, checkout first available, wait for tile

• Paging: swaps old data for newPaging: swaps old data for new– Temporal strategies – view projection independentTemporal strategies – view projection independent

• LRU priority queue – LRU priority queue – OO((# of layers# of layers))– Spatial strategies – view projection dependentSpatial strategies – view projection dependent

• Brute force distance calculation – Brute force distance calculation – OO((# of cached tiles# of cached tiles))• Modular ROI array – Modular ROI array – OO(1)(1)• Graph traversal algorithms – Graph traversal algorithms – OO((# of cached tiles# of cached tiles))

Texture CacheTexture Cache

• Similar to main memory LRU cacheSimilar to main memory LRU cache– Replace Replace allocatorallocator object object

• Synchronous I/OSynchronous I/O– Adequate for single data display windowAdequate for single data display window

• Can be implemented asynchronouslyCan be implemented asynchronously– Useful for multiple displaysUseful for multiple displays

• One-to-one correspondence between thread and One-to-one correspondence between thread and contextcontext

• Each display is essentially synchronousEach display is essentially synchronous• Frame rate not bounded by longest display timeFrame rate not bounded by longest display time• Implemented by OpenRM Scene GraphImplemented by OpenRM Scene Graph

Interactive Visualization Interactive Visualization ComponentsComponents• Viewing modesViewing modes

– Orthogonal projectionOrthogonal projection– Oblique projection (separate source tree)Oblique projection (separate source tree)– Spherical projectionSpherical projection– Arbitrary geometry projection (future work)Arbitrary geometry projection (future work)– Volumetric visualization (separate source tree)Volumetric visualization (separate source tree)

• Tile renderersTile renderers– Raster rendererRaster renderer– Texture rendererTexture renderer– Terrain rendererTerrain renderer– Ray casting renderer (separate source tree)Ray casting renderer (separate source tree)

• Navigation modesNavigation modes– Roam, zoom, and tiltRoam, zoom, and tilt– Fly mode (separate source tree)Fly mode (separate source tree)– Embedded dataset positioningEmbedded dataset positioning– Image crop (future work)Image crop (future work)– Geometry and terrain manipulation (future work)Geometry and terrain manipulation (future work)

Visualization Running TimeVisualization Running Time

• Measured by number of tiles processed per frameMeasured by number of tiles processed per frame• DefinitionsDefinitions

– nn((II)) : total number of tiles in image : total number of tiles in image II– mm((II, , tt)) : number of visible tiles in image : number of visible tiles in image II at time at time tt

• Processing Processing OO((nn)) tiles per frame is too much tiles per frame is too much• Processing Processing ((mm)) tiles per frame is the goal tiles per frame is the goal• Analytic solution: Analytic solution:

– OO((mm)) : single resolution solution : single resolution solution– OO((m + m + lglg nn)) : multiple resolution solution : multiple resolution solution– Not suitable for explicitly defined projections, i.e. triangular meshesNot suitable for explicitly defined projections, i.e. triangular meshes– View projection dependent, often difficult to implementView projection dependent, often difficult to implement

• Iterative solution:Iterative solution:– OO((mm lglg m + m + lglg nn)) : single/multiple resolution solution : single/multiple resolution solution– Relies on quadtree/octree/scenegraph traversal to Relies on quadtree/octree/scenegraph traversal to

iteratively refine solution from coarse to fine resolutionsiteratively refine solution from coarse to fine resolutions– View projection independent, relatively east to implementView projection independent, relatively east to implement

• Visible rectangular region Visible rectangular region width/height:width/height:

• Visible rectangular region x/y Visible rectangular region x/y offset:offset:

• Resulting region Resulting region RR is in tile is in tile coordinates at scale coordinates at scale

Orthogonal Projection Orthogonal Projection Analytic SolutionAnalytic Solution• tt : 2D translation/position : 2D translation/position

vectorvector in original resolution pixel in original resolution pixel coordscoords

• zz : pixel zoom factor : pixel zoom factor• {{wwTT, , hhTT}} : tile width/height : tile width/height• {{wwVV, , hhVV}} : viewport : viewport

width/heightwidth/height• Global image resolution:Global image resolution:

• Scaling matrix (to tile Scaling matrix (to tile coordinates):coordinates):

Oblique ProjectionOblique ProjectionAnalytic SolutionAnalytic Solution• Single resolution solutionSingle resolution solution

– Intersect view frustum planes with image planeIntersect view frustum planes with image plane• Result is a polygon in the image planeResult is a polygon in the image plane• Clip polygon to image boundariesClip polygon to image boundaries

– Run polygon fill algorithm to find tiles in polygon at Run polygon fill algorithm to find tiles in polygon at a fixed resolutiona fixed resolution

• Flood fill algorithm (recommended)Flood fill algorithm (recommended)– First use Bresenham to discretize polygon to tile coordsFirst use Bresenham to discretize polygon to tile coords

• Scanline fill algorithmScanline fill algorithm

• Multiple resolution solutionMultiple resolution solution– First subdivide frustum into regionsFirst subdivide frustum into regions

• Calculate distance from eye Calculate distance from eye pp when when zz = 1 = 1• : FovY/2, : FovY/2, rr : resolution, : resolution, dd : distance from eye : distance from eye• Assume pixel aspect ratio = 1Assume pixel aspect ratio = 1• dd and and pp both measured in pixels both measured in pixels

– Perform single resolution solution Perform single resolution solution for each sectionfor each section

Oblique ProjectionOblique ProjectionIterative SolutionIterative Solution• Quadtree traversalQuadtree traversal• Determine if tile is visibleDetermine if tile is visible

– View-frustum culling testView-frustum culling test• Performed in either Performed in either 33 or canonical view volume or canonical view volume

(CVV) space(CVV) space• Test 4 tile vertices w/ 6 frustum planes (24 tests)Test 4 tile vertices w/ 6 frustum planes (24 tests)• Test 4 tile edges w/ 6 frustum planes (24 tests)Test 4 tile edges w/ 6 frustum planes (24 tests)• Test 12 frustum edges w/ tile plane (12 tests)Test 12 frustum edges w/ tile plane (12 tests)• Maximum of 60 tests per tileMaximum of 60 tests per tile

• Determine tile resolutionDetermine tile resolution– Project to viewing plane and calculate Project to viewing plane and calculate

quadrilateral areaquadrilateral area

– Caclulate resolution at sample pointsCaclulate resolution at sample points

Spherical ProjectionSpherical ProjectionAnalytic SolutionAnalytic Solution• Convert Euler coords to sphericalConvert Euler coords to spherical

– uu : [0, 2 : [0, 2], ], vv : [- : [-/2, /2, /2]/2]• Intersect view frustum planes with sphereIntersect view frustum planes with sphere

– Result will be up to 6 circles in 3D spaceResult will be up to 6 circles in 3D space• nn : unit normal of plane : unit normal of plane PP,, pp : any point on : any point on PP• ccc/sc/s : circle/sphere center, : circle/sphere center, rrc/sc/s : circle/sphere radius : circle/sphere radius• Find distance Find distance dd from from ccss to plane along to plane along nn

• Project circles of intersection onto image planeProject circles of intersection onto image plane– Parametric equations for circle and sphereParametric equations for circle and sphere

• tt : [0, 2 : [0, 2], ], xx : : vertex in vertex in 33 • nn : unit normal of circle plane, : unit normal of circle plane, pp : up vector : up vector• aa = = nn pp, , bb = = nn aa

– Solve for Solve for uu and and vv– For each circleFor each circle

• Pick sample Pick sample tt values values• uu((tt)) and and vv((tt)) are points in image plane are points in image plane

– Compute polygon intersectionsCompute polygon intersections• Tesselate non-convex polygonsTesselate non-convex polygons• Perform pairwise convex polygon intersection testsPerform pairwise convex polygon intersection tests

Spherical ProjectionSpherical ProjectionIterative SolutionIterative Solution• Determine visible tilesDetermine visible tiles

– Tiles no longer planar, can be composed of many Tiles no longer planar, can be composed of many polygonspolygons

• Use bounding spheres to approximate curved tile shapesUse bounding spheres to approximate curved tile shapes• Similar to vertex/frustum intersection testSimilar to vertex/frustum intersection test

– Hidden surface removal Hidden surface removal • Back-face culling methods don’t work on curved surfacesBack-face culling methods don’t work on curved surfaces• Additional clipping plane to divide sphere into front- and Additional clipping plane to divide sphere into front- and

back-facing regionsback-facing regions– Orthogonal to vector Orthogonal to vector vv– Need to determine distance from eyeNeed to determine distance from eye

• Determine tile resolutionDetermine tile resolution– Projecting and calculating area is difficult for curved Projecting and calculating area is difficult for curved

tilestiles• Overlapping regions not accounted forOverlapping regions not accounted for


Arbitrary Geometry ProjectionArbitrary Geometry ProjectionIterative SolutionIterative Solution

• Image projected onto arbitrary triangular Image projected onto arbitrary triangular meshmesh

• Similar to spherical projectionSimilar to spherical projection– Requires mapping from mesh vertices to Requires mapping from mesh vertices to uu//vv coords coords

• Determine visible tilesDetermine visible tiles– Use hierarchy of bounding spheresUse hierarchy of bounding spheres– Hidden surface removalHidden surface removal

• ID image : a lookup table of visible trianglesID image : a lookup table of visible triangles

• Determine tile resolutionDetermine tile resolution– Caclulate resolution at sample pointsCaclulate resolution at sample points

Volumetric VisualizationVolumetric VisualizationIterative SolutionIterative Solution• Octree traversalOctree traversal• Determine if tile (cube) is visibleDetermine if tile (cube) is visible

– View-frustum culling testView-frustum culling test• Performed in either Performed in either 33 or canonical view volume (CVV) space or canonical view volume (CVV) space• Test 8 tile vertices w/ 6 frustum planes (48 tests)Test 8 tile vertices w/ 6 frustum planes (48 tests)• Test 12 tile edges w/ 6 frustum planes (72 tests)Test 12 tile edges w/ 6 frustum planes (72 tests)• Test 8 frustum vertices w/ 6 tile planes (48 tests)Test 8 frustum vertices w/ 6 tile planes (48 tests)• Test 12 frustum edges w/ 6 tile planes (72 tests)Test 12 frustum edges w/ 6 tile planes (72 tests)• Maximum of 240 tests per tile Maximum of 240 tests per tile • Using bounding spheres only 6 tests required, but less accurateUsing bounding spheres only 6 tests required, but less accurate• Optimization: use spheres during octree traversal, use cubes on leaf nodesOptimization: use spheres during octree traversal, use cubes on leaf nodes

• Determine tile resolutionDetermine tile resolution– Project to viewing plane and calculate convex hull (polygon) areaProject to viewing plane and calculate convex hull (polygon) area


Raster RendererRaster Renderer

• Renders images directly to framebufferRenders images directly to framebuffer• AdvantagesAdvantages

– Simple to implementSimple to implement– No additional texture/VRAM cache requiredNo additional texture/VRAM cache required– No additional overhead from texture load timeNo additional overhead from texture load time– Tile data changes are applied instantly (on next display Tile data changes are applied instantly (on next display

update)update)• No need to flush texture cacheNo need to flush texture cache

• DisadvantagesDisadvantages– Can only be used in orthogonal viewing modeCan only be used in orthogonal viewing mode– Not all affine transformations supported Not all affine transformations supported

• Rotations and shearingRotations and shearing• Only 2D transformationsOnly 2D transformations

– Can be slow on some graphics hardwareCan be slow on some graphics hardware

Texture RendererTexture Renderer

• Stores tile data in texture/VRAM memory on graphics hardwareStores tile data in texture/VRAM memory on graphics hardware• AdvantagesAdvantages

– Often faster than rendering directly to framebufferOften faster than rendering directly to framebuffer– Allows any affine transformation to be applied to a tileAllows any affine transformation to be applied to a tile– Allows hardware-accellerated interpolation to be applied when renderingAllows hardware-accellerated interpolation to be applied when rendering– Can be used by all viewing modesCan be used by all viewing modes

• Volumetric viewing mode requires 3D texture support, otherwise must use ray casting rendererVolumetric viewing mode requires 3D texture support, otherwise must use ray casting renderer

• DisadvantagesDisadvantages– Wasted space when compositing color channelsWasted space when compositing color channels

• Luminance component always stored in red channel in OpenGLLuminance component always stored in red channel in OpenGL• Must use RGB color textures for color masking to be effectiveMust use RGB color textures for color masking to be effective

– Need to flush texture cache when tile data changes or global filters are appliedNeed to flush texture cache when tile data changes or global filters are applied• Texture cacheTexture cache

– Similar to main memory LRU cacheSimilar to main memory LRU cache• Synchronous rather than asynchronousSynchronous rather than asynchronous• Can reuse some code by replacing the Can reuse some code by replacing the allocatorallocator object object

• Texture bordersTexture borders– Use OpenGL GL_CLAMP_TO_EDGE texture parameterUse OpenGL GL_CLAMP_TO_EDGE texture parameter

• Widely supported, produces artifacts when z Widely supported, produces artifacts when z 1 1– Use OpenGL texture bordersUse OpenGL texture borders

• Not widely supported, often slowNot widely supported, often slow• Tradeoff between efficiency and storageTradeoff between efficiency and storage

– Simulate texture borders (not implemented)Simulate texture borders (not implemented)• Tile size: Tile size: 22nn – 2 – 2, Texture coords: , Texture coords: [1 / 2[1 / 2nn,, (2(2nn – 1) / 2 – 1) / 2nn]]• Requires major change to file formatRequires major change to file format• Tradeoff between efficiency and storageTradeoff between efficiency and storage

Terrain RendererTerrain Renderer

• Similar to texture rendererSimilar to texture renderer– Also converts a heightmap to visible terrainAlso converts a heightmap to visible terrain– Rendering a single tile requires neighboring heightmap Rendering a single tile requires neighboring heightmap

tiles to be availabletiles to be available– Two possible implementationsTwo possible implementations

• Triangular mesh terrainTriangular mesh terrain– Problems with seams across tiles of disparate Problems with seams across tiles of disparate

resolutionsresolutions

• Bump-mapped terrain (Ian Scott)Bump-mapped terrain (Ian Scott)– Horizon still looks flatHorizon still looks flat– Requires fragment shader support (GPU)Requires fragment shader support (GPU)– Requires additional texture memoryRequires additional texture memory

Ray Casting RendererRay Casting Renderer

• Used by volumetric view modeUsed by volumetric view mode• Can be implemented on GPUCan be implemented on GPU

– Graphics hardware must support 3D textures and fragment shadersGraphics hardware must support 3D textures and fragment shaders• ProceedureProceedure

1.1. Project tile onto viewing planeProject tile onto viewing plane1.1. To find regions of plane that intersect tileTo find regions of plane that intersect tile

1.1. Intersect 6 tile planes with viewing planeIntersect 6 tile planes with viewing plane2.2. Map polygon vertices to voxel coordinatesMap polygon vertices to voxel coordinates

To find regions of plane that do not intersect tileTo find regions of plane that do not intersect tile1.1. Project 8 tile vertices onto viewing planeProject 8 tile vertices onto viewing plane

2.2. Compute convex hull of result (a polygon)Compute convex hull of result (a polygon)3.3. Traverse each pixel of polygon in voxel spaceTraverse each pixel of polygon in voxel space

– Flood fill algorithmFlood fill algorithm– Scanline fill algorithm (recommended)Scanline fill algorithm (recommended)

2.2. For each pixel of polygonFor each pixel of polygon1.1. If pixel coordinate in voxel space intersects tileIf pixel coordinate in voxel space intersects tile

1.1. Tile voxel coordinate = pixel coordinate in voxel space modulo tile width/height/depthTile voxel coordinate = pixel coordinate in voxel space modulo tile width/height/depth2.2. Can Can

OtherwiseOtherwise1.1. Tile voxel coordinate = intersection of ray from eye through pixel Tile voxel coordinate = intersection of ray from eye through pixel

with frontmost tile planewith frontmost tile plane

2.2. Traverse ray from eye through pixel starting with voxel coordinateTraverse ray from eye through pixel starting with voxel coordinate1.1. Accumulate voxel data along rayAccumulate voxel data along ray

Navigation ModesNavigation Modes

• Roam, zoom and tiltRoam, zoom and tilt– Drag image using combination of mouse buttonsDrag image using combination of mouse buttons

• Fly modeFly mode– Constant velocity based on ray from screen center to mouse Constant velocity based on ray from screen center to mouse

cursorcursor• Embedded dataset positioningEmbedded dataset positioning

– Position, resize and rotate a single datasetPosition, resize and rotate a single dataset• Image cropImage crop

– Drag edges of dataset to change size without stretchingDrag edges of dataset to change size without stretching• Geometry manipulationGeometry manipulation

– Move individual vertices of geometryMove individual vertices of geometry– Select group of vertices to move at onceSelect group of vertices to move at once– Use mass-spring system to pull connected verticesUse mass-spring system to pull connected vertices– Use finite element method to simulate objects with physical Use finite element method to simulate objects with physical

propertiesproperties

OptimizationsOptimizations

• Hiding disk latencyHiding disk latency– Search upward through quadtree for equivalent coarser resolutionsSearch upward through quadtree for equivalent coarser resolutions– Never see blank tiles, see blurred tiles insteadNever see blank tiles, see blurred tiles instead– Implemented in “checkout first available” cache checkout methodImplemented in “checkout first available” cache checkout method

• Maintaining constant FPSMaintaining constant FPS– Maintain maximum number of visible tilesMaintain maximum number of visible tiles– Replace group(s) of 2–4 tiles with a single coarser tile while roaming and/or Replace group(s) of 2–4 tiles with a single coarser tile while roaming and/or

zoomingzooming• Prioritizing requestsPrioritizing requests

– Maintain list of visible tiles for current and previous framesMaintain list of visible tiles for current and previous frames– Compare lists Compare lists

• Remove requests for tiles that are no longer visibleRemove requests for tiles that are no longer visible• Move requests for tiles that are no longer visible to lower priority queueMove requests for tiles that are no longer visible to lower priority queue

• PrefetchingPrefetching– Load constant size radius of tiles from around ROI when idleLoad constant size radius of tiles from around ROI when idle– Load adjacent tiles from coarser/finer resolutions when idleLoad adjacent tiles from coarser/finer resolutions when idle– Use motion vectors from recent user navigation to predict next ROIUse motion vectors from recent user navigation to predict next ROI

PerformancePerformance

• InteractivityInteractivity– FPS & # of dropped requests w/ varying number of FPS & # of dropped requests w/ varying number of

layerslayers– FPS & # of dropped requests w/ varying cache sizesFPS & # of dropped requests w/ varying cache sizes– FPS w/ varying # of (embedded/non-embedded) layersFPS w/ varying # of (embedded/non-embedded) layers– Hardware-specific testsHardware-specific tests

• FPS w/ and w/o colormapsFPS w/ and w/o colormaps• FPS of raster vs. texture modeFPS of raster vs. texture mode

• Cache efficiencyCache efficiency– I/O latency w/ and w/o compressionI/O latency w/ and w/o compression– Paging time analysisPaging time analysis

• File format efficiencyFile format efficiency– Compression ratiosCompression ratios

Performance – Performance – MultithreadingMultithreading

Frames Displayed vs. Number of Threads

0

30

60

90

120

0 10 20 30 40 50 60Seconds

Fram

es

Dis

pla

yed

1 Thread

2 Threads

4 Threads

8 Threads

16 Threads

32 Threads

Avg. FPSAvg. FPS

1 Thread1 Thread 69.669.6

2 Threads2 Threads 48.648.6





Requests Dropped vs. Number of Threads

0

10

20

30

40

0 10 20 30 40 50 60Seconds

Request

s D

ropped

1 Thread

2 Threads

4 Threads

8 Threads

16 Threads

32 Threads

Avg. Requests DroppedAvg. Requests Dropped

1 Thread1 Thread 6.936.93






Performance – Cache SizePerformance – Cache Size

Frames Displayed vs. Cache Size

0

30

60

90

120

0 10 20 30 40 50 60Seconds

Fram

es

Dis

pla

yed

8 MB Cache

16 MB Cache

32 MB Cache

64 MB Cache

128 MB Cache

256 MB Cache

Avg. FPSAvg. FPS

8 MB Cache8 MB Cache 75.575.5




128 MB 128 MB CacheCache

68.868.8


67.067.0

Requests Dropped vs. Cache Size

0

10

20

30

40

0 10 20 30 40 50 60Seconds

Request

s D

ropped

8 MB Cache

16 MB Cache

32 MB Cache

64 MB Cache

128 MB Cache

256 MB Cache

Avg. Requests DroppedAvg. Requests Dropped






3.663.66


1.541.54

Application ExtensionsApplication Extensions

• Plugins: dynamically linked librariesPlugins: dynamically linked libraries– Plugins have access to all application dataPlugins have access to all application data– Plugins can attach processing routines to predefined Plugins can attach processing routines to predefined

breakpoints in the codebreakpoints in the code– A list of A list of onlyonly the attached routines is processed at each the attached routines is processed at each

breakpointbreakpoint

• Typical extension categoriesTypical extension categories– Image/dataset file formatsImage/dataset file formats– GUI componentsGUI components– Viewing modesViewing modes– Tile renderersTile renderers– Navigation modesNavigation modes

Example PluginsExample Plugins

• Apply ColormapApply Colormap– Provides GUI control similar to layer dialogProvides GUI control similar to layer dialog– Uses “prerender” breakpoint to apply colormap to image Uses “prerender” breakpoint to apply colormap to image

before displaybefore display

• Histogram Display & EnhancementHistogram Display & Enhancement– Provides GUI control w/slidersProvides GUI control w/sliders– Uses “layer initialized” breakpoint to spawn new thread Uses “layer initialized” breakpoint to spawn new thread

to compute histogram of overviewto compute histogram of overview– Uses “wait for tile” tile checkout method to avoid busy Uses “wait for tile” tile checkout method to avoid busy

waitingwaiting– Uses “prerender” breakpoint to apply histogram Uses “prerender” breakpoint to apply histogram

enhancement to image before displayenhancement to image before display

Conclusions and Future Conclusions and Future WorkWork• ConclusionsConclusions

– A stable and basic visualization toolA stable and basic visualization tool– Many features yet to be implementedMany features yet to be implemented

• Need funding for full time programmers!Need funding for full time programmers!

• SBIR ProposalSBIR Proposal– Extension to Extension to nn-dimensional data-dimensional data

• View slices of both sparse and dense datasetsView slices of both sparse and dense datasets– Adoption of HDF5 file formatAdoption of HDF5 file format

• Chunking, grouping, Chunking, grouping, nn-D support-D support• Possibly provide our own extensions to the formatPossibly provide our own extensions to the format

– Image projection onto arbitrarily large triangular meshesImage projection onto arbitrarily large triangular meshes• Unified framework for sphere mapping, terrain, etc.Unified framework for sphere mapping, terrain, etc.• Fast image registrationFast image registration• Incorporate modeling and deformation toolsIncorporate modeling and deformation tools

– Network streaming of tile dataNetwork streaming of tile data• Similar to Keyhole softwareSimilar to Keyhole software

Similar Large Data Similar Large Data Visualization SoftwareVisualization Software• ESRI – ArcGISESRI – ArcGIS

– Full-featured GIS packageFull-featured GIS package– May not handle large images (bigger than available RAM/virtual memory)May not handle large images (bigger than available RAM/virtual memory)

• Keyhole – EarthViewerKeyhole – EarthViewer– Network streaming of remote sensing & vector dataNetwork streaming of remote sensing & vector data

• LizardTech – GeoExpress ViewLizardTech – GeoExpress View– MrSID file format & image viewing softwareMrSID file format & image viewing software– Achieves high compression ratios at expense of qualityAchieves high compression ratios at expense of quality

• Sensor Systems – Remote ViewSensor Systems – Remote View– Image processing and analysis software used in DoDImage processing and analysis software used in DoD– Doesn’t handle large images (bigger than available RAM/virtual memory)Doesn’t handle large images (bigger than available RAM/virtual memory)

• EPFL – Real-time Visible Human NavigatorEPFL – Real-time Visible Human Navigator– Only works on visible human datasetOnly works on visible human dataset

• SGI – OpenGL Performer/VolumizerSGI – OpenGL Performer/Volumizer– C/C++ API for 2D/3D large data visualizationC/C++ API for 2D/3D large data visualization– IRIX/Linux platforms onlyIRIX/Linux platforms only

• Kitware – VTK (Visualization Toolkit)Kitware – VTK (Visualization Toolkit)– C++/Tcl/Java/Python API for 2D/3D large data visualizationC++/Tcl/Java/Python API for 2D/3D large data visualization– Large mesh dataset support based on multiple LODs, no inherent datset segmentationLarge mesh dataset support based on multiple LODs, no inherent datset segmentation

Screenshot – Display Screenshot – Display WindowsWindows

Screenshot – GUI & Screenshot – GUI & ColormapsColormaps

AnimationAnimation

• Embedded DatasetsEmbedded Datasets

• Layers & ColormapsLayers & Colormaps

• Spherical View ModeSpherical View Mode

IBM T221 DisplayIBM T221 Display

Documents

Real-time Visualization of Massive Imagery and Volumetric Datasets Thesis Defense by Ian Roth