03/31/03 © 2003 University of Wisconsin

Last Time

• Image-Based Rendering for Architecture and Faces

03/31/03 © 2003 University of Wisconsin

Today

• Image-Based methods for real-time walkthroughs

• Project arrangements– I will require a brief (1-2 page) description of your project by the

end of next week (April 11)

– If you haven’t already, talk to me this week

• NPR presentations– I will have a schedule up this week

– Days will be April 7, 9, 11, 14, 16

– People with April 7 deadlines will be off the hook for April 7

03/31/03 © 2003 University of Wisconsin

Rendering Massive Models

• “Images” for image-based rendering need not be images of real-things

• IBR is an important technique for reducing the cost of rendering large models– Large these days means millions of polygons

– Many models exist of this size: cities, architectural models, ships, factories, …

• Level-of-detail techniques aim to reduce the cost of rendering by reducing the triangle count– For a full discussion of LOD, take CS679

03/31/03 © 2003 University of Wisconsin

Image-Based Rendering Revisited

• Image-based rendering may be applied to the problem of rendering large databases– Primary advantage: Rendering cost depends mostly on the number

of pixels in the images, not the number of objects those pixels represent

– Disadvantage: Have to allow a wide range of viewer motion, which exacerbates typical IBR problems (cracks, stretching…)

• Basic idea: Replace geometry with a few texture mapped polygons– Polygons and texture maps can be generated in a pre-process, or as

required at run-time

• Environment maps are an instance of this approach

03/31/03 © 2003 University of Wisconsin

Billboards

• A billboard is extreme LOD, reducing all the geometry to one or more textured polygons– The geometry is replaced by an image in the form of a

texture

• Issues in designing billboards– How are they generated?– How are they oriented with respect to the viewer?– How can they be improved?– None of the answers in this class

• Also called sprites, but a sprite normally stays aligned parallel to the image plane

03/31/03 © 2003 University of Wisconsin

Multi-Polygon Billboards

• Use two polygons at right angles:– No alignment with viewer

– What is this good for?

– How does the apparent width change with viewing angle?

• Use more polygons is desired for better appearance– How does it affect the apparent width?

• Rendering options: Blended or just depth buffered

03/31/03 © 2003 University of Wisconsin

View Dependent Billboards

• What if the object is not rotationally symmetric?– Appearance should change from different viewing angles

• This can still be done with billboards:– Compute multiple textures each corresponding to a different view– Keep polygon fixed but vary texture according to viewer direction– Best: Interpolate, with texture blending, between the two nearest

views• Can use 3D textures and hardware texture filtering to achieve good

results

• Polygons are typically fixed in this approach, which restricts the viewing angles– Solution: Use more polygons each with a set of views associated

with it

03/31/03 © 2003 University of Wisconsin

View Dependent Textures

Screen shots from an Nvidia demo

03/31/03 © 2003 University of Wisconsin

Impostor Example

• Another methods uses slices from the original volume and blends them

http://zeus.fri.uni-lj.si/~aleks/slicing-and-blending/

03/31/03 © 2003 University of Wisconsin

Textured Clusters(Maciel and Shirley, 1995)

• Provide a range of representations for each object, and a benefit associated with each representation– Texture-mapped box is one representation, as is an average color box– Benefit may be view dependent – only texture one face

• Use an octree to group objects into clusters– Representations for a cluster are a textured and average color box– Benefit is derived from benefit of children (maximum of children)

• Deciding what to draw at run time is a bin-packing variation– Solve approximately

• First pass computes cost, benefit and visibility, and sets initial model• Second pass starts at root, expands nodes in order of decreasing benefit

– Not very scalable – looks at every representation of every object on each frame

03/31/03 © 2003 University of Wisconsin

Clusters and Tree Walk

03/31/03 © 2003 University of Wisconsin

Clusters Results - Image

03/31/03 © 2003 University of Wisconsin

Clusters Results – Top View

03/31/03 © 2003 University of Wisconsin

Textured Clusters Discussion

• Coined the term “impostor”

• Benefit does not add: The benefit of two objects together may be much greater than their sum (eg a man and a gun)

• Data-requirements may be very large– The appearance of a cluster can change greatly with viewing

position• Shading highlights will change rapidly

• Bigger clusters near the root contain many objects that may show significant disparity

– Solution: Generate the necessary textures at run-time

• The problem is easier for special cases…

03/31/03 © 2003 University of Wisconsin

Urban Scenery(Sillion, Drettakis and Bodelet, 1997)

• Dense cities have two useful properties when viewed from the ground:– The view location is highly constrained to be on streets

– When standing on a street, the only distant views are through the ends of the street

• Place an impostor at the ends of each street– Each impostor is a textured 3D mesh

• At run-time, render the local model (the street and its buildings) and an impostor (showing everything else)

• Additional space cost is linear in the number of streets

• Rendering is much faster, because a relatively constant, small number of polygons are drawn for each frame

03/31/03 © 2003 University of Wisconsin

Impostor Construction

03/31/03 © 2003 University of Wisconsin

Urban Impostors Details

• Impostors are generated in a pre-processing stage– Render the view from the center of each street out each end

– Take the depth map, and triangulate it while attempting to preserve major discontinuities

– Use the image as a texture for the triangle mesh

• Remaining problems:– Transitions at intersections are problematic

• At an intersection, the relevant impostors are most incorrect

– 3D mesh impostors stretch over what should be holes

– More impostors per street are required for accurate occlusion effects

03/31/03 © 2003 University of Wisconsin

Urban Impostors - Results

03/31/03 © 2003 University of Wisconsin

Urban Impostors - Results

03/31/03 © 2003 University of Wisconsin

Multi-Layered Impostors(Decoret, Schaufler, Sillion and Dorsey, 1999)

• Extend urban impostors to use multiple impostors per street

• Objects visible through the end of the street are grouped into layers– Grouping uses the maximum visible distance between two points to

identify candidates for each group (nearby objects are grouped)

• At run-time, try to exploit free time by re-rendering the layers from the current viewpoint– Produces much better images if the viewer stays still

– Allows for poorer (and cheaper) basic impostors, because the viewer won’t see them for long

03/31/03 © 2003 University of Wisconsin

Multi-Layer Impostors - Results

03/31/03 © 2003 University of Wisconsin

Coherent Layers(Lengyel and Snyder, 1997)

• Hand animators use layers to reduce the number of cells to draw– One layer for background, one for middle ground, one for character,…

– Background layers need to be changed less frequently than foreground, slow moving less frequently than fast moving,…

– Layers are composited as a final step

• Coherent layers was designed to work with hardware that supports fast compositing and layer warping

• Approach:– Break scene into layers by hand

– At run-time, warp some layers, re-render others

– Composite the layers into the frame buffer (back to front)

03/31/03 © 2003 University of Wisconsin

Chicken Run (80 layers)

03/31/03 © 2003 University of Wisconsin

Other Examples

03/31/03 © 2003 University of Wisconsin

Image Caching (1)(Schaufler and Sturzlinger, 1996)

• Hierarchically subdivide world into boxes (kd-tree)• Replace boxes with a textured rectangles at run-time

– Given current view, render the box contents onto the rectangle– If not a leaf, render child boxes and use impostors to render parent

• Textures are warped in subsequent frames– Estimate error in warped image by considering maximum disparity between

points in the box– Efficiency depends on the viewer rate of motion

• Hardware issues:– Assumes hardware clipping planes – not unusual– Reading frame buffer is not typically fast (some machines put the frame

buffer in main memory – faster)– Cannot be multi-threaded (everyone needs the frame buffer)

03/31/03 © 2003 University of Wisconsin

Image Caching Example

03/31/03 © 2003 University of Wisconsin

Image Caching (2)(Shade, Lischinski, Salesin, DeRose, Snyder, 1996)

• Same idea as previous paper, but superior implementation• Build BSP hierarchy (actually a kd-tree in this paper)

– Look for cuts that produce balanced trees– Look for cuts that produce near-square boxes– Look for cuts that don’t split objects (prevents cracking)– Grow each box by 10%-20% to avoid cracks

• Use a predictive approach to decide whether it is worth producing a texture map– Viewer’s motion is assumed bounded (not unreasonable)– Uses an off-line estimation of the costs of rendering the geometry

and creating the texture maps

03/31/03 © 2003 University of Wisconsin

Open Problems in IBR

• Dynamic IBR! Design an IBR algorithm that is efficient for scenes that change– Image based approaches cannot handle moving objects in the

background

– Other level-of-detail methods fail too

03/31/03 © 2003 University of Wisconsin

Layered Depth Images(Shade, Gortler, He and Szeliski, 1998)

• Store more than one color value, with depths, at each pixel

• Several ways to generate layered image– Standard ray tracer sampling different rays and combining results

– Modified ray tracer returning multiple hits per pixel

– Vision techniques from multiple views

• Rendering uses a splatting operation to push pixels onto the image plane– Render pixels in a specific order (McMillan and Bishop, 1995)

– Project depth forward then back to determine the splat size

• Restricted viewing range for each image– Must combine several images to cover a large area (not done in paper)