Jan. 19, 1999 CS260 Winter 1999-Wittenbrink, lect. 5 1 CS 260 Computer Graphics Craig M. Wittenbrink Lecture 5: Image Based Rendering Techniques: Shade

Jan. 19, 1999CS260 Winter 1999-Wittenbrink, lect. 51

CS 260 Computer Graphics

Craig M. WittenbrinkLecture 5: Image Based Rendering Techniques: Shade et al., Wittenbrink et al.

CS260 Winter 1999-Wittenbrink, lect. 5 2Jan. 19, 1999

Overview

• Intro to Warping• Shade et al.

– Sprites with depth– Layered depth images

• SIGGRAPH’98 Videos• Wittenbrink et al.

– Karhunen-Loeve Transform– 2D texture mapping from satellite images

• Conclusions


Intro to Warping

• Geometric Transformations– Castelman, Chapter 8, page 115– Gonzalez and Woods, Section 5.9, page 296– G. Wolberg. Digital Image Warping, Computer

Society Press, 1988.– Wittenbrink and Somani, 2D&3D Optimal Parallel

Image Warping• Two aspects:

– 1) Algorithm that defines spatial transformation for image pixels

– 2) Algorithm for gray-level interpolation

y

xT

y

x


Warping cont.

• Spatial transformation examples

• Rotation(counter)

• Translation

• Scaling (shrink)– (expand)

y

x

y

x

cossin

sincos

1100

10

01

1

y

x

t

t

y

x

y

x

1100

00

00

1

y

x

s

s

y

x

y

x

Homogeneouscoordinate


Warping cont.

• Numerous methods to calculate include:– Forwards (Pixel Carryover)[splatting]– Backwards (Pixel Filling)[ray casting]

• Backwards is often more practicalT

T-1

Input imageOutput image

T

T-1

Interpolatefrom surroundingpixels


Warping/Interpolation

• Types of interpolation• 1) zero-order, or nearest neighbor• 2) first-order, or bilinear

dcxybyaxyxf ),(

Bilinear term

Zero-order hold bilinear

quadratic cubic

Zero-order holdBilinear


Shade et al.

• Goal is to create parallax and remove disocclusions

• Sprites with depth– Improve over just affine warps

• Layered-Depth Images (LDI)– faster than Max


Shade et al./ Texture mapping as IBR?

• Texture mapping is the most familiar image-based rendering method

• Two main challenges with texture mapping– 1) sample rate of texture and scene are vastly

different– 2) texture mapping is compute and bandwidth

intensive• Implications

– 1) Antialiasing for textures is difficult– 2) Texture mapping is key focus for hardware 3D

graphics design


Shade et al. / Big Picture of IBR

• Grab bag of Image-based rendering primitives (from back to front, see Fig. 1)– 1) Environment map– 2) Planar sprites– 3) Sprites with depth– 4) Layered-depth image (LDI)– 5) Polygons

Camera Center forLDI and Sprites withdepth

Viewing Region


Shade et al. / Epipolar Geometry

• Output camera and input camera define epipolar point

• Back-to-front ordering defined

Epipolar Point

Layered DepthImageCamera

OutputCamera


Shade et al. / Sprites with Depth, Transformations

• 3D Plane equation computed for sprite Z

• d1 is scaled perpendicular distance to plane• Add a displacement per pixel of sprite,

resulting transfer function transform sprite coord to output image coord

1

ˆ1

1

11

11

11

Z

Y

X

C

w

dw

yw

xw

Note: d1 for z

2,111

1

2,1

2

22

22

1

edy

x

H

w

yw

xw

Eq 2

Eq 4


Sprite with depth, cont.transformation from sprite to image

• d1’s - depth with sprite• H1,2 - 2D planar perspective transformation• (x2,y2) - coordinate in output camera image• (x1,y1) - sprite’s coordinates• e1,2 - epipole (from 3rd column of T1,2)

2,111

1

2,1

2

22

22

1

edy

x

H

w

yw

xw

Layered DepthImageCamera

OutputCamera

sprite

)1,,,( 111 dyx

2,1e

2,1T

),( 22 yx

1122,1

ˆˆ CCT


Forward Mapping/Splatting

• Eq 4, forward map sprite pixels to output image

• Or, Interchange 1 & 2 indices– both cameras are “views” and epipole is the same

• Now forward map displacements, and use EQ 4 with interchanged indices to do backward mapping with new view based displacements

2,111

1

2,1

2

22

22

1

edy

x

H

w

yw

xw


Factoring warping equation

• Create intermediate space between d1 (sprite) and d2 (output image) or d3

d1’s, sprite

d3’s forward warped

d2’s backward warp

C2, backwards warpfor colors(EQ 4 reversed)

(EQ6)(EQ5)

Output image

Sprite2,1

*11

1

3

33

33

1

edy

x

w

yw

xw

13

3

2,1

2

22

22

y

x

H

w

yw

xw

1,222

2

1,2

1

11

11

1

edy

x

H

w

yw

xw

Step 1, forward map

Step 2a, backward map

Step 2b, backward map


Factoring warping equationMore efficient

• Shortcut d3 conversion to d2• And, can be even faster by skipping d1 to d3

transformd1’s, sprite

d3’s forward warped

d2’s backward warp

(EQ6)(EQ5)

Output image

Sprite(x2,y2)

1,233333

33

1

11

11

),(

1

eyxdyw

xw

w

yw

xw

1,211133

33

1

11

11

),(

1

eyxdyw

xw

w

yw

xw

shortcut

Even faster (approximate)


Warping by lookup table

• Can precompute warp factor and store in lookup table

),(

),(

333

333

3

3

1

1

yxv

yxu

y

x

y

x Eq 9


Sprites with depth results

No parallax

Incorrectusing d1

Correctd3 used

Better gapfilling

Lessslantedpyramid

Perspectiveprojection

Pure parallax

Both parallaxand perspective

Original colorand depth

Reproduced from Shade et al. Copyright SIGGRAPH’98


Extracted Sprites with DepthInteractive 3DGraphics Symposium

• 5 images used

Fig. 4 reproducedfrom Shade et al.Copyright SIGGRAPH’98and Dayton Taylor

1 of 5images

6 layer segmentation Recovered depth map

5 layer sprites Residual depthimage for fifthlayer

Resynthesized3rd image

2 novel views


Extracted Sprites

• Novel view with residual depth

Fig 4 h

Fig. 4 reproducedfrom Shade et al.Copyright SIGGRAPH’98and Dayton Taylor


Layered Depth Images (LDI’s)

• Attempt to handle more disocclusions and large amounts of parallax

• contains potentially multiple depth pixels per pixel location

• Farther pixels help to fill holes (disocclusions)• Use linked during construction, and packed for

rendering

Fig. 5 reproduced from Shadeet al. Copyright SIGGRAPH ‘98


Creation of LDI’s

• 1a)Use synthetic ray tracer that provides depth per pixel

• 1b) Or scan conversion and read z-buffer– Choose one camera position as LDI camera and

warp images to that camera• 2)Use Less regular sampling with ray tracer

– which rays to choose?• 3)Or use Computer Vision from multiple

images– Modified Seitz and Dyer algorithm, view centered

voxelization


Reconstruction

• Common events:• 1) disocclusions as viewpoint changes• 2) surfaces that cover large areas of the screen

• Define an LDI from each cube face (they don’t appear to actually do this)

Cube of possible newviewpoints


Creation with ray caster cont.

• Parametrize the rays• Use cosine weighted direction over the

hemisphere• use Stratified Stochastic sampling• divide uniformly into NxNxNxN strata• for each stratum cast m rays• N=32, m=16, gives 32^4*16=16million


Sampling with ray caster

• Main point, get lots of rays to cover many viewpoints

• Rays are in all directions


LDI from real images (not that you could tell ;) )

• Seitz and Dyer dinosaur toy


Rendering Layered Depth Images

• Splatting used• Space efficient representation

– pack LDI, bottom-to-top, left-to-right in screen space, and back-to-front along ray

– Store offsets for fast access– 1) beginning of scanline– 2) pixel in scanline


Incremental Warping Computation

• Given 4x4 matrix for LDI view• and 4x4 matrix for desired new view• You can transform a point in the LDI view to

the world coordinates, and then to the new view

11

1

1

112

2

22

22

22

z

y

x

CC

w

wz

wy

wx

C1C2

aAworld

11C

2C

a2


Incremental Warping

• Reuse matrix results by factoring

• New start is simply incremented

depthzstartTzy

x

Tz

y

x

CC

w

wz

wy

wx

12,111

1

2,11

1

1

112

2

22

22

22

0

1

0

0

1

0

1

xincrstartTy

x

Ty

x

Tnewstart

0

0

0

1

1

0

1

0

1

2,11

1

2,11

1

2,1


Rendering of LDI

• For each pixel– For number of layers

– result=start +z1*depth (location)– clip either behind camera or out of frustum– splat (pick appropriate splat size)– increment for next pixel on scanline

• Splat size chosen by projected pixel area approximiation

– Put approximations in lookup table],,[ 12 dnnlookupzsize yx


LDI Rendering Results

LDI’s max 10 layers per pixel, 1.24 average depth complexity300x300 resolution, at 8-10 frames/second on Pentium II, 300 MHz



Cross eyed stereo pairs. LDI’s from Rayshade raytracer. LDI has 1.1 million depthpixels. 4-10 frames/second on Pentium II @300MHz



Cross eyed stereo pairs. LDI’s from Rayshade raytracer. LDI has 1.1 million depthpixels. 4-10 frames/second on Pentium II @300MHz


SIGGRAPH’98 VideosImage Based Rendering (24 min.)

• 9: The Office of the Future: …Raskar, Fuchs et al.

• 10: Rendering Synthetic Objects Into Real Scenes: … Debevec

• 11: Multiple-Center-of-Projection Images, Rademacher and Bishop

• 12: Recovering Photometric Properties of Architectural Scenes from Photographs, Uy and Malik


SIGGRAPH’98 VideosImage Based Rendering (cont.)

• 13: Visibility Sorting and Compositing Without Splitting for Image Layer Decomposition, Snyder and Lengyel

• 14: Layered Depth Images, Shad, Gortler, He, and Szeliski


Wittenbrink et al.

• SPIE’96 35 mm slides• Feature extraction of clouds from GOES satellite data for integrated

model measurement visualization, Craig M. Wittenbrink, Glen Langdon, Jr., and Gabriel Fernandez, In Proceedings of IS&T/SPIE Symposium on Electronic Imaging: Image and Video Processing IV 1996, Vol. 2666, pages 212-222, R. Stevenson and M.I. Sezan, San Jose, CA 1996.

• www.cse.ucsc.edu/research/slvg/cloud.html


Conclusions

• Shade et al.– Sprites with depth– Layered depth images

• SIGGRAPH’98 Videos: • Wittenbrink et al.

– Karhunen-Loeve Transform– 2D texture mapping from satellite images

• Next time: Seitz and Dyer “Photorealistic Scene Reconstruction by Voxel Coloring” CVPR’97

• Conclusions

Documents

Jan. 19, 1999 CS260 Winter 1999-Wittenbrink, lect. 5 1 CS 260 Computer Graphics Craig M. Wittenbrink Lecture 5: Image Based Rendering Techniques: Shade