CSL 859: CSL 859: Advanced Advanced Computer Computer GraphicsGraphicsDept of Computer Sc. & Engg.Dept of Computer Sc. & Engg.

IIT DelhiIIT Delhi

Image-Based RenderingImage-Based Rendering

So far:So far: Geometry -> imagesGeometry -> images Object space model, even volumetricObject space model, even volumetric

Image-based renderingImage-based rendering:: Image -> Another imageImage -> Another image

Zoom, Pan etc.Zoom, Pan etc. Just image processing?Just image processing?

Images with depthImages with depth

Quicktime VR:Quicktime VR: 2D panoramic photograph2D panoramic photograph Spin around, zoom in and outSpin around, zoom in and out Can add objects closer to viewerCan add objects closer to viewer

Tour into the pictureTour into the picture Assign depth to parts of the imageAssign depth to parts of the image One might add objects hidden behind One might add objects hidden behind

some object in the imagesome object in the image Layered depth imagesLayered depth images

Image Based Rendering Image Based Rendering

Store image from every conceivable viewStore image from every conceivable view Rendering would reduce to database queryRendering would reduce to database query

Generality demand infinite sized databaseGenerality demand infinite sized database Could store enough imagesCould store enough images

Given a desired viewpoint (viewmatrix)Given a desired viewpoint (viewmatrix) Choose an image from a saved view near the Choose an image from a saved view near the

desired viewdesired view Warp the imageWarp the image Or, interpolate from nearby known viewpointsOr, interpolate from nearby known viewpoints

Ray EquationRay Equation

CorrespondenceCorrespondence

Warp x1 to x2+

General 3D WarpGeneral 3D Warp

[Courtesy L Mcmillan]

Occlusion DeterminationOcclusion Determination

Project the desired center-of-Project the desired center-of-projection onto the reference imageprojection onto the reference image

Occlusion DeterminationOcclusion Determination

Draw towards the Draw towards the projected pointprojected point Guarantees painter’s Guarantees painter’s

orderingordering Independent of the Independent of the

scene's contentsscene's contents Generalizes to non-Generalizes to non-

planar viewing planar viewing surfacessurfaces

ReconstructionReconstruction

Radiances in a SceneRadiances in a Scene

Account for all raysAccount for all rays OriginOrigin

3 dimensions3 dimensions DirectionDirection

2 dimensions2 dimensions

Space of rays is 5 dimensionalSpace of rays is 5 dimensional

PanoramaPanorama

All rays from a single point

Plenoptic FunctionPlenoptic Function

p = P(Θ, Φ, x, y, z , λ, t)

All rays from all points

Courtesy L. Mcmillan

Radiances in a Scene IIRadiances in a Scene II

Account for all raysAccount for all rays OriginOrigin

3 dimensions3 dimensions DirectionDirection

2 dimensions2 dimensions

Space of rays is 5 dimensionalSpace of rays is 5 dimensional Radiance is constant along rayRadiance is constant along ray

4 dimensional space4 dimensional space Subject to occlusionSubject to occlusion

Capturing RadiancesCapturing Radiances

Capture images from many placesCapture images from many places Camera positioningCamera positioning

Parameterize the 4D spaceParameterize the 4D space Camera position and 2D image?Camera position and 2D image?

Sample the 4D spaceSample the 4D space Coverage and sampling uniformityCoverage and sampling uniformity AliasingAliasing Too much dataToo much data

Representing Scene Representing Scene RadianceRadiance

Like texture mapLike texture map Except ray origin is not Except ray origin is not

fixedfixed Source and destination Source and destination

of ray variesof ray varies 2 coordinates (2 coordinates (uu,,vv) for ) for

ray originray origin 2 coordinates (2 coordinates (ss,,tt) for ) for

ray destinationray destinations

t

u

v

[Light-field: Hanrahan & Levoy][Light-field: Hanrahan & Levoy]

Sampling CoverageSampling Coverage

θr

θ

r

With four slabs the (r,θ) space is well covered (for an outside looking in case)

Ray Source

Ray Target

Stanford Multi-Stanford Multi-camera Arraycamera Array

640 640 ×× 480 pixels 480 pixels ××30 fps × 128 30 fps × 128 camerascameras

Synchronized timingSynchronized timing Continuous Continuous

streamingstreaming Flexible arrangementFlexible arrangement

Light Field as Array of Light Field as Array of ImagesImages

For each pixel (x, y)For each pixel (x, y) Compute rayCompute ray Map to (u,v,s,t)Map to (u,v,s,t) Look up “4D” textureLook up “4D” texture

Store as many 2D texturesStore as many 2D textures Quadri-linear interpolationQuadri-linear interpolation

Rendering of Light FieldsRendering of Light Fields

Good and BadGood and Bad

Advantages:Advantages: Simpler computation vs. traditional CGSimpler computation vs. traditional CG Cost independent of scene complexityCost independent of scene complexity Cost independent of material Cost independent of material

properties and other optical effectsproperties and other optical effects Disadvantages:Disadvantages:

Static geometryStatic geometry Fixed lightingFixed lighting High storage costHigh storage cost