From Global Illumination To Inverse Global Illumination

Yizhou Yu

From Global IlluminationFrom Global IlluminationTo Inverse Global IlluminationTo Inverse Global Illumination

From Global IlluminationFrom Global IlluminationTo Inverse Global IlluminationTo Inverse Global Illumination

Yizhou Yu

Computer Science Division

University of California at Berkeley

Yizhou Yu

Computer Science Division

University of California at Berkeley

Yizhou Yu

PublicationsPublicationsPublicationsPublications

• Y. Yu and H. Wu, A Rendering Equation for Specular Transfers and Its Integration into Global Illumination, Eurographics’97

• P. Debevec, Y. Yu and G. Borshukov, Efficient View-Dependent Image-Based Rendering with Projective Texture-Mapping, Eurographics Workshop on Rendering’98

• Y. Yu and J. Malik, Recovering Photometric Properties of Architectural Scenes from Photographs, Siggraph’98

• Y. Yu, P. Debevec, J. Malik and T. Hawkins, Inverse Global Illumination: Recovering Reflectance Models of Real Scenes from Photographs, Siggraph’99





http://www.cs.berkeley.edu/~yyz

Yizhou Yu

Computer Graphics & VisionComputer Graphics & VisionComputer Graphics & VisionComputer Graphics & Vision

• Graphics– Solving forward simulation

– Synthesizing images from geometry and reflectance

• Vision– Recovering geometry and reflectance

– Extracting data from images

• Graphics– Solving forward simulation

– Synthesizing images from geometry and reflectance

• Vision– Recovering geometry and reflectance

– Extracting data from images

Images Geometry & Reflectance Imagesvision graphics

Yizhou Yu

Global IlluminationGlobal IlluminationGlobal IlluminationGlobal Illumination

Reflectance Properties

Radiance Maps

Geometry Light Sources

LightTransport

Yizhou Yu

Reflectance PropertiesReflectance PropertiesReflectance PropertiesReflectance Properties

Bidirectional Reflectance Distribution Function(BRDF)(wavelength dependent)

i

rri dx

dxddx

along at Light Incident

along at Light Reflected),,(

Incident light

Diffuse

Specular

Yizhou Yu

GeometryGeometryGeometryGeometry

• A polygonal mesh and/or a set of curved surface patches

• A polygonal mesh and/or a set of curved surface patches

Yizhou Yu

Light SourcesLight SourcesLight SourcesLight Sources

• 3D positions and directional radiance distributions

• 3D positions and directional radiance distributions

Yizhou Yu

Light TransportLight TransportLight TransportLight Transport

rd

id

x

The Rendering Equation [ Kajiya’86 ]

ddxLddxdxLdxL iiirirerr cos),(),,(),(),(

Yizhou Yu

Example of Rendering Using Global Example of Rendering Using Global IlluminationIlluminationExample of Rendering Using Global Example of Rendering Using Global IlluminationIllumination

With a mirror Without the mirror

Yu & Wu [ Eurographics’97 ] use bi-directional wavefront tracingto calculate illumination from area sources via curved ideal specular reflectors.

Yizhou Yu

A ComparisonA ComparisonA ComparisonA Comparison

Yizhou Yu

The ProblemThe ProblemThe ProblemThe Problem

• The physics of light transport has been well understood.

• In the absence of real-world geometry and reflectance, rendered images still look synthetic.

• Solution: Image-based Modeling and Rendering (IBMR)

• The physics of light transport has been well understood.

• In the absence of real-world geometry and reflectance, rendered images still look synthetic.

• Solution: Image-based Modeling and Rendering (IBMR)

Yizhou Yu

Image-based Modeling and RenderingImage-based Modeling and RenderingImage-based Modeling and RenderingImage-based Modeling and Rendering

• 1st Generation----vary viewpoint but not lighting– Recover geometry ( explicit or implicit )

– Acquire texture maps

– Facade, Virtualized Reality, View Morphing, Plenoptic Modeling etc.

• 1st Generation----vary viewpoint but not lighting– Recover geometry ( explicit or implicit )

– Acquire texture maps

– Facade, Virtualized Reality, View Morphing, Plenoptic Modeling etc.

Yizhou Yu

Input Photographs and Geometric ModelInput Photographs and Geometric ModelInput Photographs and Geometric ModelInput Photographs and Geometric Model

Yizhou Yu

Synthetic RenderingsSynthetic RenderingsSynthetic RenderingsSynthetic Renderings

Yizhou Yu

A Synthetic Sunrise SequenceA Synthetic Sunrise SequenceA Synthetic Sunrise SequenceA Synthetic Sunrise Sequence

5:00am 5:30am 6:00am 6:30am

7:00am 8:00am 9:00am 10:00am

One Day at the End of March

Yizhou Yu

The ProblemThe ProblemThe ProblemThe Problem

• Texture Maps are not Reflectance Maps !

• Need to factorize images into lighting and reflectance maps

• Texture Maps are not Reflectance Maps !

• Need to factorize images into lighting and reflectance maps

Illumination Radiance

Reflectance

Yizhou Yu

Image-based Modeling and RenderingImage-based Modeling and RenderingImage-based Modeling and RenderingImage-based Modeling and Rendering

• 2nd Generation----vary viewpoint and lighting– Recover geometry

– Recover reflectance maps

– Permits rendering using physically based light transport methods

• 2nd Generation----vary viewpoint and lighting– Recover geometry

– Recover reflectance maps

– Permits rendering using physically based light transport methods

Yizhou Yu

Outline of the Rest of the TalkOutline of the Rest of the TalkOutline of the Rest of the TalkOutline of the Rest of the Talk

• 1st Generation IBMR– Real-Time View-Dependent Projective Texture-

Mapping

• 2nd Generation IBMR– General Problem: closely positioned multiple objects

– Simplified Situation: Isolated Objects


Mapping



Yizhou Yu

OutlineOutlineOutlineOutline

• 1st Generation IBMRReal-Time View-Dependent Projective Texture-

Mapping



• 1st Generation IBMRReal-Time View-Dependent Projective Texture-

Mapping



Yizhou Yu

Real-Time View-Dependent Texture Real-Time View-Dependent Texture MappingMappingReal-Time View-Dependent Texture Real-Time View-Dependent Texture MappingMapping

• VDTM was originally from Façade [ Debevec, Taylor & Malik, Siggraph’96].– Software implementation

– 10 seconds per frame

• Real-Time VDTM– Software object-space visibility preprocessing +

hardware projective texture-mapping

– 20 frames per second on SGI RealityEngine

– 60 frames per second on SGI Onyx2 InfiniteReality

• VDTM was originally from Façade [ Debevec, Taylor & Malik, Siggraph’96].– Software implementation

– 10 seconds per frame

• Real-Time VDTM– Software object-space visibility preprocessing +

hardware projective texture-mapping

– 20 frames per second on SGI RealityEngine

– 60 frames per second on SGI Onyx2 InfiniteReality

Yizhou Yu

Motivation for Visibility Processing: Motivation for Visibility Processing: Artifacts Caused by HardwareArtifacts Caused by HardwareMotivation for Visibility Processing: Motivation for Visibility Processing: Artifacts Caused by HardwareArtifacts Caused by Hardware

Camera

Image

Geometry

Yizhou Yu

Visibility Processing ResultsVisibility Processing ResultsVisibility Processing ResultsVisibility Processing Results

The tower The rest of the campus

Yizhou Yu

VideoVideoVideoVideo

Yizhou Yu



Mapping

• 2nd Generation IBMRGeneral Problem: closely positioned multiple objects



Mapping

• 2nd Generation IBMRGeneral Problem: closely positioned multiple objects


Yizhou Yu

Previous WorkPrevious WorkPrevious WorkPrevious Work

• BRDF Measurement in the Laboratory– [ Ward 92 ], [Dana, Ginneken, Nayar & Koenderink 97]

• Isolated Objects under Direct Illumination– [ Sato, Wheeler & Ikeuchi 97 ]

• BRDF Measurement in the Laboratory– [ Ward 92 ], [Dana, Ginneken, Nayar & Koenderink 97]

• Isolated Objects under Direct Illumination– [ Sato, Wheeler & Ikeuchi 97 ]

General case of multiple objects under mutual illumination has not been studied.

Yizhou Yu

Image-based Reflectance RecoveryImage-based Reflectance RecoveryImage-based Reflectance RecoveryImage-based Reflectance Recovery

• Start from photographs

• Recover geometric model

• Measure and/or recover illumination

• Recover parametric models for reflectance

• Design or Predict Novel illumination

• Re-render the scene

• Start from photographs

• Recover geometric model

• Measure and/or recover illumination

• Recover parametric models for reflectance

• Design or Predict Novel illumination

• Re-render the scene

Yizhou Yu

Global IlluminationGlobal IlluminationGlobal IlluminationGlobal Illumination


Radiance Maps


Yizhou Yu

Inverse Global IlluminationInverse Global IlluminationInverse Global IlluminationInverse Global Illumination


Radiance Maps


Yizhou Yu

Input ImagesInput ImagesInput ImagesInput Images

Yizhou Yu

In Detail ... In Detail ... In Detail ... In Detail ...

Yizhou Yu

Geometry Recovered Using FacadeGeometry Recovered Using FacadeGeometry Recovered Using FacadeGeometry Recovered Using Facade

Yizhou Yu

Synthesized Images of the Room under Synthesized Images of the Room under Original and Novel LightingOriginal and Novel LightingSynthesized Images of the Room under Synthesized Images of the Room under Original and Novel LightingOriginal and Novel Lighting

Yizhou Yu

IGI OutlineIGI OutlineIGI OutlineIGI Outline

• IGI for Lambertian surfaces

• IGI for isolated specular surface

• IGI for general surfaces

• Computing diffuse albedo maps

• Results

• IGI for Lambertian surfaces

• IGI for isolated specular surface

• IGI for general surfaces

• Computing diffuse albedo maps

• Results

Yizhou Yu

Inverse Radiosity with Lambertian Inverse Radiosity with Lambertian SurfacesSurfacesInverse Radiosity with Lambertian Inverse Radiosity with Lambertian SurfacesSurfaces

nj

ijjiii FBEB1

nj

ijjiii FBEB1

• Bi, Bj, Ei measured using HDR photographs

• Fij known because geometry is known

• Solve for diffuse albedo

• Bi, Bj, Ei measured using HDR photographs

• Fij known because geometry is known

• Solve for diffuse albedo i

Cv

Ck

Aj

Pi

LPiAj

LCkAj

LCvPi

Yizhou Yu

Recovering Specular Properties from Recovering Specular Properties from Direct IlluminationDirect IlluminationRecovering Specular Properties from Recovering Specular Properties from Direct IlluminationDirect Illumination

2

1

))((min iisimi

di IrKIrL

2

1

))((min iisimi

di IrKIrL

• Specular Kernel Ki as in [ Ward 92 ]• Specular Kernel Ki as in [ Ward 92 ]

NH

Yizhou Yu

Parametric BRDF Model [ Ward 92 ]Parametric BRDF Model [ Ward 92 ]Parametric BRDF Model [ Ward 92 ]Parametric BRDF Model [ Ward 92 ]

)(

Ksd

2

22

4

]/tan[exp

coscos

1)(

ri

K

yx

yx

ri

K

4

)]/sin/cos(tan[exp

coscos

1)(

22222

Isotropic Kernel

Anisotropic Kernel

NHi

r

Yizhou Yu

Recovering Diffuse and Specular Recovering Diffuse and Specular Reflectance under Mutual IlluminationReflectance under Mutual IlluminationRecovering Diffuse and Specular Recovering Diffuse and Specular Reflectance under Mutual IlluminationReflectance under Mutual Illumination

• Specular component of LPiAj is not known. ( unlike diffuse case, where LPiAj = LCkAj )

• Specular component of LPiAj is not known. ( unlike diffuse case, where LPiAj = LCkAj )

nj nj

APCAPsAPAPdPC jivjijijiivsd

KLFLL1 1

2

,,)( min

nj nj

APCAPsAPAPdPC jivjijijiivsd

KLFLL1 1

2

,,)( min

Cv

Ck

Aj

Pi

LPiAj

LCkAj

LCvPi

Yizhou Yu

Solution: iteratively estimate specular Solution: iteratively estimate specular component.component.Solution: iteratively estimate specular Solution: iteratively estimate specular component.component.

jikjkji APCACAP SLL jikjkji APCACAP SLL

• Initialize

• Repeat– Estimate BRDF parameters for each surface

– Update and

• Initialize

• Repeat– Estimate BRDF parameters for each surface

– Update and

0jik APCS 0jik APCS

jik APCS jik APCS jiAPL

Yizhou Yu

Estimation of SEstimation of SEstimation of SEstimation of S

• Estimate specular component of by ray-tracing using current guess of reflectance parameters.

• Similarly for

• Difference gives S

• Currently we use one-bounce approximation, but could be generalized.

• Estimate specular component of by ray-tracing using current guess of reflectance parameters.

• Similarly for

• Difference gives S

• Currently we use one-bounce approximation, but could be generalized. Cv

Ck

Aj

Pi

LPiAj

LCkAj

LCvPi

LPiAj

LCkAj

Yizhou Yu

Inverse Global IlluminationInverse Global IlluminationInverse Global IlluminationInverse Global Illumination

• Detect specular highlight blobs on the surfaces.• Choose a set of sample points inside and around each highlight area.• Build hierarchical links between sample points and facets in the

environment and use ray tracing to detect occlusion.• Assign to each facet one photograph and one average radiance value

captured at the camera position.• Assign zero to Delta_S at each hierarchical link.• For iter = 1 to n

– For each hierarchical link, use its Delta_S to update its radiance value.– For each surface having highlight areas, optimize its BRDF parameters.– For each hierarchical link, estimate its Delta_S with the new BRDF

parameters.• End

• Detect specular highlight blobs on the surfaces.• Choose a set of sample points inside and around each highlight area.• Build hierarchical links between sample points and facets in the

environment and use ray tracing to detect occlusion.• Assign to each facet one photograph and one average radiance value

captured at the camera position.• Assign zero to Delta_S at each hierarchical link.• For iter = 1 to n

– For each hierarchical link, use its Delta_S to update its radiance value.– For each surface having highlight areas, optimize its BRDF parameters.– For each hierarchical link, estimate its Delta_S with the new BRDF

parameters.• End

Yizhou Yu

Recovering Diffuse Albedo MapsRecovering Diffuse Albedo MapsRecovering Diffuse Albedo MapsRecovering Diffuse Albedo Maps

• Estimate specular component at each pixel from the recovered BRDF parameters using Monte Carlo ray-tracing.

• Subtract specular component to get the diffuse component of radiance, Ld(x).

• Gather irradiance, Ir(x), using Form-Factors over each surface.

•

• Combine from multiple photographs by robust weighted average.

• Estimate specular component at each pixel from the recovered BRDF parameters using Monte Carlo ray-tracing.

• Subtract specular component to get the diffuse component of radiance, Ld(x).

• Gather irradiance, Ir(x), using Form-Factors over each surface.

•

• Combine from multiple photographs by robust weighted average.

)(/)()( xIrxLx dd )(xd

Yizhou Yu

Results for a Simulated Cubical Room: Results for a Simulated Cubical Room: IIResults for a Simulated Cubical Room: Results for a Simulated Cubical Room: II

Diffuse Albedo

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

1 2 3 4 5 6

Real

Recovered

Yizhou Yu

Results for a Simulated Cubical Room: Results for a Simulated Cubical Room: IIIIResults for a Simulated Cubical Room: Results for a Simulated Cubical Room: IIII

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

1 2 3 4 5 6

Real

Recovered

Specular Roughness

Yizhou Yu

Real vs. Synthetic for Original Lighting Real vs. Synthetic for Original Lighting Real vs. Synthetic for Original Lighting Real vs. Synthetic for Original Lighting

Yizhou Yu

Diffuse Albedo Maps of Identical Posters Diffuse Albedo Maps of Identical Posters in Different Parts of the Roomin Different Parts of the RoomDiffuse Albedo Maps of Identical Posters Diffuse Albedo Maps of Identical Posters in Different Parts of the Roomin Different Parts of the Room

Yizhou Yu

Inverting Color BleedInverting Color BleedInverting Color BleedInverting Color Bleed

Input Photograph Output Albedo Map

Yizhou Yu

Real vs. Synthetic for Novel LightingReal vs. Synthetic for Novel LightingReal vs. Synthetic for Novel LightingReal vs. Synthetic for Novel Lighting

Yizhou Yu

VideoVideoVideoVideo

Yizhou Yu

ContributionsContributionsContributionsContributions• A digital camera is the only data acquisition

equipment used.

• Adopt an iterative procedure to obtain radiance distributions from specular surfaces.

• Exploit spatial coherence to recover specular reflectance models from one single photograph.

• Make use of multiple photographs to recover high-resolution diffuse albedo maps.

• A digital camera is the only data acquisition equipment used.

• Adopt an iterative procedure to obtain radiance distributions from specular surfaces.

• Exploit spatial coherence to recover specular reflectance models from one single photograph.

• Make use of multiple photographs to recover high-resolution diffuse albedo maps.

Yizhou Yu



Mapping

• 2nd Generation IBMR– General Problem: closely positioned multiple objectsSimplified Situation: Isolated Objects


Mapping

• 2nd Generation IBMR– General Problem: closely positioned multiple objectsSimplified Situation: Isolated Objects

Yizhou Yu

Some Input PhotographsSome Input PhotographsSome Input PhotographsSome Input Photographs

Yizhou Yu

Modeling the IlluminationModeling the IlluminationModeling the IlluminationModeling the Illumination

• The sun– Its diameter extends 31.8’ seen from the earth.

• The sky– A hemispherical area light source.

• The surrounding environment– May contribute more light than the sky on shaded side.

– Modeled as a set of oriented Lambertian facets.

• The sun– Its diameter extends 31.8’ seen from the earth.

• The sky– A hemispherical area light source.

• The surrounding environment– May contribute more light than the sky on shaded side.

– Modeled as a set of oriented Lambertian facets.

Yizhou Yu

A Recovered Sky Radiance ModelA Recovered Sky Radiance ModelA Recovered Sky Radiance ModelA Recovered Sky Radiance Model

R,G,B channels

Yizhou Yu

Coarse-grain Environment Radiance MapsCoarse-grain Environment Radiance MapsCoarse-grain Environment Radiance MapsCoarse-grain Environment Radiance Maps

• Partition the lower hemisphere into small regions

• Take photographs at several times of day

• Project pixels into regions and obtain the average radiance

• Partition the lower hemisphere into small regions

• Take photographs at several times of day

• Project pixels into regions and obtain the average radiance

Yizhou Yu

Predicting Novel Illumination from the Predicting Novel Illumination from the EnvironmentEnvironmentPredicting Novel Illumination from the Predicting Novel Illumination from the EnvironmentEnvironment

• Use photometric stereo to recover a Lambertian facet model for each region

• Use photometric stereo to recover a Lambertian facet model for each region

Synthetic Real

Yizhou Yu

Comparison with Real PhotographsComparison with Real PhotographsComparison with Real PhotographsComparison with Real Photographs

Synthetic Real

Yizhou Yu

Video Video Video Video

Yizhou Yu

Facial Skin Reflectance and WrinklesFacial Skin Reflectance and WrinklesFacial Skin Reflectance and WrinklesFacial Skin Reflectance and Wrinkles

Yizhou Yu

AcknowledgmentsAcknowledgmentsAcknowledgmentsAcknowledgments

• This is joint work with Jitendra Malik, Paul Debevec, George Borshukov, Tim Hawkins, C.J. Taylor and Hong Wu.

• Supported by ONR BMDO, the California MICRO program, Philips Corporation, Interval Research Corporation and Microsoft Graduate Fellowship.

• This is joint work with Jitendra Malik, Paul Debevec, George Borshukov, Tim Hawkins, C.J. Taylor and Hong Wu.

• Supported by ONR BMDO, the California MICRO program, Philips Corporation, Interval Research Corporation and Microsoft Graduate Fellowship.

Yizhou Yu

PublicationsPublicationsPublicationsPublications









http://www.cs.berkeley.edu/~yyz

Yizhou Yu

Yizhou Yu

Optimization TechniqueOptimization TechniqueOptimization TechniqueOptimization Technique

Isotropic Kernel ( Golden Section Search )

Anisotropic Kernel ( Simplex Search )

2

1

))()()(

( min iisimi

di IrKIrL

2

1

))()()(

( min iisimi

di IrKIrL

2xx

1

x

,,)),,(),,(

),,(( min

xiyiysi

mi

ydi IrKIrL

y

2xx

1

x

,,)),,(),,(

),,(( min

xiyiysi

mi

ydi IrKIrL

y

Yizhou Yu

Efficiency ConsiderationEfficiency ConsiderationEfficiency ConsiderationEfficiency Consideration

Indirect illumination is computed only at a sparse set of points and then linearly interpolated.

Yizhou Yu

Recovering Sky Radiance ModelRecovering Sky Radiance ModelRecovering Sky Radiance ModelRecovering Sky Radiance Model

) cos ) exp( 1 ))( /cos exp( 1 Lvz( 2 edcba f ) cos ) exp( 1 ))( /cos exp( 1 Lvz( 2 edcba f

• Recover a set of parameters for each color channel– Take photographs for parts of the sky

– Use Levenberg-Marquardt algorithm to fit data

• Recover a set of parameters for each color channel– Take photographs for parts of the sky

– Use Levenberg-Marquardt algorithm to fit data

sun

zenithSky element

Lvz, a, b, c, d, e, f

based on [Perez 93]

Yizhou Yu

A Local Facet Model for the EnvironmentA Local Facet Model for the EnvironmentA Local Facet Model for the EnvironmentA Local Facet Model for the Environment

• Recover a distinct model for each environment region– Obtain environment radiance maps.

– Set up over-determined systems as in photometric stereo and ignore inter-reflections.

– Solve for

• Recover a distinct model for each environment region– Obtain environment radiance maps.

– Set up over-determined systems as in photometric stereo and ignore inter-reflections.

– Solve for

otherwise.

,0n if

,

),n(

envsun

skysky

envsunsunsun

skysky

env

l

E

lEEI

otherwise.

,0n if

,

),n(

envsun

skysky

envsunsunsun

skysky

env

l

E

lEEI

envsunsky n ,,

nenvlsun

Yizhou Yu

Diffuse Pseudo-Albedo MapsDiffuse Pseudo-Albedo MapsDiffuse Pseudo-Albedo MapsDiffuse Pseudo-Albedo Maps

For the sky For the sun

Yizhou Yu

A Hybrid Visibility AlgorithmA Hybrid Visibility AlgorithmA Hybrid Visibility AlgorithmA Hybrid Visibility Algorithm

• Occlusion testing in image-space using Z-buffer hardware– Render polygons using their identifiers as their colors

– Retrieve occluding polygons’ ids from color buffer

• Object-space shallow clipping to generate fewer polygons

• Occlusion testing in image-space using Z-buffer hardware– Render polygons using their identifiers as their colors

– Retrieve occluding polygons’ ids from color buffer

• Object-space shallow clipping to generate fewer polygons

Yizhou Yu

Camera Radiance Response CurveCamera Radiance Response CurveCamera Radiance Response CurveCamera Radiance Response Curve

• Pixel brightness value is a nonlinear function of radiance.– Debevec & Malik[Siggraph’97]

give a method to recover this nonlinear mapping.

• Pixel brightness value is a nonlinear function of radiance.– Debevec & Malik[Siggraph’97]

give a method to recover this nonlinear mapping.

RadianceRadiance

IntensitySaturation

Documents

From Global Illumination To Inverse Global Illumination