Real-Time Rendering and Interaction with Complex Lighting and Materials Ravi Ramamoorthi Rendering...

Real-Time Rendering and Interaction Real-Time Rendering and Interaction with Complex Lighting and Materialswith Complex Lighting and Materials

Ravi Ramamoorthi

Rendering LaboratoryColumbia University

Intel: August 13, 2004

High quality real-time renderingHigh quality real-time rendering

Photorealism, not just more polygons

Natural lighting, materials, shadows

Interiors by architect Frank Gehry. Note rich lighting, ranging from localized sources to reflections off vast sheets of glass.

High quality real-time renderingHigh quality real-time rendering

Photorealism, not just more polygons

Natural lighting, materials, shadows

Real materials diverse and not easy to represent by simple parameteric models. Want to support measured reflectance.

Glass VaseGlass Star (courtesy Intel) Peacock feather

High quality real-time renderingHigh quality real-time rendering

Photorealism, not just more polygons

Natural lighting, materials, shadows

Natural lighting creates a mix of soft diffuse and hard shadows.

Agrawala et al. 00small area light, sharp shadows soft and hard shadows

Ng et al. 03

DemoDemo

Industrial RelevanceIndustrial Relevance

Very difficult computational problem Both preprocessing (hours to days) and real-time rendering Related to computational problems in other areas

Driving application Interactive 3D graphics major use of PCs Realism increases use, applications

Leverages high-performance processors Exploit parallelism (at pixel level), modern feature SIMD instructions, MMX and SSE instructions

ApplicationsApplications

Entertainment: Lighting design

Architectural visualization

Material design: Automobile industry

Realistic Video games

Electronic commerce

ChallengesChallenges

Illumination complexity

Material / view complexity

Transport complexity (shadows)

Putting it together

Ramamoorthi and Hanrahan, SIGGRAPH 01, 02

Environment MapsEnvironment Maps

Miller and Hoffman, 1984

Previous work: Blinn 76, Greene 86, Cabral 87, 99

Irradiance Environment MapsIrradiance Environment Maps

Incident Radiance(Illumination Environment Map)

Irradiance Environment Map

R N

AssumptionsAssumptions

Diffuse surfaces (relaxed later)

Distant illumination

No shadowing (relaxed later)

Hence, Irradiance is a function of surface normal

Computing IrradianceComputing Irradiance

Classically, hemispherical integral for each pixel

Lambertian surface is like low pass filter

Frequency-space analysis

IncidentRadiance

Irradiance

Analytic Irradiance FormulaAnalytic Irradiance Formula

Lambertian surface acts like low-pass filter

lm l lmE A LlA

2 / 3

/ 4

0

2 1

2

2

( 1) !2

( 2)( 1) 2 !

l

l l l

lA l even

l l

l0 1 2

Basri & Jacobs 01Ramamoorthi & Hanrahan 01

9 Parameter Approximation9 Parameter Approximation

-1-2 0 1 2

0

1

2

( , )lmY

xy z

xy yz 23 1z zx 2 2x y

l

m

Exact imageOrder 29 terms

RMS Error = 1%

For any illumination, average error < 3% [Basri Jacobs 01]

Real-Time RenderingReal-Time Rendering

Simple procedural rendering method (no textures) Requires only matrix-vector multiply and dot-product In software or NVIDIA vertex programming hardware

Widely used in Games (AMPED for Microsoft Xbox), Movies (Pixar, Framestore CFC, …)

( ) tE n n Mn

surface float1 irradmat (matrix4 M, float3 v) {

float4 n = {v , 1} ;

return dot(n , M*n) ;

}

Convolution: General MaterialsConvolution: General Materials

Ramamoorthi and Hanrahan 01

( , ) ( ) ,B N V L R N l l V dl

B L

Frequency: product

Spatial: integral

ij i ijB L

Spherical harmonic analysis

Natural Lighting, Realistic Materials Natural Lighting, Realistic Materials

Ramamoorthi and Hanrahan 02

ChallengesChallenges

Illumination complexity

Material (BRDF)/view complexity

Transport complexity (shadows): Relighting

Putting it together

Ng, Ramamoorthi and Hanrahan SIGGRAPH 03

Existing Fast Shadow TechniquesExisting Fast Shadow TechniquesWe know how to render very hard, very soft shadows

Sloan, Kautz, Snyder 2002

Shadows from smooth lighting (precomputed radiance transfer)

Sen, Cammarano, Hanrahan, 2003

Shadows from point-lights(shadow maps, volumes)

AssumptionsAssumptions

Static geometry

Precomputation

Real-Time Rendering (relight all-frequency effects)

1

2

3

N

P

P

P

P

11 12 11

21 22 22

31 32 3

1 2

M

M

M

MN N NM

T T TL

T T TL

T T T

LT T T

Relighting as a Matrix-Vector MultiplyRelighting as a Matrix-Vector Multiply

1

2

3

N

P

P

P

P

11 12 11

21 22 22

31 32 3

1 2

M

M

M

MN N NM

T T TL

T T TL

T T T

LT T T

Input Lighting (Cubemap Vector)

Output Image(Pixel Vector)

TransportMatrix

Relighting as a Matrix-Vector MultiplyRelighting as a Matrix-Vector Multiply

Problem DefinitionProblem Definition

Matrix is Enormous 512 x 512 pixel images 6 x 64 x 64 cubemap environments

Full matrix-vector multiplication is intractable On the order of 1010 operations per frame

How to relight quickly?

Sparse Matrix-Vector MultiplicationSparse Matrix-Vector Multiplication

Choose data representations with mostly zeroes

Vector: Use non-linear wavelet approximation on lighting

Matrix: Wavelet-encode transport rows

11 12 11

21 22 22

31 32 3

1 2

M

M

M

MN N NM

T T TL

T T TL

T T T

LT T T

Non-linear Wavelet ApproximationNon-linear Wavelet Approximation

Wavelets provide dual space / frequency locality Large wavelets capture low frequency area lighting Small wavelets capture high frequency compact

features

Non-linear Approximation Use a dynamic set of approximating functions (depends

on each frame’s lighting) By contrast, linear approx. uses fixed set of basis

functions (like 25 lowest frequency) We choose 10’s - 100’s from a basis of 24,576 wavelets

Non-linear Wavelet Light ApproximationNon-linear Wavelet Light Approximation

Wavelet Transform

1

2

3

4

5

6

N

L

L

L

L

L

L

L

2

6

0

0

0

0

0

L

L

Non-linearApproximation

Retain 0.1% – 1% terms

Non-linear Wavelet Light ApproximationNon-linear Wavelet Light Approximation

Performance: TimingPerformance: Timing

Precomputation of Transport Matrix Ray tracing: several days (depends on ray tracer) Graphics hardware: 80,000 vertex buddha = 3 hours

Rendering (2.8 GHz Pentium IV) For 100 terms, 4-6 fps RGB, 9-15 fps monochrome Relighting alone (core matrix-vector) somewhat faster Use almost full CPU perf. (implementation details later)

VideoVideo

Broader Computational RelevanceBroader Computational Relevance

Sparse matrix-vector multiplication standard numerical problem Challenging problem (giant matrices, vectors) Even more demanding: we need real-time performance Note: we sparsify vector, not matrix unlike most methods

Most computation in tight inner loop General kernel to focus optimization effort on

Broader Computational RelevanceBroader Computational Relevance

Sparse matrix-vector multiplication standard numerical problem Challenging problem (giant matrices, vectors) Even more demanding: we need real-time performance Note: we sparsify vector, not matrix unlike most methods

Most computation in tight inner loop General kernel to focus optimization effort on

Very parallelizable Each pixel/vertex has same operations Vectorization possible (even RGB parallel x3 faster) Not yet exploited (but improved performance with SSE2)

ImplementationImplementation

Quantize matrix elements (6-8 bits) Discard 0 coefficients, gives 3-20% matrix sparsity Note that both matrix, vector now sparse

Memory bandwidth Don’t use bandwidth for lights not in current frame Store matrix coeffs for each light contiguously in memory

ImplementationImplementation

Quantize matrix elements (6-8 bits) Discard 0 coefficients, gives 3-20% matrix sparsity Note that both matrix, vector now sparse

Memory bandwidth Don’t use bandwidth for lights not in current frame Store matrix coeffs for each light contiguously in memory

Cache coherence Segment matrix into blocks 256 pixels Store coefficients as 8 bit value and 8 bit block index

Sparse matrix-vector multiply is 3 – 4 orders of magnitude less work than full multiplication

Total CompressionTotal Compression

11 12 11

21 22 22

31 32 3

1 2

M

M

M

MN N NM

T T TL

T T TL

T T T

LT T T

Compress to 3% – 20% Compress to 0.1% – 1%

ChallengesChallenges

Illumination complexity

Material (BRDF)/view complexity

Transport complexity (shadows)

Putting it together (relight and change view)

Ng, Ramamoorthi and Hanrahan, SIGGRAPH 04

Changing Only The ViewChanging Only The View

Problem CharacterizationProblem Characterization

6D Precomputation Space

Distant Lighting (2D)

View (2D)

Rigid Geometry (2D)

With ~ 100 samples per dimension~ 1012 samples total!! : Intractable computation, rendering

Factorization ApproachFactorization Approach6D Transport

~ 1012 samples

~ 108 samples ~ 108 samples

4D Visibility 4D BRDF

*

=

Double Product Integral RelightingDouble Product Integral Relighting

Triple Product Integral RelightingTriple Product Integral Relighting

Changing Surface Position

Changing Camera Viewpoint

Triple Product Integral RelightingTriple Product Integral Relighting

300,000 vertices

6 x 64 x 64 cubemaps

0.1% - 1% sparsity for visibility, BRDF

3-5 seconds / frame

Relit ImagesRelit Images

Broader Computational RelevanceBroader Computational Relevance

*=

Same machinery applies to basic operation: multiplication• Signal multiplication for audio, image compositing,….• Compressed signals/videos (e.g. wavelets JPEG 2000)

Broadly relevant computational problem in computer graphics,multimedia, numerical analysis with little previous work

ChallengesChallenges

Illumination complexity

Material (BRDF)/view complexity

Transport complexity (shadows)

Putting it together

Other approaches: local coherenceOther approaches: local coherence

Software: Image Relighting Framework [Enrique, Ramamoorthi]http://www.cs.columbia.edu/cg/relighting

Industrial ImpactIndustrial Impact

Environment maps: Lambertian 9 term formula [RH 01] Standard in games, movies

Spherical harmonic lighting [RH 01, SKS 02, 03, …] Full session at SIGGRAPH 03, EGSR 04 Sessions at Game Developer Conference Adopted in Microsoft DirectX 9 Used by Sony, Microsoft, …

Future WorkFuture Work

Exploit parallelism

Efficiency in precomputation

Dynamic scenes

Broader mathematical, computational problems

AcknowledgementsAcknowledgements

Collaborators and students: Ren Ng, Pat Hanrahan, Sameer Agarwal, Henrik Wann Jensen, Sebastian Enrique, Nolan Goodnight, Greg Humphreys

Funding: NSF and Intel

http://www1.cs.columbia.edu/~ravir

http://www1.cs.columbia.edu/~ravir/projects/rendering.html

The EndThe End

Questions?

More demos/videos if time permits

Research ProjectsResearch Projects

Mathematical foundations of appearance

High quality real-time rendering

Volumetric and Multiple Scattering Effects

Acquisition, Rendering with real lighting, materials

Complex lighting, materials in computer vision

Error in Lighting: St Peter’s BasilicaError in Lighting: St Peter’s Basilica

Approximation Terms

Rel

ativ

e L2 E

rror

(%

) Sph. Harmonics

Non-linear Wavelets

Ng, Ramamoorthi, Hanrahan 03

Output Image ComparisonOutput Image ComparisonTop: Linear Spherical Harmonic ApproximationBottom: Non-linear Wavelet Approximation

25 200 2,000 20,000

Matrix Compression RatesMatrix Compression Rates

Teapot (Grace) Plant (St Peter’s)

Light Res.

Fraction non-zero Size

Fraction non-zero Size

6 x 4 x 4 19% 8.5 MB 17% 8.4 MB

6 x 8 x 8 16% 34 MB 15% 31 MB

6 x 16 x 16 14% 113 MB 14% 115 MB

6 x 32 x 32 10% 185 MB 12% 394 MB

6 x 64 x 64 3.1% 314 MB 7.4% 1.0 GB

Matrix Compression RatesMatrix Compression Rates

Teapot (Grace) Plant (St Peter’s)

Light Res.

Fraction non-zero Size

Fraction non-zero Size

6 x 4 x 4 19% 8.5 MB 17% 8.4 MB

6 x 8 x 8 16% 34 MB 15% 31 MB

6 x 16 x 16 14% 113 MB 14% 115 MB

6 x 32 x 32 10% 185 MB 12% 394 MB

6 x 64 x 64 3.1% 314 MB 7.4% 1.0 GB

Double Product Integral RelightingDouble Product Integral Relighting

Changing Surface Position

Changing Camera Viewpoint

Lighting

Transport

Triple Product IntegralsTriple Product Integrals

Tripling Coefficient DensityTripling Coefficient Density

Basis Complexity

Pixels N

Fourier 1/16 (1+3N)2 = O(N2)

Spherical Harmonics9/20 N5/2 + 1/4 N3/2 + 3/10 N1/2 = O(N5/2)

Haar Wavelets 2 – N + 3N log4 N = O(N log N)

General Basis N3

Summary of Wavelet ResultsSummary of Wavelet Results

O(N) for Haar through factorization and dynamic programming exploiting regularity of coefficients

Actually essentially O(n) where n is number of terms used (in a nonlinear approximation)

Wavelets can represent all frequencies unlike pixels or spherical harmonics

Triple products are also efficient

Recent relevant SIGGRAPH papersRecent relevant SIGGRAPH papers1. R. Ng, R. Ramamoorthi and P. Hanrahan, “Triple Product Wavelet Integrals for

All-Frequency Relighting” SIGGRAPH 04 (ACM TOG 23(3))

2. J. Lawrence, R. Ramamoorthi and S. Rusinkiewicz, “Efficient BRDF Importance Sampling using a Factored Representation” SIGGRAPH 04 (ACM TOG 23(3))

3. R. Ng, R. Ramamoorthi and P. Hanrahan, “All-Frequency Shadows Using Non-Linear Wavelet Lighting Approximation” SIGGRAPH 03 (ACM TOG 22(3), pages 376-381).

4. S. Agarwal, R. Ramamoorthi, S. Belongie and H. Jensen, “Structured Importance Sampling of Environment Maps” SIGGRAPH 03 (ACM TOG 22(3), pages 605-612).

5. R. Ramamoorthi and P. Hanrahan, “Frequency Space Environment Map Rendering” SIGGRAPH 02 (ACM TOG 21(3), pages 517-526)

6. R. Ramamoorthi and P. Hanrahan, “A Signal-Processing Framework for Inverse Rendering” SIGGRAPH 01, pages 117-128.

7. R. Ramamoorthi and P. Hanrahan, “An Efficient Representation for Irradiance Environment Maps” SIGGRAPH 01, pages 497-500.

8. M. Agrawala, R. Ramamoorthi, A. Heirich and L. Moll, “Efficient Image-Based Methods for Rendering Soft Shadows” SIGGRAPH 00, pages 375-384.