View
218
Download
0
Tags:
Embed Size (px)
Citation preview
Real-Time Rendering and Interaction Real-Time Rendering and Interaction with Complex Lighting and Materialswith Complex Lighting and Materials
Ravi Ramamoorthi
Rendering LaboratoryColumbia University
Intel: August 13, 2004
High quality real-time renderingHigh quality real-time rendering
Photorealism, not just more polygons
Natural lighting, materials, shadows
Interiors by architect Frank Gehry. Note rich lighting, ranging from localized sources to reflections off vast sheets of glass.
High quality real-time renderingHigh quality real-time rendering
Photorealism, not just more polygons
Natural lighting, materials, shadows
Real materials diverse and not easy to represent by simple parameteric models. Want to support measured reflectance.
Glass VaseGlass Star (courtesy Intel) Peacock feather
High quality real-time renderingHigh quality real-time rendering
Photorealism, not just more polygons
Natural lighting, materials, shadows
Natural lighting creates a mix of soft diffuse and hard shadows.
Agrawala et al. 00small area light, sharp shadows soft and hard shadows
Ng et al. 03
DemoDemo
Industrial RelevanceIndustrial Relevance
Very difficult computational problem Both preprocessing (hours to days) and real-time rendering Related to computational problems in other areas
Driving application Interactive 3D graphics major use of PCs Realism increases use, applications
Leverages high-performance processors Exploit parallelism (at pixel level), modern feature SIMD instructions, MMX and SSE instructions
ApplicationsApplications
Entertainment: Lighting design
Architectural visualization
Material design: Automobile industry
Realistic Video games
Electronic commerce
ChallengesChallenges
Illumination complexity
Material / view complexity
Transport complexity (shadows)
Putting it together
Ramamoorthi and Hanrahan, SIGGRAPH 01, 02
Environment MapsEnvironment Maps
Miller and Hoffman, 1984
Previous work: Blinn 76, Greene 86, Cabral 87, 99
Irradiance Environment MapsIrradiance Environment Maps
Incident Radiance(Illumination Environment Map)
Irradiance Environment Map
R N
AssumptionsAssumptions
Diffuse surfaces (relaxed later)
Distant illumination
No shadowing (relaxed later)
Hence, Irradiance is a function of surface normal
Computing IrradianceComputing Irradiance
Classically, hemispherical integral for each pixel
Lambertian surface is like low pass filter
Frequency-space analysis
IncidentRadiance
Irradiance
Analytic Irradiance FormulaAnalytic Irradiance Formula
Lambertian surface acts like low-pass filter
lm l lmE A LlA
2 / 3
/ 4
0
2 1
2
2
( 1) !2
( 2)( 1) 2 !
l
l l l
lA l even
l l
l0 1 2
Basri & Jacobs 01Ramamoorthi & Hanrahan 01
9 Parameter Approximation9 Parameter Approximation
-1-2 0 1 2
0
1
2
( , )lmY
xy z
xy yz 23 1z zx 2 2x y
l
m
Exact imageOrder 29 terms
RMS Error = 1%
For any illumination, average error < 3% [Basri Jacobs 01]
Real-Time RenderingReal-Time Rendering
Simple procedural rendering method (no textures) Requires only matrix-vector multiply and dot-product In software or NVIDIA vertex programming hardware
Widely used in Games (AMPED for Microsoft Xbox), Movies (Pixar, Framestore CFC, …)
( ) tE n n Mn
surface float1 irradmat (matrix4 M, float3 v) {
float4 n = {v , 1} ;
return dot(n , M*n) ;
}
Convolution: General MaterialsConvolution: General Materials
Ramamoorthi and Hanrahan 01
( , ) ( ) ,B N V L R N l l V dl
B L
Frequency: product
Spatial: integral
ij i ijB L
Spherical harmonic analysis
Natural Lighting, Realistic Materials Natural Lighting, Realistic Materials
Ramamoorthi and Hanrahan 02
ChallengesChallenges
Illumination complexity
Material (BRDF)/view complexity
Transport complexity (shadows): Relighting
Putting it together
Ng, Ramamoorthi and Hanrahan SIGGRAPH 03
Existing Fast Shadow TechniquesExisting Fast Shadow TechniquesWe know how to render very hard, very soft shadows
Sloan, Kautz, Snyder 2002
Shadows from smooth lighting (precomputed radiance transfer)
Sen, Cammarano, Hanrahan, 2003
Shadows from point-lights(shadow maps, volumes)
AssumptionsAssumptions
Static geometry
Precomputation
Real-Time Rendering (relight all-frequency effects)
1
2
3
N
P
P
P
P
11 12 11
21 22 22
31 32 3
1 2
M
M
M
MN N NM
T T TL
T T TL
T T T
LT T T
Relighting as a Matrix-Vector MultiplyRelighting as a Matrix-Vector Multiply
1
2
3
N
P
P
P
P
11 12 11
21 22 22
31 32 3
1 2
M
M
M
MN N NM
T T TL
T T TL
T T T
LT T T
Input Lighting (Cubemap Vector)
Output Image(Pixel Vector)
TransportMatrix
Relighting as a Matrix-Vector MultiplyRelighting as a Matrix-Vector Multiply
Problem DefinitionProblem Definition
Matrix is Enormous 512 x 512 pixel images 6 x 64 x 64 cubemap environments
Full matrix-vector multiplication is intractable On the order of 1010 operations per frame
How to relight quickly?
Sparse Matrix-Vector MultiplicationSparse Matrix-Vector Multiplication
Choose data representations with mostly zeroes
Vector: Use non-linear wavelet approximation on lighting
Matrix: Wavelet-encode transport rows
11 12 11
21 22 22
31 32 3
1 2
M
M
M
MN N NM
T T TL
T T TL
T T T
LT T T
Non-linear Wavelet ApproximationNon-linear Wavelet Approximation
Wavelets provide dual space / frequency locality Large wavelets capture low frequency area lighting Small wavelets capture high frequency compact
features
Non-linear Approximation Use a dynamic set of approximating functions (depends
on each frame’s lighting) By contrast, linear approx. uses fixed set of basis
functions (like 25 lowest frequency) We choose 10’s - 100’s from a basis of 24,576 wavelets
Non-linear Wavelet Light ApproximationNon-linear Wavelet Light Approximation
Wavelet Transform
1
2
3
4
5
6
N
L
L
L
L
L
L
L
2
6
0
0
0
0
0
L
L
Non-linearApproximation
Retain 0.1% – 1% terms
Non-linear Wavelet Light ApproximationNon-linear Wavelet Light Approximation
Performance: TimingPerformance: Timing
Precomputation of Transport Matrix Ray tracing: several days (depends on ray tracer) Graphics hardware: 80,000 vertex buddha = 3 hours
Rendering (2.8 GHz Pentium IV) For 100 terms, 4-6 fps RGB, 9-15 fps monochrome Relighting alone (core matrix-vector) somewhat faster Use almost full CPU perf. (implementation details later)
VideoVideo
Broader Computational RelevanceBroader Computational Relevance
Sparse matrix-vector multiplication standard numerical problem Challenging problem (giant matrices, vectors) Even more demanding: we need real-time performance Note: we sparsify vector, not matrix unlike most methods
Most computation in tight inner loop General kernel to focus optimization effort on
Broader Computational RelevanceBroader Computational Relevance
Sparse matrix-vector multiplication standard numerical problem Challenging problem (giant matrices, vectors) Even more demanding: we need real-time performance Note: we sparsify vector, not matrix unlike most methods
Most computation in tight inner loop General kernel to focus optimization effort on
Very parallelizable Each pixel/vertex has same operations Vectorization possible (even RGB parallel x3 faster) Not yet exploited (but improved performance with SSE2)
ImplementationImplementation
Quantize matrix elements (6-8 bits) Discard 0 coefficients, gives 3-20% matrix sparsity Note that both matrix, vector now sparse
Memory bandwidth Don’t use bandwidth for lights not in current frame Store matrix coeffs for each light contiguously in memory
ImplementationImplementation
Quantize matrix elements (6-8 bits) Discard 0 coefficients, gives 3-20% matrix sparsity Note that both matrix, vector now sparse
Memory bandwidth Don’t use bandwidth for lights not in current frame Store matrix coeffs for each light contiguously in memory
Cache coherence Segment matrix into blocks 256 pixels Store coefficients as 8 bit value and 8 bit block index
Sparse matrix-vector multiply is 3 – 4 orders of magnitude less work than full multiplication
Total CompressionTotal Compression
11 12 11
21 22 22
31 32 3
1 2
M
M
M
MN N NM
T T TL
T T TL
T T T
LT T T
Compress to 3% – 20% Compress to 0.1% – 1%
ChallengesChallenges
Illumination complexity
Material (BRDF)/view complexity
Transport complexity (shadows)
Putting it together (relight and change view)
Ng, Ramamoorthi and Hanrahan, SIGGRAPH 04
Changing Only The ViewChanging Only The View
Problem CharacterizationProblem Characterization
6D Precomputation Space
Distant Lighting (2D)
View (2D)
Rigid Geometry (2D)
With ~ 100 samples per dimension~ 1012 samples total!! : Intractable computation, rendering
Factorization ApproachFactorization Approach6D Transport
~ 1012 samples
~ 108 samples ~ 108 samples
4D Visibility 4D BRDF
*
=
Double Product Integral RelightingDouble Product Integral Relighting
Triple Product Integral RelightingTriple Product Integral Relighting
Changing Surface Position
Changing Camera Viewpoint
Triple Product Integral RelightingTriple Product Integral Relighting
300,000 vertices
6 x 64 x 64 cubemaps
0.1% - 1% sparsity for visibility, BRDF
3-5 seconds / frame
Relit ImagesRelit Images
Broader Computational RelevanceBroader Computational Relevance
*=
Same machinery applies to basic operation: multiplication• Signal multiplication for audio, image compositing,….• Compressed signals/videos (e.g. wavelets JPEG 2000)
Broadly relevant computational problem in computer graphics,multimedia, numerical analysis with little previous work
ChallengesChallenges
Illumination complexity
Material (BRDF)/view complexity
Transport complexity (shadows)
Putting it together
Other approaches: local coherenceOther approaches: local coherence
Software: Image Relighting Framework [Enrique, Ramamoorthi]http://www.cs.columbia.edu/cg/relighting
Industrial ImpactIndustrial Impact
Environment maps: Lambertian 9 term formula [RH 01] Standard in games, movies
Spherical harmonic lighting [RH 01, SKS 02, 03, …] Full session at SIGGRAPH 03, EGSR 04 Sessions at Game Developer Conference Adopted in Microsoft DirectX 9 Used by Sony, Microsoft, …
Future WorkFuture Work
Exploit parallelism
Efficiency in precomputation
Dynamic scenes
Broader mathematical, computational problems
AcknowledgementsAcknowledgements
Collaborators and students: Ren Ng, Pat Hanrahan, Sameer Agarwal, Henrik Wann Jensen, Sebastian Enrique, Nolan Goodnight, Greg Humphreys
Funding: NSF and Intel
http://www1.cs.columbia.edu/~ravir
http://www1.cs.columbia.edu/~ravir/projects/rendering.html
The EndThe End
Questions?
More demos/videos if time permits
Research ProjectsResearch Projects
Mathematical foundations of appearance
High quality real-time rendering
Volumetric and Multiple Scattering Effects
Acquisition, Rendering with real lighting, materials
Complex lighting, materials in computer vision
Error in Lighting: St Peter’s BasilicaError in Lighting: St Peter’s Basilica
Approximation Terms
Rel
ativ
e L2 E
rror
(%
) Sph. Harmonics
Non-linear Wavelets
Ng, Ramamoorthi, Hanrahan 03
Output Image ComparisonOutput Image ComparisonTop: Linear Spherical Harmonic ApproximationBottom: Non-linear Wavelet Approximation
25 200 2,000 20,000
Matrix Compression RatesMatrix Compression Rates
Teapot (Grace) Plant (St Peter’s)
Light Res.
Fraction non-zero Size
Fraction non-zero Size
6 x 4 x 4 19% 8.5 MB 17% 8.4 MB
6 x 8 x 8 16% 34 MB 15% 31 MB
6 x 16 x 16 14% 113 MB 14% 115 MB
6 x 32 x 32 10% 185 MB 12% 394 MB
6 x 64 x 64 3.1% 314 MB 7.4% 1.0 GB
Matrix Compression RatesMatrix Compression Rates
Teapot (Grace) Plant (St Peter’s)
Light Res.
Fraction non-zero Size
Fraction non-zero Size
6 x 4 x 4 19% 8.5 MB 17% 8.4 MB
6 x 8 x 8 16% 34 MB 15% 31 MB
6 x 16 x 16 14% 113 MB 14% 115 MB
6 x 32 x 32 10% 185 MB 12% 394 MB
6 x 64 x 64 3.1% 314 MB 7.4% 1.0 GB
Double Product Integral RelightingDouble Product Integral Relighting
Changing Surface Position
Changing Camera Viewpoint
Lighting
Transport
Triple Product IntegralsTriple Product Integrals
Tripling Coefficient DensityTripling Coefficient Density
Basis Complexity
Pixels N
Fourier 1/16 (1+3N)2 = O(N2)
Spherical Harmonics9/20 N5/2 + 1/4 N3/2 + 3/10 N1/2 = O(N5/2)
Haar Wavelets 2 – N + 3N log4 N = O(N log N)
General Basis N3
Summary of Wavelet ResultsSummary of Wavelet Results
O(N) for Haar through factorization and dynamic programming exploiting regularity of coefficients
Actually essentially O(n) where n is number of terms used (in a nonlinear approximation)
Wavelets can represent all frequencies unlike pixels or spherical harmonics
Triple products are also efficient
Recent relevant SIGGRAPH papersRecent relevant SIGGRAPH papers1. R. Ng, R. Ramamoorthi and P. Hanrahan, “Triple Product Wavelet Integrals for
All-Frequency Relighting” SIGGRAPH 04 (ACM TOG 23(3))
2. J. Lawrence, R. Ramamoorthi and S. Rusinkiewicz, “Efficient BRDF Importance Sampling using a Factored Representation” SIGGRAPH 04 (ACM TOG 23(3))
3. R. Ng, R. Ramamoorthi and P. Hanrahan, “All-Frequency Shadows Using Non-Linear Wavelet Lighting Approximation” SIGGRAPH 03 (ACM TOG 22(3), pages 376-381).
4. S. Agarwal, R. Ramamoorthi, S. Belongie and H. Jensen, “Structured Importance Sampling of Environment Maps” SIGGRAPH 03 (ACM TOG 22(3), pages 605-612).
5. R. Ramamoorthi and P. Hanrahan, “Frequency Space Environment Map Rendering” SIGGRAPH 02 (ACM TOG 21(3), pages 517-526)
6. R. Ramamoorthi and P. Hanrahan, “A Signal-Processing Framework for Inverse Rendering” SIGGRAPH 01, pages 117-128.
7. R. Ramamoorthi and P. Hanrahan, “An Efficient Representation for Irradiance Environment Maps” SIGGRAPH 01, pages 497-500.
8. M. Agrawala, R. Ramamoorthi, A. Heirich and L. Moll, “Efficient Image-Based Methods for Rendering Soft Shadows” SIGGRAPH 00, pages 375-384.