Upload
fraunhofer-igd-competence-center-interactive-engineering-technologies
View
1.321
Download
1
Embed Size (px)
DESCRIPTION
C. Schwartz, R. Klein3D COFORM Star Day – VAST 2011, October 20 - Prato, Italy
Citation preview
Institute of Computer Science II
Computer Graphics
Realistic Object Appearance using Bidirectional Texture Functions Christopher Schwartz, Reinhard Klein
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 1 20/10/2011
INTRODUCTION
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 2 20/10/2011
Object Appearance
• Common presentation: Geometry (+ Texture)
• … but is this sufficient?
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 3 20/10/2011
Geometry only With texture Correct appearance
Importance of Surface Appearance
• Same shape, different materials
• Different „look-and-feel“ • Important hints about the object • Misleading vs. better understanding
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 4 20/10/2011
Object Appearance
• Impression of reflection of incident light
• Influenced by features on different scales
• Macroscopic
• Mesoscopic
• Microscopic
• Viewpoint and Illumination dependent
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 5 20/10/2011
Form of Representation
Macroscopic scale
• 3D shape
• Explicit representation (e.g. polygon mesh)
Mesoscopic scale
• Individually resolved by human perception
• Statistical representation not accurate
• Explicit representation too costly
Microscopic scale
• Alignment of microscopic structures
• Statistical representation (e.g. BRDF)
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 6 20/10/2011
Bidirectional Reflectance Distribution Function
• Opaque, uniform Materials • No texture!
• Ratio of incident irradiance to outgoing radiance • Defined over local
hemisphere
• Depends on • Solid angle of light ωi
• Solid angle of view ωo
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 7 20/10/2011
Bidirectional Reflectance Distribution Function
• Example BRDF • sampled at discrete angles
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 8 20/10/2011
ωi
ωo
BRDF Tabulated Example: BRDF on Sphere Discrete sample positions on Hemisphere
Specular reflection
Model-driven vs. Data-driven
• Matusik et al. 2003 [1]: Measured BRDF ground-truth data
• 100 real world materials
• 1° resolution for view- and light directions
• > 1,000,000 samples
• Ngan et al. 2005 [2]: Experimental analysis of BRDF models
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 9 20/10/2011
Model-driven vs. Data-driven
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 10 20/10/2011
Red phenolic
Measured distribution from [1]
Fitted model (Cook-Torrance) from [2]
Model-driven vs. Data-driven
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 11 20/10/2011
Red phenolic
Measured distribution from [1]
Fitted model (Cook-Torrance) from [2]
Data-driven Reflectance
• Example Surface with Meso structure
• Results are „ABRDFs“ (apparent BRDFs [3])
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 12 20/10/2011
ωi
ωo
ωi
ωo
Specular reflection
Retro-reflection
Hard to fit with analytical model
influence from neighborhood
Model-driven Reflectance
• Loss of Mesoscale depth impression…
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 13 20/10/2011
Fitted analytical SVBRDF Photograph McAllister 2002 [10]
Data-driven Reflectance
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 14 20/10/2011
Texture Mesoscale approximated by Bump-mapping
Data-Driven (BTF)
Images taken from Müller et al. 2005 [7]
Form of Representation
Macroscopic scale
• 3D shape
• Explicit representation (e.g. polygon mesh)
Mesoscopic scale
• Individually resolved by human perception
• Statistical representation not accurate
• Explicit representation too costly
Microscopic scale
• Alignment of microscopic structures
• Statistical representation (e.g. BRDF)
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 15 20/10/2011
Form of Representation
Macroscopic scale
• 3D shape
• Explicit representation (e.g. polygon mesh)
Mesoscopic scale
• Individually resolved by human perception
• Statistical representation not accurate
• Explicit representation too costly
Microscopic scale
• Alignment of microscopic structures
• Statistical representation (e.g. BRDF)
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 16 20/10/2011
Data-driven: Image based
Form of Representation
Macroscopic scale
• 3D shape
• Explicit representation (e.g. polygon mesh)
Mesoscopic scale
• Individually resolved by human perception
• Statistical representation not accurate
• Explicit representation too costly
Microscopic scale
• Alignment of microscopic structures
• Statistical representation (e.g. BRDF)
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 17 20/10/2011
Bidirectional Texture Function
ACQUISITION
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 18 20/10/2011
First use of BTF: CUReT Database
• 1996 – 1999 by Dana et al. [4] • 61 materials
• 205 different view- and light directions
• 24-bit RGB images
Manual placement of camera
BTF only partially measured
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 19 20/10/2011
University of Bonn BTF Database
• 2001 – 2003 Sarlette et al. [5] • 6561 view and light directions
• 36-bit RGB images
• Fully automated
• 12 hrs per acquisition
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 20 20/10/2011
University of Bonn Multiview Dome
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 21 20/10/2011
• 2004 – now by Sarlette et al.
• 22,801 view and light directions
• HDR images
• Fully automated
• No moving parts
• 2hrs per acquisition
First Capture of Complete Objects with BTF
• Furukawa et al. 2002 [6] • Laser scanned geometry
• Separate BTF capture
Very sparse sampling
Alignment errors
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 22 20/10/2011
Photograph Rendering
First Capture of Complete Objects with BTF
• Furukawa et al. 2002 [6] • Laser scanned geometry
• Separate BTF capture
Very sparse sampling
Alignment errors
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 23 20/10/2011
Photograph Rendering
Integrated Acquisition of Objects with BTF
• Müller et al. 2005 [7] • Use University of Bonn Dome
• Dense sampling (22,801), 2hrs/acquisition
• Measurements integrated in one setup • No registration necessary
• Geometry via Shape-from-Silhouette
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 24 20/10/2011
Integrated Acquisition of Objects with BTF
• Müller et al. 2005 Drawbacks:
No radiometric calibration
• Misleading colors
• Only LDR
Shape-from-Silhouette
• Not automatable
• Coarse geometry – Misleading Shape
– Blur due to misalignment
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 25 20/10/2011
Example from Havemann et al. 2008 [12]
Integrated HQ Acquisition
• Holroyd et al. 2010 [17] • Integrated setup
• Geometry with Structured Light
• 42 view and light directions
• 5hrs per acquisition
Sparse sampling Model-driven
• Fit Cook-torrance BRDFs
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 26 20/10/2011
Photograph Rendering
Integrated HQ Acquisition
• Holroyd et al. 2010 [17] • Integrated setup
• Geometry with Structured Light
• 42 view and light directions
• 5hrs per acquisition
Sparse sampling Model-driven
• Fit Cook-torrance BRDFs
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 27 20/10/2011
Photograph Rendering
Integrated HQ Acquisition with BTF
• Schwartz et al. 2011 [8] • Use University of Bonn Dome
• Extended with projectors for Structured Light
• integrated measurement
• Rapid: 3.7 hrs per acquisition
• Proper calibration and HDR
• Geometry: Weinmann et al. 2011 [11]
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 28 20/10/2011
Visual Hull [7], [12] Laser Scan Proposed Method [11]
Quality
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 29 20/10/2011
Faithfulness
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 30 20/10/2011
Photographic picture
(tonemapped HDR)
BTF + Geometry Schwartz et al. 2011 [8]
(tonemapped HDR)
Polynomial Texture Maps
• Malzbender et al. 2001 [9]: PTM
• Image-based, Sampling of Light (ωi)
Incomplete appearance information
• View-dependent part of reflectance missing
For 3D Objects: only one fixed viewpoint
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 31 20/10/2011
PTM
Texture
Faithfulness
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 32 20/10/2011
Photographic picture
(tonemapped HDR)
BTF + Geometry Schwartz et al. 2011 [8]
(tonemapped HDR)
Polynomial Texture Map Malzbender et al. 2001 [9]
(Single view and LDR!)
Multiview PTMs
• Gunawardane et al. 2009 [18]: • PTMs from multiple viewpoints
No macro scale
interpolation of views via optical flow
Limited amount of views
• Combining multiple objects is hard • incorrect silhouttes, occlusion,
shadows, etc.
• Full light transport (i.e. path-tracing) not possible
33 20/10/2011 3D COFORM STAR – VAST 2011 Prato, Italy – Christopher
Schwartz
Full Light Transport with BTFs
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 34 20/10/2011
Full Light Transport with BTFs
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 35 20/10/2011
COMPRESSION, RENDERING AND TRANSMISSION
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 36 20/10/2011
Datasizes
• Schwartz et al. 2011 [8]:
• Uncompressed BTF: ≈ 500 GB per object
• Not feasible even for offline rendering…
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 37 20/10/2011
BTF Compression
• Fitting analytical models
• Wu et al. 2011 [15]: SPMM
• Model-driven…
• lost meso-structure
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 38 20/10/2011
BTF
SPMM
BTF Compression
• High degree of redundancy in the BTF
• Perform statistical data analysis
• Find low dimensional basis
• learn how to best describe the data
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 39 20/10/2011
Compression using Statistical Analysis
• Organization of the discrete BTF
• As matrix
• As Tensor
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 40 20/10/2011
Angles ωi, ωo
Pix
els
(x,y
,co
lor)
Pixels (x,y)
ωo
Note: Color (or wavelength) can be additional dimension
Full Matrix Factorization
• E.g. Liu et al. 2004 [14]: FMF
• Representation is compact and realtime renderable [5]
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 41 20/10/2011
…
...
M
… V
Angular components
“Eigen-ABRDFs“
U …
Spatial components
“Eigen-Textures“ SVD
TM USV
Importance
Decorrelated Full Matrix Factorization
• Gero Müller 2009 [13]: DFMF
• Insight from image compression (e.g. JPEG)
• Human perception is more sensitive to variation in intensity than color
• Also: chromacity in images exhibits less variation
• Decorrelate the BTF data into luminance and chrominance – BTFRGB BTFY BTFU BTFV
• Use fewer components for chrominance channels U, V
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 42 20/10/2011
FMF Rendering
• Uncompressed: ≈ 500GB
• Compressed: ≈ 640MB
• 32 components
• Even fits on the GPU
• Random access to BTF
• Angular combination a = (ωi, ωo)
• Pixel p = (x,y)
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 43 20/10/2011
com
po
nen
ts
Angular component #1
Spatial component #1
Angular component #2
Spatial component #2
…
…
pixel angles
BTF(a,p) = < , >
Interactive Inspection via GPU Rendering
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 44 20/10/2011
Streaming of BTF over the Internet
• Schwartz et al. 2011 [16]: • Spatial components
≈ natural images
• Angular components ≈ low frequency
• Apply additional wavelet compression
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 45 20/10/2011
0.4bpp Wavelet
16bpp Reference
Streaming of BTF over the Internet
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 46 20/10/2011
0.87 MB First renderable
version
1 MB 7 MB 46.4 MB Fully
transmitted
534 GB Reference
Questions?
Learn more about BTFs for Cultural Heritag on Wednesday [8] and Thursday [16]
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 47 20/10/2011
References [1] “A Data-Driven Reflectance Model”, Matusik W., Pfister H., Brand M. and McMillan L., ACM TOG 22, 3(2003), 759-769. [2] “Experimental Analysis of BRDF models”, Ngan A., Durand F. and Matusik W., Proceedings of EGSR, 2005, 117-226. [3] „Image-based Rendering with Controllable Illumination“, Wong T., Heng P., Or S., and Ng W., Proceedings of EGWR, 1997, 13-22 [4] „Reflectance and Texture of Real-World Surfaces “, Dana K.J., van Ginneken B., Nayar S.K. and Koenderink J.J., Proceedings of CVPR, 1997, 151-157 [5] „Efficient and Realistic Visualization of Cloth“, Sattler M., Sarlette R. and Klein R., Proceedings of EGSR, 2003 [6]„Appearance based object modeling using texture database: acquisition, compression and rendering“, Furukawa R., Kawasaki H. Ikeuchi K. and Sakauchi M., Proceedings of EGRW 2002, 257-266 [7] „Rapid Synchronous Acquisition of Geometry and BTF for Cultural Heritage Artefacts“, Müller G., Bendels G.H. and Klein R., Proceedings of VAST, 2005 [8] „Integrated High-Quality Acquisition of Geometry and Appearance for Cultural Heritage”, Schwartz C., Weinmann W., Roland R. and Klein R., Proceedings of VAST, 2011 [9] „Polynomial Texture Maps“, Malzbender T., Gelb D. and Wolters H., Proceedings of SIGGRAPH, 2001 [10] „A Generalized Surface Appearance Representation For Computer Graphics“, McAllister D.K., PhD. Thesis, University of North Carolina at Chapel Hill, 2002 [11] „A Multi-Camera, Multi-Projector Super-Resolution Framework for Structured Light“, Weinmann M., Schwartz C., Ruiters R. and Klein R., Proceedings of 3DIMPVT, 2011, 397-404 [12] „The Presentation of Cultural Heritage Models in Epoch“, Havemann S., Settgast V., Fellner D., Willems G., Van Gool, L., Müller G., Schneider M. and Klein R., EPOCH Conference on Open Digital CH Systems, 2008 [13] „Data-Driven Methods for Compression and Editing of Spatially Varying Appearance“, Müller G., PhD. Thesis, University of Bonn, 2009 [14] „Synthesis and Rendering of Bidirectional Texture Functions on Arbitrary Surfaces“, Liu X., Hu Y., Zhang J. Tong X., Guo B. and Shum H.-Y., IEEE Transactions on Visualization and Computer Graphics 10 (3), 2004, 278-289 [15] „A Sparse Parametric Mixture Model for BTF Compression, Editing and Rendering“, Wu H., Dorsey J. and Rushmeier H., Computer Graphics Forum 30 (2), 2011, 465-473 [16] „WebGL-based Streaming and Presentation Framework for Bidirectional Texture Function“, Schwartz C., Ruiters R., Weinmann M. and Klein R., Proceedings of VAST, 2011 [17] „A coaxial optica scanner for synchronous acquisition of 3D geometry and surface reflectance“, Holroyd M., Lawrence J. and Zickler T., ACM Trans. Graph. 29 (4), 2010 [18] „Optimized Image Sampling for View and Light Interpolation“, Gunawardane P., Wang O., Scher S., Rickards I., Davis J. And Malzbender T., Proceedings of VAST, 2009 [19] „Principles and practices of robust, photography-based digital imaging techniques for museums.“, Mudge M., Schroer C., Ear G., Martinez K., Pagi H., Toler-Franklin C., Rusinkiewicz S., Palma G., Wochowiak M., Ashley M., Matthews N., Noble T. and Dellepiane M., Proceedings of VAST, 2010
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 48 20/10/2011