Interactive BRDF Estimation for Mixed-Reality Applications

Martin Knecht, Georg Tanzmeister, Christoph Traxler, Michael Wimmer

Institute of Computer Graphics and Algorithms

Vienna University of Technology

Motivation

Goal of our mixed reality frameworkLight interaction between real and virtual objects

Materials of real objects must be known

Martin Knecht 2

Problem Statement

Material estimation should not need any preprocessing

Use Kinect sensor and fish-eye lense camera for data acquisition

Should run at interactive framerates

Use GPU wherever possible

Should estimate Phong parameters Used in mixed reality framework

3Martin Knecht

Similar to pipeline of Zheng et al. 2009

Estimation Pipeline

4Martin Knecht

Input Data for Estimation

5Martin Knecht

Highlight Removal

6Martin Knecht

Highlight Removal

7Martin Knecht

Diffuse Reflectance Estimation

8Martin Knecht

Diffuse Reflectance Estimation

9Martin Knecht

Inverse shading:

Clustering

10Martin Knecht

Clustering 1/5

Assumption: similar color same material

Same material same specular parameters

Clustering executed on the diffuse estimation

Novel hybrid CPU/GPU K-Means1) Initialize cluster centers

2) Assign pixel to nearest cluster center

3) Calculate new cluster centers

4) Repeat steps 2 & 3

11Martin Knecht

Clustering 2/5

1) Initialize cluster centersRandom cluster centers

Exploit temporal coherenceReuse of cluster centers of previous frame

12Martin Knecht

Clustering 3/5

2) Assign pixel to nearest cluster center

13Martin Knecht

Cluster ShaderRGBC1

RGBC2

...

...

Cluster 1 Cluster 2 Cluster 6 Bitmask 1 Bitmask 2...

Clustering 4/5

3) Calculate new cluster centers

1x1 Mipmap is the average over all pixel

New cluster center:

14Martin Knecht

1x1 RGBD

1x1 Bitmask

TRGBD

T*

Clustering 5/5

4) Repeat steps 2 & 3

Repeat until no pixel changes cluster Standard stopping criteria too conservative

Max. 20 iterations

Check variance change of distances

15Martin Knecht

Clustering 5/5

16Martin Knecht

Specular Reflectance Estimation

17Martin Knecht

Specular Reflectance Estimation

Done on a per cluster basis same material

CPU based nonlinear function solver

Variables: Specular parameter

Light positions

Evaluation of objective function done on GPUSimilar mipmap method used as for clustering

18Martin Knecht

Specular Reflectance Estimation

19Martin Knecht

Results - Estimation

20Martin Knecht

Diffuse component

Phong shaded image

Specular component+

Results – Timings

BRDF estimation runs at ~2.8 fps

Two tasks with major impact K-Means clustering

Specular estimation

Martin Knecht 21

Normal estimation 0,57 msHighlight removal 0,94 msDiffuse estimation 0,23 msK-Means 39,08 msSpecular estimation 315,76 ms

< 0.5 %

~ 11 %

~ 88,5 %

Differential Instant Radiosity

22Martin Knecht

Results – Mixed Reality Integration

23Martin Knecht

Limitations

Kinect sensor does not work everywhere

Bright objects are discarded from estimation

Shadows are not considered

No estimation of optimal amount of clusters

No integration of data over time

Simplifications lower quality of estimation

24Martin Knecht

Conclusion & Future work

BRDF estimation without any preprocessing

Hybrid CPU/GPU K-Means implementation

Runs at interactive framerates

Future workImprove speed specular estimation

Improve quality BRDF estimation

Exploit temporal coherence more often

25Martin Knecht

Thank you for your attention!

Supported by grand from the FFG-Austrian Research Promotion Agency under the Program “FIT-IT Visual

Computing” Project Nr.: 820916

26