Upload
others
View
3
Download
0
Embed Size (px)
Citation preview
DENOISING OF VOLUMETRIC DEPTH CONFIDENCE FOR VIEW RENDERING
Srinivas Parthasarathy, Akul Chopra, Emilie Baudin, Pravin Kumar Rana, and Markus Flierl
School of Electrical Engineering, KTH Royal Institute of Technology, Stockholm, Sweden
1 MOTIVATION
Free-Viewpoint Television
Digital Camera
Virtual Camera
Depth Map Estimation
Stereo
Matching
Stereo
Matching
view ♯1 view ♯2 view ♯3 view ♯4 view ♯5
depth map ♯2 depth map ♯4
Inconsistency of Multiview Depth Imageryview ♯1 view ♯2 view ♯3
(a) Dancer.
(b) Newspaper.
2 APPROACH
Estimated Depth Maps
depth map ♯1 depth map ♯2 depth map ♯n
Volumetric Depth Confidence
PerspectiveProjection
b
b
b
b
b
Camera 1 Camera 2 Camera 3
A(+1)
B(+2)
C(+3)
D(+2)
E(+1)
(0) Occluded Area
-2-2
-2
-2
-2
-2
-2
-2
-2
TransparentMedium
Example: Confidence Assignment to Voxels
•Each mapped depth pixel from a viewpoint con-tributes +1 as confidence for a voxel
•Each occluded pixel from a viewpoint does notchange the confidence for a voxel
•Each ray traversing in a transparent voxel con-tributes -1 as confidence for a voxel
3 3D WAVELET DENOISING
c© Disney
Why wavelet denoising?
• It can create sparse coefficients and spreads outi.i.d. noise equally among all the coefficients
• It can distinguish between signal and noise effi-ciently
Wavelet Denoising Process
VolumeDecomposition
Thresholding
VolumeReconstruction
Wavelet Coefficients
Modified Coefficients
Noisy Volume
DiscreteWaveletTransform(DWT)
DenoisingMethods
InverseDWT
Denoised Volume
Adaptive Thresholding Using SURE Shrink
Goal: Determine thresholds that remove noise efficiently
•SURE (Stein’s Unbiased Risk Estimator) is usedto estimate sub–band adaptive thresholds
•For multivariate normal observations, SURE offersan unbiased estimate of the expected squared er-ror of the mean
•Let {ci : i = 1, . . . , l} be the noisy wavelet coeffi-cients in the j-th sub-band.
SURE(θ; c) = l − 2|{i : |ci| < θ}| +l∑
i
min(|ci|, θ)2,
where θ is a given threshold and | · | denotes thecardinality of a set
•Set the SURE threshold θS by
θS = argminθ
SURE(θ; c)
4 VIEW RENDERING
Virtual
View
Hole
Filling
DenoisedVolume
Perspective Projection
center depth map
virtual center view
left
depth
map
right
depth
map
left view right view
3DWarping
-by checking depth consistency
-by using 3× 3 median filter
5 EXPERIMENTAL RESULTS
Objective Results
Rendered Views with Confidence Range [-3,3]
Noisy
Volume
db1
Th.=0.01
db2
Th.=1.40
db3
Th.=1.20
db3
Adaptive
db4
Th.=1.00
38.87 38.7 38.71 38.72 38.87 38.64
31.8431.33 31.42 31.43
32.08
31.38
PSNR
[dB]
Dancer
Newspaper
Effect of Confidence Range on NewspaperRendered Views
MPEG
Depth Maps
Noisy Volume db3 Adaptive
31.63
31.84
32.0832.04
32.33 32.32
PSNR
[dB]
[−3,+3]
[−7,+7]
Subjective Results
Rendered views of Newspaper with ConfidenceRange [-7,+7]
(a) Original view. (b) With MPEG depth maps.
(c) With noisy depth volume. (d) With denoised volume.
c© Disney
... making important notes!
•Adaptive thresholding in the 3D wavelet domainimproves the quality compared to rendering withMPEG depth maps and noisy volumetric data,respectively
•For synthetic scenes, the improvement is insignif-icant due to consistent description of the geom-etry
•Volumetric depth confidence improves renderingresults even without wavelet denoising when pro-jecting highest confidence voxels into the cameraplane
6 CONCLUSIONS
•Define volumetric depth confidence to handle in-consistent depth estimates
•Use superposition principle to incorporate confi-dence information from multiple camera views
•Denoise volumetric depth confidence by usingadaptive 3D wavelet thresholding
• Improve the visual quality of rendered views byusing the denoised volumetric depth confidence