Upload
phoebe-palmer
View
221
Download
0
Tags:
Embed Size (px)
Citation preview
Depth Matters: Influence of Depth Depth Matters: Influence of Depth Cues onCues on
Visual SaliencyVisual Saliency
Congyan Lang , Tam V. Nguyen, Harish Katti , Karthik Yadati ,Mohan Kankanhalli , and Shuicheng Yan
Todays’ Presenter : Daniel Segal
Computer Vision – ECCV 201212th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part II
Congyan Lang Tam V. Nguye Harish Katti
AuthorsAuthors
Problem motivation
Challenges
Related work
Proposed solution
Pros & Cons
Limitations
Future ideas
Seminar outlineSeminar outline
Problem presentation
Does depth affects saliency?
If so How to incorporate depth data?
Problem motivationProblem motivation
Why saliency?SurveillanceSearch and RescueMedical applicationImage automated croppingAdvertisingTarget recognitionVideo summarizationsPreprocess for other algorithms
Problem motivationProblem motivation
Why incorporate depth?
Are observers fixation different when viewing 3D images?
Human natural visual attention evolved in 3D environment
Absent of 3D fixation data
Problem motivationProblem motivation
Finding efficient saliency models
Difficult to model top-down processes effect
Integrating depth additional information
Absent datasets for 3D stimulus
Absent of images dataset with corresponding depth maps
Subjectivel
ChallengesChallenges
Which one is more conspicuous?
Computing Visual Attention from Scene Depth (2000) in ICPRAuthors :Authors :Nabil Ouerhani and Heinz Hiigli
Related workRelated work
Intro
Based on Itti & Koch saliency algorithm
Easy to compute in parallel
Easy to incorporate depth features
Only visual evaluation method
Top-Down overview
VideoCamera
Range Finder
Feature Map 1
Feature Map i
Feature Map n+m
Center Surround
Center Surround
Center Surround
Conspicuity Map 1
Conspicuity Map i
Conspicuity Map n+m
Feature extraction Integration
SaliencyMap
Assign weights that promotes conspicuity
Σ Wi
M
m
Wi
Conspicuity map i
Integration
step
Limitation
No statistical analysis to prove
improvements
No comparison with other methods
No quantities evaluation method
Proposed solution advantages
A new 3D dataset was created for statitical analysis
Evaluation and comparison with different methods
Experiments to investigate 3D saliency
Related workRelated workPre-Attentive detection of depth saliency using stereo vision(2010) in AIPRAuthors :Authors :M. Zaheer Aziz and Barbel Mertsching
Intro
Depth approximation using stereo images
Relates only to depth saliency
Top-Down overview
Algorithm
IR
IL
Depth saliency magnitude
Preprocessing
IR
IL
Depth saliency magnitude
Clipp
ing
Smooth
ing
Segmentat
ion
1 1 1
1 1 1
1 1 1
1/9
IR
Key insight
IRIL
IR IL
Main Algorithm explained/1
IR
IL
Segmentation
-
Δ(x, y) Remove extra stripes
Main Algorithm explained/2
Remove occluded stripes Assign region depth
Human subjects marked depth saliency
labels
Efficiency factor defined
Capability factor defined
Experiments and results
Nf =# of labels found
Ns =# of labels
Nf
Ns
Nf
Ns
Nf<Ns
Nf>NsPenalty
Experiments and results
Related workRelated work
Limitation
Requires stereo images
Approximated depth calculation
Emphasis on runtime
Experiments somehow dubious
Self invented evaluation index
Hard to compare with other methods
Ignoring fusion with contrast saliency algo.
Related workRelated work
Proposed solution advantages
Messured depth data
Statistical analysis and comparison
Applicable on all 2D saliency algorithms
Using conventional evaluation index
Top-Down overview
Dataset collection (3D/2D) and analysisExtracting Stereoscopic image pair generation for 3D display
Perform experiments to gain fixation maps
Observations and Statistics
Incorporating depth priors
Experiments and results
Proposed solutionProposed solution
Experiment setup
Rejected images from dataset images overlapping content with other images Images with significant artifacts after the smoothing process
Dataset Examples
1 1 11 0 11 1 1
D==0 ?
1/8∙
D==0 ?
In the same super pixel
LaplacianSmoothDepth map
Depth Map
Apply smoothing to depth mapsApply smoothing to depth maps
Stereoscopic image pair generation for 3D display
Pixels translation:xl = xp+ρ, xr = xp−ρρ = parallax/2
Experiments conducted usingactive shutter glasses on a3D LCD display
Three quantitative methods for performance evaluation
Correlation coefficient(for all images in dataset)
o Correlation between to given maps for different depth range bins
Saliency Ratio
o Saliency mean energy in depth bin
AUC of the ROC
o Use each one saliency map to predict another
Quantitative evaluation methods/1
Correlation coefficient
Saliency Ratio
AUC of the ROC
Quantitative evaluation methods/2
Observation 1
Depth cues modulate visual saliency at farther depth ranges
humans fixate preferentially at closer depth ranges
Observations and statistics
Observation 2
few objects account for majority of the fixations
The average AUC for the entire 3D fixation dataset is 0.7399and 0.7046 for 2D fixation dataset
Observations and statistics
Using fixation maps as predictor for labeled object
Observations and statistics
Observation 3
The relationship between depth and saliency is non-linear and
characteristic for low and high depth-of-field scenes
CC – Correlation coefficient
Observations and statistics
Observation 4:Fixation distribution discrepancy when multiple salient stimuli present different depth planes
GMM model ParametersFor each DOF bin
P(s(x)|d(x), DOF)
Training set(80% of thedataset)
200x200 pix
Incorporating depth priors/1
EMAlgo.
DownSample
S(x) = ψ(x)( / )P(s/d,DOF)⊕ ⊗
Proposed pdf P(s,d/DOF) is GMM (Gaussian Mixture Model)
640x480 pix
GMM PdfFrom training
Saliency mapSaliency
Algo.
Incorporating depth priors/2
Input image
+/*ImprovedSaliency
map
Depth mapCalcDOF
SelectedGMM Pdf
Experiments and results
Experiments and results
Experiments and results
Proposed solutionProposed solution
Depth saliency
Pros & Cons Pros & Cons
ProsNovel and provides a basis for further researchSimple approachGives an overall improvement for existing alg.
ConsFew missing implementation detailsNo source code/other descriptive materialRequires a depth mapStill no basis for comparison with other methods
Future ideasFuture ideasIntegrate depth priors in various algo. instead of late fusion
Questions ?
Thanks