Upload
biswabandan-satpathy
View
215
Download
4
Tags:
Embed Size (px)
Citation preview
Introduction to Introduction to binocular stereo visionbinocular stereo vision
Introduction to binocular stereo vision 2
What is binocular stereo vision?What is binocular stereo vision?
• A way of getting depth (3-D) information about A way of getting depth (3-D) information about a scene from two 2-D views (images) of the a scene from two 2-D views (images) of the scenescene
Introduction to binocular stereo vision 3
What is binocular stereo vision?What is binocular stereo vision?
• A way of getting depth (3-D) information about A way of getting depth (3-D) information about a scene from two 2-D views (images) of the a scene from two 2-D views (images) of the scenescene
• Used by humans and animalsUsed by humans and animals
Introduction to binocular stereo vision 4
What is binocular stereo vision?What is binocular stereo vision?
• A way of getting depth (3-D) information about A way of getting depth (3-D) information about a scene from two 2-D views (images) of the a scene from two 2-D views (images) of the scenescene
• Used by humans and animalsUsed by humans and animals• Computational stereo visionComputational stereo vision
– Programming machines to do stereo visionProgramming machines to do stereo vision– Studied extensively in the past 25 yearsStudied extensively in the past 25 years– Difficult; still being researchedDifficult; still being researched
Introduction to binocular stereo vision 5
Purpose of this lecture:Purpose of this lecture:
• An introduction to:An introduction to:– Basic principle of stereo visionBasic principle of stereo vision– Computational stereo analysisComputational stereo analysis
• How does it work?How does it work?• What is required?What is required?• Where are the difficulties?Where are the difficulties?
Introduction to binocular stereo vision 6
Purpose of this lecture:Purpose of this lecture:
• An introduction to:An introduction to:– Basic principle of stereo visionBasic principle of stereo vision– Computational stereo analysisComputational stereo analysis
• How does it work?How does it work?• What is required?What is required?• Where are the difficulties?Where are the difficulties?
Introduction to binocular stereo vision 7
Fundamentals of stereo visionFundamentals of stereo vision
• A camera model:A camera model:– Models how 3-D scene points are transformed into 2-Models how 3-D scene points are transformed into 2-
D image pointsD image points– The pinhole camera: a simple linear model for The pinhole camera: a simple linear model for
perspective projectionperspective projection
Introduction to binocular stereo vision 8
Fundamentals of stereo visionFundamentals of stereo vision
• The goal of stereo analysis:The goal of stereo analysis:– The inverse process: From 2-D image coordinates to The inverse process: From 2-D image coordinates to
3-D scene coordinates3-D scene coordinates– Requires images from at least two viewsRequires images from at least two views
Introduction to binocular stereo vision 9
Fundamentals of stereo visionFundamentals of stereo vision
• 3-D reconstruction3-D reconstruction
Introduction to binocular stereo vision 10
Fundamentals of stereo visionFundamentals of stereo vision
• 3-D reconstruction3-D reconstruction
Introduction to binocular stereo vision 11
Fundamentals of stereo visionFundamentals of stereo vision
• 3-D reconstruction3-D reconstruction
Introduction to binocular stereo vision 12
Fundamentals of stereo visionFundamentals of stereo vision
• 3-D reconstruction3-D reconstruction
Introduction to binocular stereo vision 13
Fundamentals of stereo visionFundamentals of stereo vision
• 3-D reconstruction3-D reconstruction
Introduction to binocular stereo vision 14
Fundamentals of stereo visionFundamentals of stereo vision
• 3-D reconstruction3-D reconstruction
Introduction to binocular stereo vision 15
Fundamentals of stereo visionFundamentals of stereo vision
• 3-D reconstruction3-D reconstruction
Introduction to binocular stereo vision 16
Fundamentals of stereo visionFundamentals of stereo vision
• 3-D reconstruction3-D reconstruction
Introduction to binocular stereo vision 17
PrerequisitesPrerequisites
• Camera model parameters must be known:Camera model parameters must be known:
– External parameters: External parameters: • Positions, orientationsPositions, orientations
– Internal parameters:Internal parameters:• Focal length, image center, distortion, etc..Focal length, image center, distortion, etc..
Introduction to binocular stereo vision 18
PrerequisitesPrerequisites
• Camera calibrationCamera calibration
Introduction to binocular stereo vision 19
Two subproblemsTwo subproblems
• Matching Matching – Finding corresponding elements in the two imagesFinding corresponding elements in the two images
• ReconstructionReconstruction– Establishing 3-D coordinates from the 2-D image Establishing 3-D coordinates from the 2-D image
correspondences found during matchingcorrespondences found during matching
Introduction to binocular stereo vision 20
Two subproblemsTwo subproblems
• Matching Matching (hardest)(hardest)– Finding corresponding elements in the two imagesFinding corresponding elements in the two images
• ReconstructionReconstruction– Establishing 3-D coordinates from the 2-D image Establishing 3-D coordinates from the 2-D image
correspondences found during matchingcorrespondences found during matching
Introduction to binocular stereo vision 21
• Which image entities should be matched?Which image entities should be matched?– Two main approachesTwo main approaches
• Pixel/area-based (lower-level)Pixel/area-based (lower-level)• Feature-based (higher-level)Feature-based (higher-level)
The matching problemThe matching problem
Introduction to binocular stereo vision 22
Matching challengesMatching challenges
• Scene elements do not always look the same Scene elements do not always look the same in the two imagesin the two images– Camera-related problemsCamera-related problems
• Image noise, differing gain, contrast, etc..Image noise, differing gain, contrast, etc..
– Viewpoint-related problems:Viewpoint-related problems:• Perspective distortionsPerspective distortions• OcclusionsOcclusions• Specular reflectionsSpecular reflections
Introduction to binocular stereo vision 23
Choice of camera setupChoice of camera setup
• BaselineBaseline– distance between cameras (focal points)distance between cameras (focal points)
• Trade-offTrade-off– Small baseline: Matching easierSmall baseline: Matching easier– Large baseline: Depth precision betterLarge baseline: Depth precision better
Introduction to binocular stereo vision 24
Matching cluesMatching clues
• Correspondance search is a 1-D problemCorrespondance search is a 1-D problem– Matching point must lie on a lineMatching point must lie on a line
Introduction to binocular stereo vision 25
Matching cluesMatching clues
• Epipolar geometryEpipolar geometry
Introduction to binocular stereo vision 26
Matching cluesMatching clues
• Epipolar geometryEpipolar geometry
Introduction to binocular stereo vision 27
RectificationRectification
• Simplifies the correspondance searchSimplifies the correspondance search– Makes all epipolar lines parallel and coincidentMakes all epipolar lines parallel and coincident– Corresponds to parallel camera configurationCorresponds to parallel camera configuration
Introduction to binocular stereo vision 28
Goal: disparity mapGoal: disparity map
• Disparity: Disparity: – The horizontal displacement between corresponding The horizontal displacement between corresponding
pointspoints– Closely related to scene depthClosely related to scene depth
Introduction to binocular stereo vision 29
More matching heuristicsMore matching heuristics
• Always valid:Always valid:– (Epipolar line)(Epipolar line)– UniquenessUniqueness– Minimum/maximum disparityMinimum/maximum disparity
• Sometimes valid:Sometimes valid:– OrderingOrdering– Local continuity (smoothness)Local continuity (smoothness)
Introduction to binocular stereo vision 30
Area-based matchingArea-based matching
• Finding pixel-to-pixel correspondencesFinding pixel-to-pixel correspondences– For each pixel in the left image, search for the most For each pixel in the left image, search for the most
similar pixel in the right imagesimilar pixel in the right image
Introduction to binocular stereo vision 31
Area-based matchingArea-based matching
• Finding pixel-to-pixel correspondencesFinding pixel-to-pixel correspondences– For each pixel in the left image, search for the most For each pixel in the left image, search for the most
similar pixel in the right imagesimilar pixel in the right image– Using neighbourhood windowsUsing neighbourhood windows
Introduction to binocular stereo vision 32
Area-based matchingArea-based matching
• Similarity measures for two windowsSimilarity measures for two windows– SAD (sum of absolute differences)SAD (sum of absolute differences)– SSD (sum of squared differences)SSD (sum of squared differences)– CC (cross-correlation)CC (cross-correlation)– ……
Introduction to binocular stereo vision 33
Feature-based matchingFeature-based matching
• Matching features:Matching features:– Edge pointsEdge points– lineslines– cornerscorners– ……
• Sparse reconstruction setsSparse reconstruction sets• Best if scene type is known Best if scene type is known a prioria priori
Introduction to binocular stereo vision 34
Area-based matchingArea-based matching
• Choice of window sizeChoice of window size– Factors to considers:Factors to considers:
• AmbiguityAmbiguity• Noise sensitivityNoise sensitivity• Sensitivity towards viewpoint-related distortionsSensitivity towards viewpoint-related distortions• Expected object sizesExpected object sizes• Frequency of depth jumpsFrequency of depth jumps
Introduction to binocular stereo vision 35
Area-based matchingArea-based matching
• Variable window positionVariable window position– Better matching at depth jumps (disparity edges)Better matching at depth jumps (disparity edges)
Introduction to binocular stereo vision 36
Three or more viewpointsThree or more viewpoints
• More matching informationMore matching information– Additional epipolar constraintsAdditional epipolar constraints– More confident matchesMore confident matches
Introduction to binocular stereo vision 37
SummarySummary
• Stereo vision: Stereo vision: – A method for 3-D analysis of a scene using images A method for 3-D analysis of a scene using images
from two or more viewpointsfrom two or more viewpoints
• Two subproblems:Two subproblems:– MatchingMatching– ReconstructionReconstruction
• Most difficult part: MatchingMost difficult part: Matching– Two main approaches:Two main approaches:
• Area based: Dense reconstructionArea based: Dense reconstruction• Feature based: Sparse reconstructionFeature based: Sparse reconstruction
Introduction to binocular stereo vision 38
Modelling stereo quantification Modelling stereo quantification errorerror
Introduction to binocular stereo vision 39
Stereo error quantificationStereo error quantification
The variance:
Numerical solution:
Introduction to binocular stereo vision 40
Error analytical vs. Numerical Error analytical vs. Numerical solutionsolution