Slide 1

Recap from Previous LectureTone MappingPreserve local contrast or detail at the expense of large scale contrast.Changing the brightness within objects or surfaces unequally leads to halos.We are now transitioning to more geometric reasoning about light.Exam 1 ResultsAverage Grade: 80%Question 17 not graded.Project 2 Resultshttp://www.cs.brown.edu/courses/cs129/results/proj2/zyp/http://www.cs.brown.edu/courses/cs129/results/proj2/damoreno/Project 2 ResultsJonathon Mace

4Project 2 ResultsXiaofeng Tao

Project 2 ResultsChen Xu

Project 2 ResultsJonathan Koh

Project 2 ResultsMarvin Arroz

Stereo and Depth Estimation (Sz 11)cs129: Computational PhotographyJames Hays, Brown, Fall 2012Slides from Steve Seitz and Robert Collins

3d movies are trendy lately, but arguably they should be called stereo 3d movies because movies already have many 3d cues.9How is depth estimated?Time of Flight

Humans can learn to echolocate!

Stereo VisionNot that important for humans, especially at longer distances. Perhaps 10% of people are stereo blind.Many animals dont have much stereo overlap in their fields of view

Public Library, Stereoscopic Looking Room, Chicago, by Phillips, 1923

Teesta suspension bridge-Darjeeling, India

Woman getting eye exam during immigration procedure at Ellis Island, c. 1905 - 1920 , UCR Museum of Phography

Mark Twain at Pool Table", no date, UCR Museum of Photography

Stereoscene pointoptical centerimage planeStereoBasic Principle: TriangulationGives reconstruction as intersection of two raysRequires camera pose (calibration)point correspondenceStereo correspondenceDetermine Pixel CorrespondencePairs of points that correspond to same scene pointEpipolar ConstraintReduces correspondence problem to 1D search along conjugate epipolar linesepipolar planeepipolar lineepipolar lineFundamental matrixLet p be a point in left image, p in right image

Epipolar relationp maps to epipolar line l p maps to epipolar line l Epipolar mapping described by a 3x3 matrix F

It follows thatllpp

Fundamental matrixThis matrix F is calledthe Essential Matrixwhen image intrinsic parameters are knownthe Fundamental Matrixmore generally (uncalibrated case)

Can solve for F from point correspondencesEach (p, p) pair gives one linear equation in entries of F

8 points give enough to solve for F (8-point algorithm)see Marc Pollefeys notes for a nice tutorial

Stereo image rectification33Stereo image rectificationreproject image planes onto a commonplane parallel to the line between optical centerspixel motion is horizontal after this transformationtwo homographies (3x3 transform), one for each input image reprojectionC. Loop and Z. Zhang. Computing Rectifying Homographies for Stereo Vision. IEEE Conf. Computer Vision and Pattern Recognition, 1999.34Stereo matching algorithmsMatch Pixels in Conjugate Epipolar LinesAssume brightness constancyThis is a tough problemNumerous approachesA good survey and evaluation: http://www.middlebury.edu/stereo/ Your basic stereo algorithm

For each epipolar lineFor each pixel in the left imagecompare with every pixel on same epipolar line in right imagepick pixel with minimum match costImprovement: match windowsThis should look familar...

Window sizeSmaller window Larger window

W = 3W = 20Better results with adaptive windowT. Kanade and M. Okutomi, A Stereo Matching Algorithm with an Adaptive Window: Theory and Experiment,, Proc. International Conference on Robotics and Automation, 1991. D. Scharstein and R. Szeliski. Stereo matching with nonlinear diffusion. International Journal of Computer Vision, 28(2):155-174, July 1998

Effect of window sizesmaller window: more detail, more noisebigger window: less noise, more detailStereo results

Ground truthSceneData from University of TsukubaSimilar results on other images without ground truth

38Results with window search

Window-based matching(best window size)

Ground truthBetter methods exist...

Better MethodBoykov et al., Fast Approximate Energy Minimization via Graph Cuts, International Conference on Computer Vision, September 1999.Ground truth

For the latest and greatest: http://www.middlebury.edu/stereo/ Stereo as energy minimizationWhat defines a good stereo correspondence?Match qualityWant each pixel to find a good match in the other imageSmoothnessIf two pixels are adjacent, they should (usually) move about the same amount

Stereo as energy minimizationExpressing this mathematicallyMatch qualityWant each pixel to find a good match in the other image

SmoothnessIf two pixels are adjacent, they should (usually) move about the same amount

We want to minimizeThis is a special type of energy function known as an MRF (Markov Random Field)Effective and fast algorithms exist:Graph cuts, belief propagation.for more details (and code): http://vision.middlebury.edu/MRF/ Great tutorials available online (including video of talks)

Depth from disparityfxxbaselinezCCXf

Camera calibration errorsPoor image resolutionOcclusionsViolations of brightness constancy (specular reflections)Large motionsLow-contrast image regionsStereo reconstruction pipelineStepsCalibrate camerasRectify imagesCompute disparityEstimate depthWhat will cause errors?Active stereo with structured light

Project structured light patterns onto the objectsimplifies the correspondence problem

camera 2camera 1projector

camera 1projectorLi Zhangs one-shot stereoActive stereo with structured light

Laser scanningOptical triangulationProject a single stripe of laser lightScan it across the surface of the objectThis is a very precise version of structured light scanning

Digital Michelangelo Projecthttp://graphics.stanford.edu/projects/mich/

Laser scanned models

The Digital Michelangelo Project, Levoy et al.Laser scanned modelsThe Digital Michelangelo Project, Levoy et al.

Laser scanned models

The Digital Michelangelo Project, Levoy et al.Laser scanned models

The Digital Michelangelo Project, Levoy et al.Laser scanned models

The Digital Michelangelo Project, Levoy et al.