CSE 803 Fall 2008 Stockman 1

Structure from Motion

A moving camera/computer computes the 3D structure of the scene and its own motion

CSE 803 Fall 2008 Stockman 2

Sensing 3D scene structure via a moving camera

We now have two views over time/space compared to stereo which has multiple views at the same time.

CSE 803 Fall 2008 Stockman 3

Assumptions for now

The scene is rigid. The scene may move or the

camera may move giving a sequence of 2 or more 2D images

Corresponding 2D image points (Pi, Pj) are available across the images

CSE 803 Fall 2008 Stockman 4

What can be computed

The 3D coordinates of the scene points

The motion of the camera Camera sees many frames of 2D points

Rigid scene with many 3D interest points

From Jabara, Azarbayejani, Pentland

CSE 803 Fall 2008 Stockman 5

From 2D point correspondences, compute 3D points and TR

CSE 803 Fall 2008 Stockman 6

applications

We can compute a 3D model of a landmark from a video

We can compute the trajectory of the sensor relative to the 3D object points

CSE 803 Fall 2008 Stockman 7

Use only 2D correspondences, SfM can compute 3D jig pts

… up to one scale factor.

CSE 803 Fall 2008 Stockman 8

http://www1.cs.columbia.edu/~jebara/htmlpapers/SFM/sfm.html

Jabara, Azarbayejani, Pentland

a) Two video frames with corresponding 2D interest points. 3D points can be computed from SfM method.

b) Some edges detected from 2D gradients.

c) Texture mapping from 2D frames onto 3D polyhedral model.

d) 3D model can be viewed arbitrarily!

CSE 803 Fall 2008 Stockman 9

Virtual museums Much work, and software, from about 10

years ago. 3D models, including shape and texture

can be made of famous places (Notre Dame, Taj Mahal, Titanic, etc.) and made available to those who cannot travel to see the real landmark.

Theoretically, only quality video is required.

Usually, some handwork is needed.

CSE 803 Fall 2008 Stockman 10

SfM methods Typically require careful mathematics EX: from 5 matched points, get 10

equations to estimate 10 unknowns; also a more popular 8 pt linear method

Methods must consider effects of noise See Faugeras et al and ******** Methods can run in real time

CSE 803 Fall 2008 Stockman 11

Special mathematics

Epipolar geometry Fundamental matrix: computed

from a pair of cameras Essential matrix: specialization of

fundamental matrix when calibration is available

CSE 803 Fall 2008 Stockman 12

Finishing the course: 2 options

HW7 on stereo computation (Sat 6 Dec) (60%/7)

Final Exam (25%): Monday 8 Dec 12:45-2:45

SfM report (6 Dec) and demonstration. Find existing code and get it to work on CSE 803 images

(60%/7 + 10%) Final Exam (15%)

limited problems