95

Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

  • Upload
    others

  • View
    3

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model
Page 2: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Review

• Model fitting

– Least squares / robust least squares

– RANSAC

– Iterative Closest Points

• Models

– 2D image transformations

Page 3: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Review: RANSAC

14INAlgorithm:

1. Sample (randomly) the number of points required to fit the model (#=2)

2. Solve for model parameters using samples

3. Score by the fraction of inliers within a preset threshold of the model

Repeat 1-3 until the best model is found with high confidence

Page 4: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Review: 2D image transformations

Szeliski 2.1

Page 5: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

What if I want to fit multiple models?What if my lines are in segments?

http://ostatic.com/files/images/ss_hough.jpg

Page 6: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Start with edge detection Canny

Page 7: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Edge gradients• Equation of line:

• Recall: when we detect an edge pixel, we can estimate its gradient m.

• With the (x,y) position of thepixel, we can estimate b.

• Thus, each edge pixel (edgel!) represents a line.

• Hough transform: What if each edge pixel voted for the line it might represent?

y = m x + b

P.V.C. Hough, Machine Analysis of Bubble Chamber Pictures,

Proc. Int. Conf. High Energy Accelerators and Instrumentation, 1959.

Page 8: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Hough Transform: Outline

• Create a grid of candidate m,b parameter values.• Why a grid?

• m,b are continuous; grid discretizes into hypotheses.

y

x

Edge image Parameter space (Hough space)

-b

-m+b

+m

Page 9: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Hough Transform: Outline

• Create a grid of candidate m,b parameter values.• Why a grid?

• m,b are continuous; grid discretizes into hypotheses.

• Each edge pixel votes for a set of parameters, which increments those values in grid.

+b

+my

x

Edge image Parameter space (Hough space)

-b

-m

+1

Page 10: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Hough Transform: Outline

• Create a grid of candidate m,b parameter values.• Why a grid?

• m,b are continuous; grid discretizes into hypotheses.

• Each edge pixel votes for a set of parameters, which increments those values in grid.

+b

+my

x

Edge image Parameter space (Hough space)

-b

-m

+100

Each line in

image is a

point (or cell) in

Hough space.

Page 11: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Hough Transform: Outline

• Create a grid of candidate m,b parameter values.• Why a grid?

• m,b are continuous; grid discretizes into hypotheses.

• Each edge pixel votes for a set of parameters, which increments those values in grid.

+b

+my

x

Edge image Parameter space (Hough space)

-b

-m

101 +1

Page 12: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Hough Transform: Outline

• Create a grid of candidate m,b parameter values.• Why a grid?

• m,b are continuous; grid discretizes into hypotheses.

• Each edge pixel votes for a set of parameters, which increments those values in grid.

+b

+my

x

Edge image Parameter space (Hough space)

-b

-m

101 1

+1

Page 13: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Hough Transform: Outline

• Create a grid of candidate m,b parameter values.• Why a grid?

• m,b are continuous; grid discretizes into hypotheses.

• Each edge pixel votes for a set of parameters, which increments those values in grid.

+b

+my

x

Edge image Parameter space (Hough space)

-b

-m

101 120

20

All pixels

voted427

89 64

14

2933

21

4517 164

31138 282

Page 14: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Hough Transform: Outline

• Create a grid of candidate m,b parameter values.• Why a grid?

• m,b are continuous; grid discretizes into hypotheses.

• Each edge pixel votes for a set of parameters, which increments those values in grid.

• Find maxima – our line candidates.

+b

+my

x

Edge image Parameter space (Hough space)

-b

-m

101 120

20

All pixels

voted427

89 64

14

2933

21

4517 164

31138 282

Page 15: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Hough Transform: Step back

• Hough space represents all possible lines.

• With gradient information constriction:• Edgel is single point in Hough space.

• Without gradient orientation information?

• Think-Pair-Share as orientation varies – ramifications!

+b

+my

x

Image space Parameter space (Hough space)

-b

-m

Page 16: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Hough Transform: Step back

• Hough space represents all possible lines.

• With gradient information constriction:• Edgel is single point in Hough space.

• Without gradient orientation information?

+b

+my

x

Image space Parameter space (Hough space)

-b

-m

Page 17: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Hough Transform: Step back

• Hough space represents all possible lines.

• With gradient information constriction:• Edgel is single point in Hough space.

• Without gradient orientation information?

+b

+my

x

Image space Parameter space (Hough space)

-b

-m

Page 18: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Hough Transform: Step back

• Hough space represents all possible lines.

• With gradient information constriction:• Edgel is single point in Hough space.

• Without gradient orientation information?

+b

+my

x

Image space Parameter space (Hough space)

-b

-m

Page 19: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Hough Transform: Step back

• Hough space represents all possible lines.

• With gradient information constriction:• Edgel is single point in Hough space.

• Without gradient orientation information?

+b

+my

x

Image space Parameter space (Hough space)

-b

-m

Page 20: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Hough Transform: Step back

• Hough space represents all possible lines.

• With gradient information constriction:• Edgel is single point in Hough space.

• Without gradient orientation information?• Unoriented point is line is Hough space.

+b

+my

x

Image space Parameter space (Hough space)

-b

-m

Page 21: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Hough Transform: Step back

• Hough space represents all possible lines.

• With gradient information constriction:• Edgel is single point in Hough space.

• Without gradient orientation information?• How big is Hough space?

+b

+my

x

Image space Parameter space (Hough space)

-b

-m

Problem.

m = inf

b = -inf

How to fix?

Page 22: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Hough Transform: Line Normal Form

• Use

• Space is 0 to 360

• Use r = distance to line from some origin•

• Space is ± max_𝑥2 +max _𝑦2

+ry

x

Image space Parameter space (Hough space)

r = 3

ϴ = 30

sin cos iiiii yxr

Page 23: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Hough Transform: Line Normal Form

• In this line form, unoriented edge draws a sinusoid in Hough space.

+ry

x

Image space Parameter space (Hough space)

-r

Page 24: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Image r,ϴ model parameter histogram

Hough transform - experiments

Slide from S. Savarese

r

Next few images ignore edge orientation.

Each point is one sinusoid.

y

x

Page 25: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Image

Need to adjust grid size or smooth

Hough transform - experiments

Noisy data

Slide from S. Savarese

r,ϴ model parameter histogram

y

x

Page 26: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Issue: spurious peaks due to uniform noise

Image

Hough transform - experiments

Slide from S. Savarese

ρ,ϴ model parameter histogram

y

x

Page 27: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Hough transform example

http://ostatic.com/files/images/ss_hough.jpg

Page 28: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

1. Image Canny

Page 29: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

2. Canny Hough votes

Page 30: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

3. Hough votes Edges

Find peaks and post-process.

Page 31: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Finding lines using Hough transform

• Using known edge orientation to vote for a single line (rather than accumulate over all θ).

• Practical considerations

– Bin size

– Smoothing

– Finding multiple lines

– Finding line segments

• Can ‘fit’ line to edgels that ‘survive the vote’ for more precise estimation.

Page 32: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Hough transform conclusionsGood• Robust to outliers: each point votes separately.• Edge orientation -> fairly efficient (faster than trying all parameter sets).• Provides multiple model fitting.

Bad• Some sensitivity to noise• Bin size trades off between noise tolerance, precision, and

speed/memory– Can be hard to find sweet spot.

• Not suitable for more than a few parameters– Grid size grows exponentially.

Common applications• Line fitting (also circles, ellipses, etc.)• Object instance recognition (parameters are affine transform)• Object category recognition (parameters are position/scale)

James Hays

Page 33: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

FEATURE DETECTION AND MATCHING – DONE.

Computer Vision c. 2007

Page 34: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Overview of Keypoint Matching

K. Grauman, B. Leibe

Np

ixe

ls

N pixels

Af

e.g. color

Bf

e.g. color

A1

A2 A3

Tffd BA ),(

1. Find a set of

distinctive key-

points

3. Extract and

normalize the

region content

2. Define a region

around each

keypoint

4. Compute a local

descriptor from the

normalized region

5. Match local

descriptors

Page 35: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Object Instance Recognition

1. Match keypoints to object model

2. Solve for affine transformation parameters

3. Score by inliers and choose solutions with score above threshold

A1

A2

A3

Affine Parameters

Choose hypothesis with max

score above threshold

# Inliers

Matched

keypoints

James Hays

Page 36: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Finding objects (SIFT, Lowe 2004)1. Match interest points from input image to database image.

2. Get location/scale/orientation using Hough voting.

– In database image, each point has known position/scale/orientation wrt. whole object.

– Matched points vote for the position, scale, and orientation of the entire object.

– Bins for x, y, scale, orientation• Wide bins (0.25 object length in position, 2x scale, 30 degrees orientation)

• Vote for two closest bin centers in each direction (16 votes total)

3. Geometric verification for each bin with at least 3 keypoints

– Iterate least squares fit and checking for inliers and outliers

– (Advanced) Compute affine registration to check model fit.

4. Report object if > T inliers (T is typically 3, can be computed to match some probabilistic threshold)

James Hays

Page 37: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Examples of recognized objects

Page 38: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

CAMERAS, MULTIPLE VIEWS, AND MOTION

Page 39: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

What is a camera?

Page 40: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model
Page 41: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Camera obscura: dark room

• Known during classical period in China and Greece (e.g., Mo-Ti, China, 470BC to 390BC)

Illustration of Camera Obscura Freestanding camera obscura at UNC Chapel Hill

Photo by Seth Ilys

James Hays

Page 42: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Camera obscura / lucida used for tracing

Lens Based Camera Obscura, 1568

drawingchamber.wordpress.com

Camera lucida

Page 43: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Tim’s Vermeer

Vermeer, The Music Lesson, 1665 Tim Jenison (Lightwave 3D, Video Toaster)

Page 44: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Tim’s Vermeer – video still

Page 45: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

First Photograph

Oldest surviving photograph

– Took 8 hours on pewter plate

Joseph Niepce, 1826

Photograph of the first photograph

Stored at UT Austin

Niepce later teamed up with Daguerre, who eventually created Daguerrotypes

Page 46: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

The Geometry of Image Formation

Mapping between image and world coordinates

– Pinhole camera model

– Projective geometry

• Vanishing points and lines

– Projection matrix

Szeliski 2.1, parts of 2.2

Slides from James Hays, Derek Hoiem, Alexei Efros, Steve Seitz, and David Forsyth

Page 47: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Point of observation

Figures © Stephen E. Palmer, 2002

Dimensionality Reduction Machine (3D to 2D)

3D world 2D image

Page 48: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Lake Sørvágsvatn in Faroe Islands

100 metres above sea level

Page 49: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Lake Sørvágsvatn in Faroe Islands

100 30 metres above sea level

amusingplanet.com, thanks to Aaron Gokaslan

Page 50: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model
Page 51: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model
Page 52: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Holbein’s The Ambassadors - 1533

Page 53: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Holbein’s The Ambassadors – Memento Mori

Page 54: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Cameras and World Geometry

How tall is this woman?

Which ball is closer?

How high is the camera?

What is the camera

rotation wrt. world?

James Hays

Page 55: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Let’s design a camera

Idea 1: Put a sensor in front of an object

Do we get a reasonable image?

Slide source: Seitz

sensor

Page 56: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Let’s design a camera

Idea 2: Add a barrier to block most rays– Pinhole in barrier

– Only sense light from one direction.• Reduces blurring.

– In most cameras, this aperture can vary in size.

Slide source: Seitz

sensor

Page 57: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Pinhole camera model

Figure from Forsyth

f

f = Focal length

c = Optical center of the camera

c

Real

object

Page 58: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Projection: world coordinatesimage coordinates

Camera

Center

(0, 0, 0)

Z

Y

X

P.

.

. f Z Y

V

Up

.V

U

Z

fXU *

Z

fYV *

What is the effect if f and Z are equal?

p = distance from

image center

Image

center

(u0, v0)

Page 59: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Projective Geometry

Length (and so area) is lost.

Which is closer?

Who is taller?

Page 60: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Length and area are not preserved

Figure by David Forsyth

B’

C’

A’

Page 61: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Projective Geometry

Perpendicular?

Parallel?

Angles are lost.

Page 62: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Projective Geometry

What is preserved?

• Straight lines are still straight.

Page 63: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Vanishing points and lines

Parallel lines in the world

intersect in the projected image at a “vanishing point”.

Parallel lines on the same plane in the world converge to vanishing points on a “vanishing line”.

E.G., the horizon.

Vanishing Point Vanishing Point

Vanishing Line

Page 64: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Vanishing points and lines

Vanishingpoint

Vanishingpoint

Vertical vanishingpoint

(at infinity)

Slide from Efros, Photo from Criminisi

Page 65: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model
Page 66: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Pinhole camera model

Forsyth

f

f = Focal length

c = Optical center of the camera

c

Real

object

Page 67: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Projection: world coordinatesimage coordinates

Camera

Center

(0, 0, 0)

Z

Y

X

P.

.

. f Z Y

V

Up

.V

U

Z

fXU *

Z

fYV *

What is the effect if f and Z are equal?

p = distance from

image center

Image

center

(u0, v0)

Page 68: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Slide Credit: Savarese

Camera (projection) matrix

XtRKx x: Image Coordinates: (u,v,1)

K: Intrinsic Matrix (3x3)

R: Rotation (3x3)

t: Translation (3x1)

X: World Coordinates: (X,Y,Z,1)

Ow

iw

kw

jwR,t

X

x

Extrinsic Matrix

Page 69: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Projective geometry

• 2D point in cartesian = (x,y) coordinates

• 2D point in projective = (x,y,w) coordinates

Idea from www.tomdalling.com

Y

X

Projector

Page 70: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Projective geometry

• 2D point in cartesian = (x,y) coordinates

• 2D point in projective = (x,y,w) coordinates

Y

X

Projector W

Idea from www.tomdalling.com

Page 71: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Varying w

w1 w2 < w1

Projected image becomes smaller.

Y

X

Projector

Y

X

Projector

Page 72: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Projective geometry

• 2D point in projective = (x,y,w) coordinates

– w defines the scale of the projected image.

– Each x,y point becomes a ray!

Y

X

Projector W

Page 73: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Projective geometry

• In 3D, point (x,y,z) becomes (x,y,z,w)

• Perspective is w varying with z:

– Objects far away are appear smaller

B’

C’

Page 74: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Homogeneous coordinates

Converting to homogeneous coordinates

2D (image) coordinates 3D (scene) coordinates

Converting from homogeneous coordinates

2D (image) coordinates 3D (scene) coordinates

Page 75: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Homogeneous coordinates

Scale invariance in projection space

w

y

wx

kw

ky

kwkx

kw

ky

kx

w

y

x

k

Homogeneous Coordinates

Cartesian Coordinates

E.G., we can uniformly scale the projective space, and it will still

produce the same image -> scale ambiguity

Page 76: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Slide Credit: Savarese

Camera (projection) matrix

XtRKx x: Image Coordinates: (u,v,1)

K: Intrinsic Matrix (3x3)

R: Rotation (3x3)

t: Translation (3x1)

X: World Coordinates: (X,Y,Z,1)

Ow

iw

kw

jwR,t

X

x

Extrinsic Matrix

Page 77: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

X0IKx

10100

000

000

1z

y

x

f

f

v

u

w

K

Slide Credit: Savarese

Projection matrix

Intrinsic Assumptions

• Unit aspect ratio

• Optical center at (0,0)

• No skew

Extrinsic Assumptions• No rotation

• Camera at (0,0,0)

X

x(0,0,0)

Page 78: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Projection: world coordinatesimage coordinates

Camera

Center

(0, 0, 0)

Z

Y

X

P.

.

. f Z Y

V

Up

.V

U

Z

fXU *

Z

fYV *

p = distance from

image center

Image

center

(u0, v0)

Page 79: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Remove assumption: known optical center

X0IKx

10100

00

00

1

0

0

z

y

x

vf

uf

v

u

w

Intrinsic Assumptions

• Unit aspect ratio

• No skew

Extrinsic Assumptions• No rotation

• Camera at (0,0,0)

James Hays

K

Page 80: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Remove assumption: equal aspect ratio

X0IKx

10100

00

00

1

0

0

z

y

x

vf

uf

v

u

w y

x

Intrinsic Assumptions• No skew

Extrinsic Assumptions• No rotation

• Camera at (0,0,0)

James Hays

Page 81: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Remove assumption: non-skewed pixels

X0IKx

10100

00

0

1

0

0

z

y

x

vf

usf

v

u

w y

x

Intrinsic Assumptions Extrinsic Assumptions• No rotation

• Camera at (0,0,0)

Note: different books use different notation for parameters James Hays

Page 82: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Oriented and Translated Camera

Ow

iw

kw

jw

t

R

X

x

James Hays

Page 83: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Allow camera translation

XtIKx

1100

010

001

100

0

1

0

0

z

y

x

t

t

t

vf

usf

v

u

w

z

y

x

y

x

Intrinsic Assumptions Extrinsic Assumptions• No rotation

James Hays

Page 84: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

3D Rotation of Points

Rotation around the coordinate axes, counter-clockwise:

100

0cossin

0sincos

)(

cos0sin

010

sin0cos

)(

cossin0

sincos0

001

)(

z

y

x

R

R

R

p

p’

γ

y

z

Slide Credit: Saverese

x

Page 85: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Allow camera rotation

XtRKx

1100

0

1 333231

232221

131211

0

0

z

y

x

trrr

trrr

trrr

vf

usf

v

u

w

z

y

x

y

x

James Hays

Page 86: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Demo – Kyle Simek

• “Dissecting the Camera Matrix”

• Three-part blog series

• http://ksimek.github.io/2012/08/14/decompose/

• http://ksimek.github.io/2012/08/22/extrinsic/

• http://ksimek.github.io/2013/08/13/intrinsic/

• “Perspective toy”

• http://ksimek.github.io/perspective_camera_toy.html

Page 87: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Orthographic Projection

• Special case of perspective projection

– Distance from the COP to the image plane is infinite

– Also called “parallel projection”

– What’s the projection matrix?

Image World

Slide by Steve Seitz

11000

0010

0001

1z

y

x

v

u

w

Page 88: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Field of View (Zoom, focal length)

Page 89: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Beyond Pinholes: Radial Distortion

Image from Martin Habbecke

Corrected Barrel Distortion

Page 90: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Beyond Pinholes: Real apertures

Page 91: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Accidental Cameras

Accidental Pinhole and Pinspeck Cameras Revealing the scene outside the picture.

Antonio Torralba, William T. Freeman

Page 92: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Accidental Cameras

James Hays

Page 93: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Things to remember

• Vanishing points and vanishing lines

• Pinhole camera model and camera projection matrix

• Homogeneous coordinates

Vanishingpoint

Vanishingline

Vanishingpoint

Vertical vanishingpoint

(at infinity)

XtRKx

James Hays

Page 94: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

IS THIS ENOUGH?

Page 95: Review - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures... · G Review: RANSAC Algorithm: N I 14 1. Sample (randomly) the number of points required to fit the model

Erik Johansson – The Architect