Computer Visioncs-courses.mines.edu/csci508/schedule/01/ReviewComputer... · 2017-01-12 · Colorado School of Mines Computer Vision Image Formation •To project 3D points represented

Computer VisionColorado School of Mines

Colorado School of Mines

Computer Vision

Professor William HoffDept of Electrical Engineering &Computer Science

http://inside.mines.edu/~whoff/ 1


Review ‐ Computer Vision


Topics

• Image formation• Estimating a transformation using nonlinear least squares (Matlab)

• Image warping (OpenCV)

3


Image Formation

• To project 3D points represented in the coordinate system attached to the camera, to the 2D image plane:

4

1000

0 yy

xx

cfcf

K

1//

1 ,

1

010000100001

32

31

C

3

2

1

xxxx

yx

ZYX

xxx

K

The leading superscript indicates the coordinate frame that the point is represented in

We use “homogeneous” points; the last coordinate is always a 1 (if not, divide though to make it a 1)

Typical form of intrinsic camera calibration matrix. Usually fx=fy. Usually k12=k21=0. The principal point (cx,cy) should be close to the middle of the image.

This simple 3x4 matrix just strips off the trailing “1” from the point


Extrinsic Camera Matrix

• If 3D points are in world coordinates, we first need to transform them to camera coordinates

• We can write this as an extrinsic camera matrix, that does the rotation and translation, then a projection from 3D to 2D

1

C CC C W WW Worg

W

R tP H P P

0

11 12 13

21 22 23

31 32 33

XC C

ext W Worg Y

Z

r r r tr r r tr r r t

M R t

5


Complete Perspective Projection

• Projection of a 3D point WP in the world to a point in the pixel image (xim,yim)

– where K is the intrinsic camera parameter matrix– and Mext is the 3x4 matrix given by

3231

3

2

1

/ ,/ ,

1

xxyxxxZYX

xxx

imim

W

ext

MK

6

11 12 13

21 22 23

31 32 33

XC C

ext W Worg Y

Z

r r r tr r r tr r r t

M R t


Examples

• Project a point onto an image– You are given K and P (in camera coordinates)

• If you have an image point p, determine the corresponding unit vector direction– Convert to “normalized image coordinates (i.e., focal length = 1, principal point at 0,0):

– Convert to unit vector: /

7


• Example – rotation and translation (Euclidean) transformation

• We are given N corresponding points

• We want to estimate the unknown parameters ( , , )

Estimating a 2D image transform

8

1100cossinsincos

1A

A

y

x

B

B

yx

tt

yx

Niyxyx iA

iA

iB

iB 1,,, )()()()(


Linear Least Squares

• In general– The input data can be vectors– The function can be a linear combination of the input data

• We write A x = b– The parameters to be fit are in the vector x– The input data is in A,b

• Example of a line y = f(x) = mx + b– Parameter vector

– Linear equations

9

bm

x

3

2

1

3

2

1

111

yyy

bm

xxx

• So for a line

NN y

yy

x

xx

2

1

2

1

,

1

11

bA


Solving Linear Least Squares

• Want to minimize

• Expanding we get

• To find the minimum, take derivative wrt x and set to zero, getting

• To solve, can do

• In Matlab can do – x = pinv(A)*b; – or x = A\b;

2bAx E

22 bbAxxAAx TTTTE

bAxAA TT Called the “normal equations”

bAAAx TT 1 TT AAAA 1

“pseudo inverse”

• Note – it is preferable to solve the normal equations using Cholesky decomposition

10


Nonlinear Least Squares

• Our equations are nonlinear in the unknown parameters ( , , )

• That means we can’t write it in the form Ax = b

• We can still solve for the unknowns using least squares, but it requires an iterative algorithm (Newton’s method)– We linearize the function around the current guess– We solve for the answer using linear least squares– Then repeat

11

yAAB

xAAB

tyxytyxx

cossinsincos


Nonlinear Least Squares

• We have a nonlinear function y = f(x)– x is a vector of our unknowns– y is a vector of our observations

• We start with a guess for x, call it x0• We linearize (take the Taylor series expansion) about that point

dy = [∂f/ ∂x]x0 dx

• The matrix of partial derivatives of f with respect to x is called the Jacobian matrix

MNNN

M

M

j

i

xfxfxf

xfxfxfxfxfxf

xf

21

22212

12111

J

MN x

xx

y

yy

2

1

2

1

, xy

12


Nonlinear Iterative Least Squares Algorithm

We have • y0 = observations or measurements• x0 = a guess for x• y = f(x) is a non linear function

1. Initialize x to x02. Compute y = f(x). Residual error is dy = y‐y03. Calculate Jacobian of f, evaluate it at x. We now have

dy = J dx4. Solve for dx using pseudo inverse dx = (JTJ)‐1JT dy5. Set x <= x + dx6. Repeat steps 2‐5 until convergence (no more change

in x)

13


Example – 2D rigid transformation

• Recall

• We have N corresponding points

• Let y = f(x), where

14

12

)()(

)()(

)2()2(

)2()2(

)1()1(

)1()1(

13

12)(

)(

)2(

)2(

)1(

)1(

cossinsincos

cossinsincoscossinsincos

)(,

NxyN

AN

A

xN

AN

A

yAA

xAA

yAA

xAA

xy

x

NxN

B

NB

B

B

B

B

tyxtyx

tyxtyxtyxtyx

tt

yx

yxyx

xfxy

Niyxyx iA

iA

iB

iB 1,,, )()()()(

yAAB

xAAB

tyxytyxx

cossinsincos


Example

• Jacobian?

15

32322212

322212

312111

NxNNN

j

i

xfxfxf

xfxfxfxfxfxf

xf

J

12)()(

)()(

)2()2(

)2()2(

)1()1(

)1()1(

13

cossinsincos

cossinsincoscossinsincos

)(,

NxyN

AN

A

xN

AN

A

yAA

xAA

yAA

xAA

xy

x

tyxtyx

tyxtyxtyxtyx

tt

xfx


Matlab code

16

pA, pB contain the known corresponding points

clear allclose all

pA = [213 398 401 223;29 20 293 297;1 1 1 1];

pB = [207 391 339 164;7 34 302 270;1 1 1 1];

N = size(pA, 2);

theta = 0;tx = 0;ty = 0;x = [theta; tx; ty]; % initial guess

1 of 2


Matlab code

17

while truedisp('Parameters (theta; tx; ty):'), disp(x);

y = f(x, pA); % Call function to compute expected measurements

dy = reshape(pB(1:2,:),[],1) - y; % new residual

J = zeros(2*N,3);

theta = x(1);for i=1:N

J( 2*(i-1)+1, :) = [ -sin(theta)*pA(1,i)-cos(theta)*pA(2,i) 1 0 ];J( 2*(i-1)+2, :) = [ cos(theta)*pA(1,i)-sin(theta)*pA(2,i) 0 1 ];

end

dx = pinv(J)*dy;

% Stop if parameters are no longer changing.if abs( norm(dx)/norm(x) ) < 1e-6

break;end

x = x + dx; % add correctionend

2 of 2


The function f

• This function implements the function f( , , )• It transforms a set of points using the given estimate of the parameters

– x = [theta;tx;ty] is the vector of transformation parameters– pIn = is the set of input points– y = are the output points

18

11121

21

N

N

yyyxxx

pIn

N

N

yx

yxyx

2

2

1

1

y

function y = f(x,pIn)% Do 2D rigid transform, on a set of input points

% Get paramstheta = x(1);tx = x(2);ty = x(3);

H = [ cos(theta) -sin(theta) tx;sin(theta) cos(theta) ty;0 0 1];

pOut = H*pIn;pOut = pOut(1:2, :); % 1st two rows

y = reshape(pOut,[],1);return


Lab 1

• Estimate the parameters of a “scaled similarity” transform, given a set of corresponding points

19


Image Warping in OpenCV

20

From http://docs.opencv.org, under “Core functionality”


OpenCV Code

21

// Warp input image using an affine transform.#include <iostream>#include <windows.h> // For Sleep()#include <opencv2/opencv.hpp>

// Define the elements of the affine transformation matrix.double A[2][3] = { {0.9, 0.1, 10.0}, {0.1, 0.8, 20.0}};

// Show different methods to create a Mat object.#define METHOD 1

int main(int argc, char* argv[]){ printf("Hit ESC key to quit\n");

cv::VideoCapture cap(1); // open the camera if (!cap.isOpened()) { // check if we succeeded printf("error ‐ can't open the camera or video; hit any key to quit\n"); system("PAUSE"); return EXIT_FAILURE; }

// Let's just see what the image size is from this camera. double WIDTH = cap.get(CV_CAP_PROP_FRAME_WIDTH); double HEIGHT = cap.get(CV_CAP_PROP_FRAME_HEIGHT); printf("Image width=%f, height=%f\n", WIDTH, HEIGHT);

1 of 2

Computer VisionColorado School of Mines22

// Create the (2x3) affine transformation matrix M. We'll show some different ways to create it. // See http://docs.opencv.org/master/d6/d6d/tutorial_mat_the_basic_image_container.html for more info.#if METHOD == 1 cv::Mat M = cv::Mat(2, 3, CV_64F, A).clone();#endif#if METHOD == 2 cv::Mat M = cv::Mat::zeros(2, 3, CV_64F); for (int r = 0; r < 2; r++) for (int c = 0; c < 3; c++) M.at<double>(r, c) = A[r][c];#endif#if METHOD == 3 cv::Mat M = (cv::Mat_<double>(2, 3) << A[0][0], A[0][1], A[0][2], A[1][0], A[1][1], A[1][2]);#endif

while (true) { cv::Mat imageInput; cap >> imageInput; if (imageInput.empty()) break;

cv::Mat imageInputGray; // Convert to grayscale (not necessary) cv::cvtColor(imageInput, imageInputGray, cv::COLOR_BGR2GRAY);

cv::Mat imageOutput; cv::warpAffine(imageInputGray, imageOutput, M, cv::Size(WIDTH, HEIGHT));

// Show the image. cv::imshow("My Image", imageOutput);

// Wait for xx ms (0 means wait until a keypress) if (cv::waitKey(1) == 27) break; // hit ESC (ascii code 27) to quit }

return EXIT_SUCCESS;}

2 of 2

Documents

Computer Visioncs-courses.mines.edu/csci508/schedule/01/ReviewComputer... · 2017-01-12 · Colorado School of Mines Computer Vision Image Formation •To project 3D points represented