Image Processing and Computer Vision Chapter 11: Bundle adjustment Structure reconstruction SFM from N-frames Bundle adjustment– structure reconstruction

Image Processing and Computer Vision

Chapter 11: Bundle adjustment Structure reconstruction SFM

from N-frames

Bundle adjustment– structure reconstruction V4c 1

Reconstruction from N-frames

• Factorization (linear, fast, not too accurate)• Bundle adjustment (slower but more

accurate), can use factorization results as the first guess. – Non linear iterative methods are more accurate

than linear method, require first guess (e.g. From factorization).

– Many different implementations, but the concept is the same.


Problem definition

• There are N features in the 3D object .• We take pictures of the object at different

views.• Input :

• Image sequence I1,I2,…I .• Each image has n image feature points

• Output (structure=model, and motion=pose)• 3-D coordinates of all 3-D model points X1,X2,..,XN.• Camera pose for each image taken [R(t),T(t)] t=1,…


Example: Bundle adjustment 3D reconstruction (see also http://www.cse.cuhk.edu.hk/khwong/demo/index.html)

• Grand Canyon Demo• Flask• Robot


http://www.youtube.com/watch?v=2KLFRILlOjc

http://www.youtube.com/watch?v=4h1pN2DIs6g

http://www.youtube.com/watch?v=ONx4cyYYyrIhttp://www.youtube.com/watch?v=xgCnV--wf2k

The iterative SFM alternating bundle adjustment method

• Break down the system into two phases:--SFM1: find pose phase--SFM2: find model phase

• Initialize first guess of model – The first guess is a flat model perpendicular to the image

and is Zinit away (e.g. Zinit = 0.5 meters or any reasonable guess)

• Iterative while ( Err is not small )• {

– SFM1: find pose phase– SFM2: find model phase– Measurement error(Err) or(model and pose stabilized)

• }


SFM1 : find pose phase

Pose estimationdiscussed in the last chapter


SFM2: Model finding by the iterative method• Similar to pose estimation.

– In pose estimation: model is known, pose is unknown.

– Here (Model finding by the iterative method) Assume pose is known, model is unknown.

– The algorithms are similar.

Bundle adjustment– structure reconstruction V4c

7

Problem Formulation

•


8

N iT

iiiN i

,..,Γ,t

T

tiii,t

Tt,..,Γ,t

iiviii

iiii

iiuiii

iiii

i

Tiii

thTiiii

T

ZYXM

θ

vux

],T,T,T,φ,φ[φ θ

bMgTZrYrXr

TZrYrXrfv

aMgTZrYrXr

TZrYrXrfu

P

vuxiZYXM

TTT

TTTθR,T

,..2,1,..2,1

21

32132121

,3333231

2232221

,3333231

1131211

321

321

321321

Find

pose and frames imagegiven theFrom : taskSFM2

known. are tsmeasuremen image and

, pose theAsume

features N 1,2,...,iobject the

and frames, image,1,2, tare There :definition Problem

--------------------------chapter In this---------------

)2(),(

)2(),(

(1a),1(b) rewrite , ofpoint image the

is , N),1,2,(ipoint model theis

ly,respective axes ZY,X, around anglesrotation are ,,

y,repectivel axes ZY,X, along ons translatiare ,,

posefor Assume

:)estimation pose-iv10(chapter slides estimation pose theFrom

Derivation for the model partial derivatives

•


9

23111

112

31

11312

3333231

1131211

12

13

23

333231

232221

131211

312

123

3333231

1131211

321321

1

1

. Find

1

1

1

R

),,,,,,,,(Given

Ri

iRi

Ri

Ri

Ri

Ri

RiR

i

Ri

RiR

i

Ri

Ri

Ri

Ri

RiR

i

Ri

Ri

Ri

iii

iii

i

i

iii

iiiRi

Ri

iii

iii

iiiui

Z

Xr

Z

rf

Z

rf

Z

Xrf

Z

frXr

Zf

Z

f

X

XX

X

Z

Z

Zf

ZX

fXX

X

Zf

X

ZX

f

X

TZrYrXrTZrYrXr

f

X

u

X

u

rrr

rrr

rrr

TZYX

TZYXf

Z

Xf

TZrYrXr

TZrYrXrf

ZYXTTTgu

23212

122

32

12322

3333231

1131211

33132231

21331211

312

123

3333231

1131211

321321

1

1

. Find

1,,

,,1 so

),,,,,,,,(Given

Ri

iRi

Ri

Ri

Ri

Ri

RiR

i

Ri

RiR

i

Ri

Ri

Ri

Ri

RiR

i

Ri

Ri

Ri

iii

iii

i

i

iii

iiiRi

Ri

iii

iii

iiiui

Z

Xr

Z

rf

Z

rf

Z

Xrf

Z

frXr

Zf

Z

f

Y

XX

Y

Z

Z

Zf

ZY

fXX

Y

Zf

Y

ZX

f

Y

TZrYrXrTZrYrXr

f

Y

u

Y

u

rrr

rrr

TZYX

TZYXf

Z

Xf

TZrYrXr

TZrYrXrf

ZYXTTTgu

CMSC5711: Exercise11.1:

•


10

?

Find

1

1

1

R

,1,,

,,1 so

),,,,,,,,(Given

3333231

1131211

12

13

23

333231

232221

131211

33132231

21331211

312

123

3333231

1131211

321321

Z

Z

Xf

Z

TZrYrXr

TZrYrXrf

Z

u

rrr

rrr

rrr

orrrr

rrr

TZYX

TZYXf

Z

Xf

TZrYrXr

TZrYrXrf

ZYXTTTgu

Ri

Ri

iii

iii

i

iii

iiiRi

Ri

iii

iii

iiiui

Exercise11.2: proves all these partial derivatives

• Bundle adjustment– structure reconstruction V4c

11

23323

23222

23121

23313

23212

23111

Ri

iRii

i

Ri

iRii

i

Ri

iRii

i

Ri

iRii

i

Ri

iRii

i

Ri

iRii

i

Z

Yr

Z

rf

Z

v

Z

Yr

Z

rf

Y

v

Z

Yr

Z

rf

X

v

Z

Xr

Z

rf

Z

u

Z

Xr

Z

rf

Y

u

Z

Xr

Z

rf

X

u

Recall

•


12

Continue

•


13

)5(~

~

~

)~

,()~

,()~

,(

)~

,()~

,()~

,(

~

~

~

~

~

formmatrix ain put them (4b), and (4a) combine

-(4b)-)~

()

~,(

)~

()

~,(

)~

()

~,(

)~

,(

-(4a)-)~

()

~,(

)~

()

~,(

)~

()

~,(

)~

,(

~point model D-3 guessed a is

~~~~:point model for the known, are Assume

1332

,,

,,

,,

,..,1

ii

ii

ii

iviviv

iuiuiu

ii

ii

ii

Mii

ii

iiiv

iiiv

iiiv

ttivti

iiiu

iiiu

iiiu

itiuti

titi

iiii

tht

ZZ

YY

XX

Z

Mg

Y

Mg

X

MgZ

Mg

Y

Mg

X

Mg

ZZ

YY

XX

jvv

uue

ZZZ

MgYY

Y

MgXX

X

Mg

Mgv

ZZZ

MgYY

Y

MgXX

X

Mg

Mgu

uu

)Z,Y,X(M

i

continue


14)7(*

~

~

~

,: model for theJacobain theSetup

)6(~

~

~

)~

,()~

,()~

,(

)~

,()~

,()~

,(:

)~

,()~

,()~

,(

)~

,()~

,()~

,(

~

~

~

:

~

~:

~

~

:

relationsmatrix stack the times,21 edpoint view model thehave weIf

)5(~

~

~

)~

,()~

,()~

,(

)~

,()~

,()~

,(

~

~

~

~

~

1332)(

12

1332

)(

1)(

)(

13

32

1

1332

)(

1)(

11

12

1332)()(

32)(

)(

12)(

13

12

MJE

ZZ

YY

XX

M

j

j

J

ZZ

YY

XX

Z

Mg

Y

Mg

X

MgZ

Mg

Y

Mg

X

Mg

Z

Mg

Y

Mg

X

MgZ

Mg

Y

Mg

X

Mg

ZZ

YY

XX

j

j

vv

uu

vv

uu

e

e

E

,,.Γ,ti

ZZ

YY

XX

Z

Mg

Y

Mg

X

MgZ

Mg

Y

Mg

X

Mg

ZZ

YY

XX

jvv

uue

m

ii

ii

ii

tm

tm

m

ii

ii

ii

t

iviviv

iuiuiu

t

iviviv

iuiuiu

ii

ii

ii

tm

tm

tii

ii

tii

ii

t

t

th

ii

ii

ii

t

iviviv

iuiuiu

tii

ii

iim

tii

iit

SFM2: Iteration for finding the model point i: In this algorithm each point i (i=1,2,..N) is found independently , so the following algorithm will be run N times.


15

result theis ~

loop above theof end After the

}

~~ is guessNext

enough small is ifBreak

inverse pseudo use matrx, square anot is if // and E find

{

1max_0Iterate

*

findcan we,~

guessfirst on the Base

* so , *

1

1)(

1

0)(

0

0

1)(

k

kkk

k

km

k

-kk

km

k

k

T

-m

M

MMM

M

EJM

JJ

) k,kK,k (k

MJE

M

ZYXM

ΔMekJMJE

SFM2: This algorithm is to find the model M

The iterative SFM alternating bundle adjustment method

• Break down the system into two phases:--SFM1: find pose phase--SFM2: find model phase (method (A) or (B))

• Initialize first guess of model – The first guess is a flat model perpendicular to the image

and is Zinit away (e.g. Zinit = 0.5 meters or any reasonable guess)

• Iterative while ( Err is not small )• {

– SFM1: find pose phase– SFM2: find model phase – Measurement error(Err) small or model and pose stabilized

• }


Overall processing revisit

• Given: measurements – Images of N frames

• Point feature tracked by KLT Kanade–Lucas–Tomasi_feature_tracker or SURF (Speeded Up Robust Features) methods

• Examples, demo• http://www.youtube.com/watch?v=RXpX9TJlpd0

• To find pose (Rotation R, translation T ) of every frame, and the model structure X


17

Putting it altogether

Repeat the explanations SFM1 (find pose phase) and

SFM2 (find model phase) with implementation details.


18

Putting it altogether• Use KLT (or SIFT, Harris then correlation) to obtain features in [u,v]T

• There are t=1,2,…, image frames, • So there are t=1={R,T} t=1 , t=2={R,T} t=2 , …., t=={R,T} t= poses.

• There is only one model Mi=[X,Y,Z]I,with i=1,..,N features• Initialize first guess of model

– The first guess is a flat model perpendicular to the image and is Zinit away (e.g. Zinit = 0.5 meters or any reasonable guess)

• Iterative while ( Err is not small ){– /////////////// SFM1: Pose finding ////////////////////////////////////– //(for every time frame t, use all N features, run SFM1 once); so SFM1 runs times here– For (t=1; t<; t++) – { Inputs: You have f(focal length), Mi=[X,Y,Z]i

– For each frame t, you have i=1,,,N, image feature points and measurements [u,v] Ti,t

– Output: pose t

– }

– After the above is run– t=1={R,T} t=1 , t=2={R,T} t=2 , …., t=={R,T} t= poses are found– ////////////////////// SFM2: model finding //////////////////////– (For i=1,i<N;i++) (for every feature, use all frames, run SFM2 once; so SFM2 runs N times here)

• {SFM2: find model phase}– Measurement error(Err) small or model and pose stabilized}


19

Recall: SFM1:Find Pose phase//(for every time frame t, use all N features, run SFM1 once); so SFM1 runs times here For (t=1; t<; t++){ Inputs: You have f(focal length), Mi=[X,Y,Z]i

For each frame t, you have i=1,,,N, image feature points and measurements [u,v]T

i,t}

Output: pose t

}

After the above is runt=1={R,T} t=1 , t=2={R,T} t=2 , …., t=={R,T} t= poses are foundBundle adjustment– structure

reconstruction V4c20

Recall in pose estimation SFM1: a slide in ch.iv10: pose estimation


21

)7(*

~

~

~

~

~

~

,:Jacobain Put the

)6(

~

~

~

~

~

~

0

0

:

0

0

~

~

~

~

~

~

:

~

~:

~

~

:

relationsmatrix stack the N,,1,i points, encecorrespond N have we t,frameeach For

166212

33

22

11

33

22

11

62

1

1633

22

11

33

22

11

62

222

222

1

222

222

12

1633

22

11

33

22

11

62

1

12

11

11

12

1

12

16

NN

NNi

i

NNi

Ri

Ri

Ri

Ri

i

Ri

iR

i

Ri

Rii

Rii

Ri

Ri

Ri

Ri

i

Ri

Rii

Rii

Ri

Rii

i

Ri

Ri

Ri

Ri

i

Ri

iR

i

Ri

Rii

Rii

Ri

Ri

Ri

Ri

i

Ri

Rii

Rii

Ri

Rii

N

NNi

i

NNiNi

NiNi

ii

ii

NNi

i

N

JE

TT

TT

TTj

j

J

TT

TT

TT

Z

Yf

Z

f

Z

Xf

Z

XYf

Z

ZZYYf

Z

Xf

Z

f

Z

Yf

Z

ZZXXf

Z

XYf

Z

Yf

Z

f

Z

Xf

Z

XYf

Z

ZZYYf

Z

Xf

Z

f

Z

Yf

Z

ZZXXf

Z

XYf

E

TT

TT

TTj

j

vv

uu

vv

uu

e

e

E

•

At time t, there are N features

• The formulas apply to one frame at time t. There are i=1,2,…N features.

• each time t, t is found.• SFM1 will times, each

time is independent.

Exercise• Exercise 11.3 : If the model is a

checker board plane, each square is 1cm2.. It is perpendicular to the camera principal axis and at Z=0.5 meters away from the camera center. Pixel width is 5um.

• Find the 3D positions of X1,X2,X3 and X4 in pixels

•


22

x3=[0,0]

X1

X41cm

X2

1cm

Y

X

How to get the first guess of the model?

• We have an image with [u,v]i=1,2,..,N

• Camera focal length is f• In theory

– ui=f*Xi/Zi

– vi=f*Yi/Zi

• First guess of the model is all points on a plane • X’i=ui*Zguess/f, Y’i=vi*Zguess/f,

• Z’i=Zguess=0.5 meters (for example)

• So the guessed i-th 3D point is at [X’I, Y’I, Z’i]Bundle adjustment– structure

reconstruction V4c

23

Image

Camera center

Zguessf

Exercise 11.4 Revision for pose estimation SFM1: : Point out which are know variables and unknown variables in this page.


24

•



• each time t, t is found.• SFM1 will times, each

time is independent.

)7(*

~

~

~

~

~

~

,:Jacobain Put the

)6(

~

~

~

~

~

~

0

0

:

0

0

~

~

~

~

~

~

:

~

~:

~

~

:


166212

33

22

11

33

22

11

62

1

1633

22

11

33

22

11

62

222

222

1

222

222

12

1633

22

11

33

22

11

62

1

12

11

11

12

1

12

16

NN

NNi

i

NNi

Ri

Ri

Ri

Ri

i

Ri

iR

i

Ri

Rii

Rii

Ri

Ri

Ri

Ri

i

Ri

Rii

Rii

Ri

Rii

i

Ri

Ri

Ri

Ri

i

Ri

iR

i

Ri

Rii

Rii

Ri

Ri

Ri

Ri

i

Ri

Rii

Rii

Ri

Rii

N

NNi

i

NNiNi

NiNi

ii

ii

NNi

i

N

JE

TT

TT

TTj

j

J

TT

TT

TT

Z

Yf

Z

f

Z

Xf

Z

XYf

Z

ZZYYf

Z

Xf

Z

f

Z

Yf

Z

ZZXXf

Z

XYf

Z

Yf

Z

f

Z

Xf

Z

XYf

Z

ZZYYf

Z

Xf

Z

f

Z

Yf

Z

ZZXXf

Z

XYf

E

TT

TT

TTj

j

vv

uu

vv

uu

e

e

E

Exercise 11.5 Revision for pose estimation SFM1: : Explain why E is known here in the following formula.


25

1633

22

11

33

22

11

62

1

12

11

11

12

1

12

~

~

~

~

~

~

:

~

~:

~

~

:

relationsmatrix stack the N,,1,i points, encecorrespond N have weIf

TT

TT

TTj

j

vv

uu

vv

uu

e

e

E

NNi

i

NNiNi

NiNi

ii

ii

NNi

i

N

•

Revision for pose estimation SFM1: Explain why J is known here.Answer: because guessed M, and guessed pose are known

• 626

6

3

3

6

1

6

1

2

1

2

1

1

1

1

1

),~

(

),~

(

:::::

::::::

:::),

~(

),~

(

::

::::::

),~

(

),~

(

:::),

~(

),~

(

),~

(

),~

(

N

Niv

Niu

Iiv

Iiu

iv

iu

v

u

v

u

Mg

Mg

Mg

Mg

Mg

Mg

Mg

Mg

Mg

Mg

J


Exercise 11.6 : Revision for pose estimation SFM1: Identify known variables and unknown variables when k=0 and k=5 in this iterative pose estimation algorithm

• Bundle adjustment– structure


}

~~ is guessNext


)~

(

inverse pseudo use no if // )~

( and find

{

1max_0Iterate

*)~

(

findcan we,~


* so

, *

1

10

110

01

00

0

321321

1

kkk

k

kkk

-kk

kkk

k

T

-

EJ

JJE

) k,kK,k (k

EJ

TTT

ΔθEJ

JE

6233

22

11

33

22

11

321321

321321

~

~

~

~

~

~pose guessed

~

~,

~,

~,

~,

~,

~~find want to that wepose theis

,,,,,

:

NTT

TT

TT

TTT

TTT

Note


• SFM1 will times, each time is independent,

SFM1: This algorithm is to find the pose

SFM2: Find model phaseSimilar to SFM1 but pose is known, find model here.

(for every feature, use all frames, run SFM2 once; so SFM2 runs N times here) For i=1,i<N;i++

{ SFM2: find model phase}


28

Exercise11.7: Revision of SFM2: Identify which are known which are known here. Explain why J(m) is known at this stage.

•


29)7(*

~

~

~

,: model for theJacobian Put the

)6(~

~

~

)~

,()~

,()~

,(

)~

,()~

,()~

,(:

)~

,()~

,()~

,(

)~

,()~

,()~

,(

~

~

~

:

~

~:

~

~

:

21for relationsmatrix stack the ,21 i.e. frames, have weIf

)5(~

~

~

)~

,()~

,()~

,(

)~

,()~

,()~

,(

~

~

~

~

~

1332)(

12

1332

)(

1)(

32)(

13

32

1

1332

)(

1)(

12

1

12

1

)(

,,

,,

MJE

ZZ

YY

XX

M

j

j

J

ZZ

YY

XX

Z

Mg

Y

Mg

X

MgZ

Mg

Y

Mg

X

Mg

Z

Mg

Y

Mg

X

MgZ

Mg

Y

Mg

X

Mg

ZZ

YY

XX

j

j

vv

uu

vv

uu

e

e

E

,..Γ,t,Γ,tΓ

ZZ

YY

XX

Z

Mg

Y

Mg

X

MgZ

Mg

Y

Mg

X

Mg

ZZ

YY

XX

jvv

uue

m

ii

ii

ii

tm

tm

m

ii

ii

ii

t

iviviv

iuiuiu

t

iviviv

iuiuiu

ii

ii

ii

im

im

tii

ii

tii

ii

t

t

ii

ii

ii

t

iviviv

iuiuiu

tii

ii

iim

titi

titi

• The formulas apply to one feature (i) for all time frames t=1,2,….

• SFM2 will N times , each time is independent.

• See next slide for the graphical illustration,

Measured

Result from the guessed model and given pose

JacobianCurrent Guessed model

New Guessed model

In SFM2, we handle a feature i, at one time.

• The formulas in the pervious slide apply to one feature (i) for all time farmes t=1,2,…


30

…

Camera motionOt=camera center at time t

Imaget=1

Imaget=2 Image

t=3Imaget=

…

v1

v2v3

vm

X

[u,v] i,t=2

[u,v]i,t=1

[u,v] i,t=3 [u,v] i,t= Ot=1

t=1={R,T} t=1

t=2={R,T} t=2

t=3={R,T} t=3

t=={R,T} t=

Ot=2

Ot=3

Ot=

Exercise11.8 : SFM2: Algo. to find the i-th model point (repeat this N times to get all N points)

Identify which are known and unknown when K=0, K=5


31

result theis ~


}

~~~ is guessNext


inverse pseudo use square,not is if // and E find

{

1max_0Iterate

*

findcan we,~


* so , *

1

1)(

1

0)(

0

0

1)(

k

kkk

k

km

k

-kk

km

k

k

T

-m

M

MMM

M

EJM

JJ

) k,kK,k (k

MJE

M

ZYXM

ΔMekJMJE

SFM2 previous by the found model

~ iterativeat plane a

~model Guessed]

~,

~,

~[

~find want to wemodel theis

unknwon],,[

~

~

~:

_

0_

kati

thkati

Tiiii

i

Tiiii

ii

ii

ii

M

kM

ZYXM

M

ZYXM

ZZ

YY

XX

M

Note





From [2] Result for rotation angles


From [2] Result for translations


From [2] Result: compare full/classical(+) and 2-pass algorithm (o)


From [2] Results for real images


Conclusions

• Bundle adjustment can be used for structure and motion SAM (model structure reconstruction and pose estimation).

• Bundle adjustment is an accurate method for Structure from motion SFM.

• It can made more efficient by using a two pass (pose finding step, model fining step) algorithm


Appendices


Demo Newton's method• function new_x=demo_newton1(x)• %This is to solve x^3-2x=3 • %assume x is the guessed x• % 3= f(true_x)=f(x)+f'(x)(new_x-x)+ small_terms_ignored % Taylor

series • % 3-f(x)/f'(x)=new_x-x, or• % new_x=x+((3-f(x))/f'(x))= new_x, until new_x not changed• % so that• % new_x=x+((3-f(x))/f'(x))• % new_x=x+((3-(x^3-2*x))/(3*x^2-2));• while (1)• new_x=x+((3-(x^3-2*x))/(3*x^2-2));• err=abs(x - new_x);• st=sprintf('new_x=%2.3f,err=%2.3f, err is still too big\

n',new_x,err);• disp(st);• if (err < 0.01) • break;• end• %'err still big, hit key to continue'• pause• x=new_x;• end• 'err is small new_x is the solution'• new_x

• >> demo_newton1(1)• new_x=5.000,err=4.000, err is still too big

• new_x=3.466,err=1.534, err is still too big





• new_x=1.893,err=0.000, • ans =• err is small new_x is the solution• new_x =• 1.8933• ans =• 1.8933


38

Rotation matrix

•


39

sosmall, are ,,when ,

1

1

1

R

small are ,, when Also,

1)det(,,

))cos(cos())cos(sin( +))sin()sin(cos())sin(sin( + ))cos()sin(cos(-

))cos(sin(-))cos(cos( +))sin()sin(sin(-))sin(cos( + ))cos()sin(sin(

)sin())sin(cos(-))cos(cos(

321

12

13

23

333231

232221

131211

321

31

213132131321

213132131321

23232

333231

232221

131211

rrr

rrr

rrr

RIRRRR

And

rrr

rrr

rrr

TT

Jacobian for model :JacobM% Jacobian for model :JacobM %%%%%%%%•N = size(model,2); %model=3,N•if N~=1• error('JacobM: model size must be 4*1');•end•T = size(rt,3); %rt=3,4,T•J=zeros(2*T,3);

•for t=1:T %index T vertical blocks• V = rt(:,:,t)*model;• X = V(1,:);• Y = V(2,:);• Z = V(3,:);• Z2 = Z.*Z;• XZ2 = X./Z2;• YZ2 = Y./Z2;• a11 = rt(1,1,t)./Z - rt(3,1,t).*XZ2;• a12 = rt(1,2,t)./Z - rt(3,2,t).*XZ2;• a13 = rt(1,3,t)./Z - rt(3,3,t).*XZ2;• a21 = rt(2,1,t)./Z - rt(3,1,t).*YZ2;• a22 = rt(2,2,t)./Z - rt(3,2,t).*YZ2;• a23 = rt(2,3,t)./Z - rt(3,3,t).*YZ2;• a1 = [a11' a12' a13'];• a2 = [a21' a22' a23'];• • J(t,:) = a1;• J(T+t,:) = a2;•end•J = flen.*J; Bundle adjustment– structure reconstruction V4c 40

Angles and R pose conversion

3 2

3 1

2 1

_ _ _ ,

1

1

1

When angles are small

R


11 12 13

21 22 23

31 32 33

cos( )*cos( ) cos( )*sin( )*sin( ) sin( )*cos( ) cos( )*sin( ) sin( )*sin( )

sin( )*cos( ) sin *sin *sin cos sin( )*sin( )*cos( ) cos( )*sin( )

sin( ) cos sin cos(

r r r

R r r r

r r r

) *cos( )

jacobian for chang,wong ieee_mm 2 pass lowe

• '==========test jacobian for chang,wong ieee_mm 2 pass lowe=================='• clear• % a1=yaw, a2=pitch, a3=roll,• % t1=translation in x, t2=translation in y, t3=translation in z, • syms R dR M TT XYZ ZZ x y z f u v a1 a2 a3 t1 t2 t3 aa1 aa2 aa3 tt1 tt2 tt3

• R=[1 -aa3 aa2• aa3 1 -aa1• -aa2 aa1 1];• dR=[1 -a3 a2• a3 1 -a1• -a2 a1 1];

• M=[x;y;z];• TT=[tt1;tt2;tt3];• dt=[t1;t2;t3]• % XX=(dR.*R)*M+TT; %not correct, becuase R is a matrix multiplication transform• XYZ=dR*R*M+TT+dt; %correct, becuase R is a matrix multiplication transform• % XX=(dR+R)*M+TT; %not correct becuase R is not an addition transform• u=f*XYZ(1)/XYZ(3);• v=f*XYZ(2)/XYZ(3);• %diff (u,a3)• %diff (v,a3)• ja=jacobian([u ;v],[a1 a2 a3])• jt=jacobian([u ;v],[t1 t2 t3])Bundle adjustment– structure reconstruction V4c 42

Delaunay algorithm for generation of VRML files

• VRML specifications– Viewers

• Cortona3d, Cosmoplayer, Vivaty• http://cic.nist.gov/vrml/vbdetect.html

• Delaunay algorithm


http://www.cortona3d.com/Products/Cortona-3D-Viewer.aspx

http://cic.nist.gov/vrml/cosmoplayer.html

http://vivaty-inc1.software.informer.com/

Alternative method fro finding the model

• To find model by triangulation (not iterative method )

• It is faster but may be not very accurate.


44

Alternative method for SFM2 : find model phase

There are two methods:(SFM2: method A) Direct triangulation(SFM2: method B) Iterative method (in the main body of this power point)Either (A) or (B) can be used


SFM2(method A): direct triangulation model finding procedure

• Assume you have m views, • Using the first view and each of the other views we can (m-1)pairs of images. • Each pair gives one version of X (using the triangulation method in the chapter

on stereo (chapter iv08 http://www.cse.cuhk.edu.hk/%7Ekhwong/www2/cmsc5711/iv08_stereo.ppt)

• So you have m-1 models X1, X2,… Xm-1 (all referring to the first camera coordinate system as reference)

• The solution X=Xmean is the mean of all these (X1, X2,… Xm-1 ) • So a temporally model X is found at this stage.

– Also measure the error:– Measurement error (Err)– Err=||(current model - previous model)||2


SFM2 (method B) :The iterative steps

• Initialize first guess of model and pose• The first guess is a flat model perpendicular to the image and is

Zinit away (e.g. Zinit = 0.5 meters or any reasonable guess)• Iterative while ( Err is small)• {

– SFM1 find pose– SFM2 find model– Measurement error (Err)– Err= ||(current model - previous model)||2

– //If the model is stabilized, the solution is final.• }


Pose estimation result• Recall of SFM1 (pose estimation)

– Input : • Image sequence I1,I2,…Im.• There are N features in the 3D object .• Each image has n image feature points

– Output: pose [R(t),T(t)] t=1,…m

– At time t, each image feature will give you a vector vt from the camera center O(t) to the 3D point in X passing the image point xi,t

TTTTTRθ , findingfor Posefound 321321


Xi in 3D

Xi,t=[ui,vi]tT

Camera center O(t)

vt

Image

After pose is found in SFM1• We can use triangulation to find the model• Example fro an 3D feature X• After pose estimation SFM1, v1,v2,vm can be found• 0t=camera centers at time t• R(t),T(t)=pose at time t

Bundle adjustment– structure reconstruction V4c49

…

Camera motion

Imaget=1

Imaget=2 Image

t=3Imaget=m

…

v1

v2v3

vm

X

[u,v]2

[u,v]1

[u,v]3

[u,v]m

O1

O2

O3

Om

R2,T2R3,T3

Rm,Tm

R1,T1

SFM2(method A): From vectors find the closes point

• So you have v1,v2,..vm vectors in 3D• You want to find a point closes to this point• So hew to find the closest point between 2 vectors? Of

the first sand second views– Recall: we learned this in stereo vision– We know

• P1,P2 (projection matrices) of two cameras, (yes we know it here because we have guess solution R,T)

• We know the 2D correspondences points (yes we know it here) • We can find the model point X in 3D .


SFM2(method A): Just concentrate on the first two views

• Find X from 2 views– From RT found (SFM1 pose

finding phase), we have P1,P2.

– We also have 2D point correspondences: [u,v]1,[u,v]2

– We can find X, see the next two slides


Imaget=1

Imaget=2

v1

v2

X

[u,v]2

[u,v]1

O1

O2

R2,T2

R1,T1

Projection matrixP1

Projection matrixP2

SFM2(method A): Recall: Triangulation to find X

•

Bundle adjustment– structure reconstruction V4c52

X is the point at a minimum distance between two vectors{O1,(x1,y1)} and {O2,(x2,y2)}

O2

Ol

Left Frame plane1 1

e1e2

Left epipolar line

F

(x2,y2)

(x1,y1)

Epipole e1Epipole e2Focal

length=f1

Focal length=f2

SFM2(method A): Recall:3D reconstruction: from P1 and P2 find 3D points X by

triangulation (p.312[1A],p297[1B])

•

1][appendix SVDby solve,0A

cameras 2for and use , ofrow

)(0)()(

)(0)()(

)(0)()(

0

cameras 2for ,

22

322

12

322

21

311

11

311

21

12

23

13

1111

212211

TT

TT

TT

TT

thiT

TT

TT

TT

ppy

ppx

ppy

ppx

),P(P(ii)(i)Pip

iiipypx

iippy

ippx

ions_manipulatafter_some

Pxxx

),P(PPxPx


34

33

32

31

3

24

23

22

21

2

14

13

12

11

1

141312111

3

2

1

44333231

24232221

14131211

)43(

,,

, i.e.

P of row where

p

p

p

p

p

p

p

p

p

p

p

p

p

p

p

ppppP

iP

P

P

P

pppp

pppp

pppp

P

T

thiT

T

T

T

x

References

1. D.G. Lowe, “Fitting Parameterized Three-Dimensional Models to Images”, IEEE Pattern Analysis and Machine Intelligence, Volume: 13 Issue: 5 , May 1991 Page(s): 441 -450

2. Michael Ming Yuen Chang and Kin Hong Wong, "Model reconstruction and pose acquisition using extended Lowe's method", IEEE Transactions on Multimedia, Volume: 7, Issue: 2, April 2005.Bundle adjustment– structure


http://www.cse.cuhk.edu.hk/~khwong/j2004_IEEE_chang_MM_xlowe_draft.pdf

http://www.cse.cuhk.edu.hk/~khwong/j2004_IEEE_chang_MM_xlowe_draft.pdf

Answers

•


55

Answer11.1: Exercise11.1:

•


56

233133333231

1131211

33132231

21331211

312

123

3333231

1131211

321321

Find

1,,

,,1 so

),,,,,,,,(Given

Ri

iRi

Ri

Ri

iii

iii

i

iii

iiiRi

Ri

iii

iii

iiiui

Z

Xr

Z

rf

Z

Z

Xf

Z

TZrYrXr

TZrYrXrf

Z

u

rrr

rrr

TZYX

TZYXf

Z

Xf

TZrYrXr

TZrYrXrf

ZYXTTTgu

Answer11.2: Exercise11.2: proves all these partial derivatives: Left for students’ exercises

• Bundle adjustment– structure


23323

23222

23121

23313

23212

23111

Ri

iRii

i

Ri

iRii

i

Ri

iRii

i

Ri

iRii

i

Ri

iRii

i

Ri

iRii

i

Z

Xr

Z

rf

Z

v

Z

Xr

Z

rf

Y

v

Z

Xr

Z

rf

X

v

Z

Xr

Z

rf

Z

u

Z

Xr

Z

rf

Y

u

Z

Xr

Z

rf

X

u

Exercise• Answer 11.3: Exercise 11.3 : If the model is

a checker plane, each square is 1cm2.. It is perpendicular to the camera principal axis and at Z=0.5 meters away from the camera center. Pixel width is 5um.

• Find the 3D positions of X1,X2,X3 and X4 in pixels

• Answer:• All Z are the same

Z=0.5meters/5um=100,000• X1(-2cm, -2cm,0.5 meters)= [-4000,-

4000,100,000]• X2(-2cm, -1cm,0.5 meters)=[-4000,-

2000,100,000]• X3(0,0,0.5 m)=[0,0,100,000]• X4(2cm,2cm,0.5m)=[4000,4000,100,000]

•


58

x3=[0,0]

X1

X41cm

X2

1cm

Y

X

Answer11.4: Revision for pose estimation SFM1: Exercise 11.4 : Point out which are know variables and unknown variables in this page.


59

•


found. be toand knowns are ,,,

known. is J So

],,[~~~

,~

,~

,~

on depends J

knwon, is ],,[ Model so

slide) previous (see plane a is model initial theAssume

knwon are they so,]0,0,0,0,0,0[~~~

,~

,~

,~

Gussesed

:Answer

321321

321321

321321

,T,TT

ZYXandT,T,T

ZYX

T,T,T

iii

iii

)7(*

~

~

~

~

~

~

,:Jacobain Put the

)6(

~

~

~

~

~

~

0

0

:

0

0

~

~

~

~

~

~

:

~

~:

~

~

:


166212

33

22

11

33

22

11

62

1

1633

22

11

33

22

11

62

222

222

1

222

222

12

1633

22

11

33

22

11

62

1

12

11

11

12

1

12

16

NN

NNi

i

NNi

Ri

Ri

Ri

Ri

i

Ri

iR

i

Ri

Rii

Rii

Ri

Ri

Ri

Ri

i

Ri

Rii

Rii

Ri

Rii

i

Ri

Ri

Ri

Ri

i

Ri

iR

i

Ri

Rii

Rii

Ri

Ri

Ri

Ri

i

Ri

Rii

Rii

Ri

Rii

N

NNi

i

NNiNi

NiNi

ii

ii

NNi

i

N

JE

TT

TT

TTj

j

J

TT

TT

TT

Z

Yf

Z

f

Z

Xf

Z

XYf

Z

ZZYYf

Z

Xf

Z

f

Z

Yf

Z

ZZXXf

Z

XYf

Z

Yf

Z

f

Z

Xf

Z

XYf

Z

ZZYYf

Z

Xf

Z

f

Z

Yf

Z

ZZXXf

Z

XYf

E

TT

TT

TTj

j

vv

uu

vv

uu

e

e

E

Answer 11.5 Revision for pose estimation SFM1: Exercise 11.5 : Explain why E is known here in the following formula.


60

1633

22

11

33

22

11

62

1

12

11

11

12

1

12

~

~

~

~

~

~

:

~

~:

~

~

:

relationsmatrix stack the N,,1,i points, encecorrespond N have weIf

TT

TT

TTj

j

vv

uu

vv

uu

e

e

E

NNi

i

NNiNi

NiNi

ii

ii

NNi

i

N

• )2(

)2(

~

~;

~;

~for formulas following theusing

]~

,~

,~

,~

,~

,~

[~

pose and~~~~

Model gussedby found~

~

allfor tracker feature KLT by the measured,

~

~:

~

~

In :Answer

312

213

3333231

2232221

312

123

3333231

1131211

321321

12

11

11

bTZYX

TZYXf

TZrYrXr

TZrYrXrfv

aTZYX

TZYXf

TZrYrXr

TZrYrXrfu

v

u

v

uMM

TTT]Z,Y,X[Mv

u

iv

u

vv

uu

vv

uu

iii

ii

iii

iiii

iii

iii

iii

iiii

i

i

i

iii

iiiii

i

i

i

NNiNi

NiNi

ii

ii

found. be toand knowns are ,,,

known. is J So

],,[~~~

,~

,~

,~

on depends J

knwon, is ],,[ Model so

slide) previous (see plane a is model initial theAssume

knwon are they so,]0,0,0,0,0,0[~~~

,~

,~

,~

Gussesed

:Answer

321321

321321

321321

,T,TT

ZYXandT,T,T

ZYX

T,T,T

iii

iii

Answer11.6 : Revision for pose estimation SFM1: Identify known variables and unknown variables when k=0 and k=5 in this iterative pose estimation algorithmAns: K=0, we initialize (GUESS) k=0 then E,J, can be found. Then k=0 can be calculated, then the guessed k=1 = k=0+k=0 is found, which will be used in the iteration k=1.When K=5, guessed k=5 is found during k=4 , then E,J, k=5, are calculated, i.e. k=5 =k=4 + k=4 , when k=4. Then k=5 can be found and it will be used to find guessed k=6 =k=5 + k=5

•


61

}

~~ is guessNext


)~

(

inverse pseudo use no if // )~

( and find

{

1max_0Iterate

*)~

(

findcan we,~


* so

, *

1

10

110

01

00

0

321321

1

kkk

k

kkk

-kk

kkk

k

T

-

EJ

JJE

) k,kK,k (k

EJ

TTT

ΔθEJ

JE

6233

22

11

33

22

11

321321

321321

~

~

~

~

~

~pose guessed

~

~,

~,

~,

~,

~,

~~find want to that wepose theis

,,,,,

:

NTT

TT

TT

TTT

TTT

Note

SFM1:This algorithm is to find the pose

Exercise11.7: Answer11.7: At K=0:[X,Y,Z]i are unknown. Others are known, because pose is found by SFM1, Model Mi is guessed initially at k=0 (a plane) AT K=5: [X,Y,Z]i is unknown.Others are known,because pose is found by SFM1, Model(Mi(xi,y,zi)) is (a better guess) found by the previous SFM2 phaseJ(m) depends on the guessed model M and current pose (current pose is found by SFM1)

•


62)7(*

~

~

~

,: model for theJacobian Put the

)6(~

~

~

)~

,()~

,()~

,(

)~

,()~

,()~

,(:

)~

,()~

,()~

,(

)~

,()~

,()~

,(

~

~

~

:

~

~:

~

~

:

21for relationsmatrix stack the ,21 i.e. frames, have weIf

)5(~

~

~

)~

,()~

,()~

,(

)~

,()~

,()~

,(

~

~

~

~

~

1332)(

12

1332

)(

1)(

32)(

13

32

1

1332

)(

1)(

12

1

12

1

)(

,,

,,

MJE

ZZ

YY

XX

M

j

j

J

ZZ

YY

XX

Z

Mg

Y

Mg

X

MgZ

Mg

Y

Mg

X

Mg

Z

Mg

Y

Mg

X

MgZ

Mg

Y

Mg

X

Mg

ZZ

YY

XX

j

j

vv

uu

vv

uu

e

e

E

,..Γ,t,Γ,tΓ

ZZ

YY

XX

Z

Mg

Y

Mg

X

MgZ

Mg

Y

Mg

X

Mg

ZZ

YY

XX

jvv

uue

m

ii

ii

ii

tm

tm

m

ii

ii

ii

t

iviviv

iuiuiu

t

iviviv

iuiuiu

ii

ii

ii

im

im

tii

ii

tii

ii

t

t

ii

ii

ii

t

iviviv

iuiuiu

tii

ii

iim

titi

titi




Measured

Result from the guessed model and given pose

JacobianCurrent Guessed model

New Guessed model

SFM2: Algorithm to find the i-th model point (repeat this N time to get all points)Exercise11.8 : identify which are known and unknown when K=0, K=5Answer11.8: k=0, Model [X,Y,Z]i is a point in a plane, pose (by SFM1) is known and the algorithm can find a better model.Answer: k=5, Model [X,Y,Z]iis the model found in the previous SFM2 and pose is found by SFM1, the algorithm can find a better Model.

• Bundle adjustment– structure reconstruction V4c 63

result theis ~


}

~~~ is guessNext


inverse pseudo use matrx, square anot is if // and E find

{

1max_0Iterate

*

findcan we,~


* so , *

1

1)(

1

0)(

0

0

1)(

k

kkk

k

km

k

-kk

km

k

k

T

-m

M

MMM

M

EJM

JJ

) k,kK,k (k

MJE

M

ZYXM

ΔMekJMJE

SFM2 previous by the found model

~ iterativeat plane a

~model Guessed]

~,

~,

~[

~find want to wemodel theis

unknwon],,[

~

~

~:

_

0_

kati

thkati

Tiiii

i

Tiiii

ii

ii

ii

M

kM

ZYXM

M

ZYXM

ZZ

YY

XX

M

Note


Documents

Image Processing and Computer Vision Chapter 11: Bundle adjustment Structure reconstruction SFM from N-frames Bundle adjustment– structure reconstruction