Upload
sriram-sampathkumar
View
15
Download
0
Embed Size (px)
DESCRIPTION
Panoramas
Citation preview
5/20/2018 Kari Panoramas 02mar10 Opt
1/113
Panoramas
Kari P ulli
Research Fellow
No kia Research Center Palo Alto
1
5/20/2018 Kari Panoramas 02mar10 Opt
2/113
Are you getting the whole picture?
Compact Camera FOV = 50 x 35
Slide from Brown & Lowe2
5/20/2018 Kari Panoramas 02mar10 Opt
3/113
Are you getting the whole picture?
Compact Camera FOV = 50 x 35
Human FOV = 200 x 135
Slide from Brown & Lowe3
5/20/2018 Kari Panoramas 02mar10 Opt
4/113
Are you getting the whole picture?
Compact Camera FOV = 50 x 35
Human FOV = 200 x 135
Panoramic Mosaic = 360 x 180
Slide from Brown & Lowe4
5/20/2018 Kari Panoramas 02mar10 Opt
5/113
PanoramaA wide-angle representation of the scene
Panorama ofAlong the River During Qingming Festival, 18th century remake of a 12th century original by
Chinese artist Zhang Zeduan.
Image from Wikipedia
5
5/20/2018 Kari Panoramas 02mar10 Opt
6/113
Panorama: Cinema for the early 19thcentury
Burfords Panorama, Leicester Square, London, 1801
Painting by Robert Mitchell6
5/20/2018 Kari Panoramas 02mar10 Opt
7/113
Creating panoramas with wide-angle optics
http://www.0-360.com AF DX Fisheye-NIKKOR 10.5mm f/2.8G ED
7
5/20/2018 Kari Panoramas 02mar10 Opt
8/113
Idea
rotate camera or lens so that a vertical slit is
exposed
Swing lens rotate the lens and a vertical slit (or the sensor)
typically can get 110-140 degree panoramas
Widelux, Seitz,
Full rotation
whole camera rotates
can get 360 degree panoramas
Panoscan, Roundshot,
8
Rotation cameras
5/20/2018 Kari Panoramas 02mar10 Opt
9/113
Swing-lens panoramic images
San Francisco in ruins, 1906
101 Ranch, Oklahoma, circa 19209
5/20/2018 Kari Panoramas 02mar10 Opt
10/113
Lee Frost, Val DOrcia, Tuscany, Italy
Flatback panoramic camera
10
5/20/2018 Kari Panoramas 02mar10 Opt
11/113
Disposable panoramic camerawide-angle lens, limited vertical FOV
11
5/20/2018 Kari Panoramas 02mar10 Opt
12/113
12
5/20/2018 Kari Panoramas 02mar10 Opt
13/113
13
5/20/2018 Kari Panoramas 02mar10 Opt
14/113
14
5/20/2018 Kari Panoramas 02mar10 Opt
15/113
15
5/20/2018 Kari Panoramas 02mar10 Opt
16/113
Stitching images together to make a mosaic
16
5/20/2018 Kari Panoramas 02mar10 Opt
17/113
Stitching images together to make a mosaic
Given a set of images that should stitch together by rotating the camera around its center of perspective
Step 1: Find corresponding features in a pair of image
Step 2: Compute transformation from 2
nd
to 1
st
imageStep 3: Warp 2ndimage so it overlays 1stimage
Step 4: Blend images where they overlap one another
repeat for 3rdimage and mosaic of first two, etc.17
5/20/2018 Kari Panoramas 02mar10 Opt
18/113
Aligning images: Translation?
Translations are not enough to align the images
left on top right on top
18
5/20/2018 Kari Panoramas 02mar10 Opt
19/113
A pencil of rays contains all views
real
camerasynthetic
camera
Can generate any synthetic camera view
as long as it has the same center of projection!
and scene geometry does not matter 19
5/20/2018 Kari Panoramas 02mar10 Opt
20/113
Reprojecting an image onto
a different picture plane
the view on any picture plane can be projected onto
any other plane in 3D without changing itsappearance as seen from the center of projection
the sidewalk art of Julian Beever
20
5/20/2018 Kari Panoramas 02mar10 Opt
21/113
mosaic PP
Image reprojection
The mosaic has a natural interpretation in 3D
the images are reprojected onto a common plane
the mosaic is formed on this plane
mosaic is asynthetic wide-angle camera
21
5/20/2018 Kari Panoramas 02mar10 Opt
22/113
Which transform to use?
Translation
2 unknowns
Affine
6 unknowns
Perspective
8 unknowns
22
5/20/2018 Kari Panoramas 02mar10 Opt
23/113
Homography
Projective mapping between any two PPs with thesame center of projection
rectangle should map to arbitrary quadrilateral
parallel lines arent
but must preserve straight lines
called Homography PP2
PP1
wx'
wy'
w
"
#
$$$
%
&
'''
=
h11 h12 h13
h21
h22
h23
h31
h32
h33
"
#
$$$
%
&
'''
x
y
1
"
#
$$$
%
&
'''
H pp
To apply a homography H Compute p= Hp (regular matrix multiply)
Convert pfrom homogeneous to imagecoordinates [x,y] (divide by w)
23
5/20/2018 Kari Panoramas 02mar10 Opt
24/113
24
Homography from mapping quads
Fundamentals of Texture Mapping and Image Warping
Paul Heckbert, M.Sc. thesis, U.C. Berkeley, June 1989, 86 pp.http://www.cs.cmu.edu/~ph/texfund/texfund.pdf
5/20/2018 Kari Panoramas 02mar10 Opt
25/113
Multiply out
wx = h11x + h12y + h13
wy = h21x + h22y + h23
w = h31x + h32y + h33
Get rid of w (h31x + h32y + h33)x (h11x + h12y + h13) = 0
(h31x + h32y + h33)y (h21x + h22y + h23) = 0
Create a new system Ah = 0
Each point constraint gives two rows of A
[-x -y -1 0 0 0 xx yx x]
[ 0 0 0 -x -y -1 xy yy y]
Solve with singular value decompositionof A = USVT
solution is in the nullspace of A
the last column of V (= last row of VT)
25
Homography from npoint correspondences
wx'
wy'
w
"
#
$$$
%
&
'''
=
h11
h12
h13
h21
h22
h23
h31
h32
h33
"
#
$$$
%
&
'''
x
y
1
"
#
$$$
%
&
'''
H pp
h11
h12
h13
h21
h = h22 h23
h31
h32
h33
5/20/2018 Kari Panoramas 02mar10 Opt
26/113
from numpy import *
# create original points
X = ones([3,4])
X[:2,:] = random.rand(2,4)
x,y = X[0],X[1]
# create projective matrix
H = random.rand(3,3)
# create the target points
Y = dot(H,X)
# homogeneous division
YY = (Y / Y[2])[:2,:]
u,v = YY[0],YY[1]
A = zeros([8,9])
for i in range(4):
A[2*i ] = [-x[i], -y[i], -1, 0, 0, 0, x[i] * u[i], y[i] * u[i], u[i]]
A[2*i+1] = [ 0, 0, 0, -x[i], -y[i], -1, x[i] * v[i], y[i] * v[i], v[i]]
[u,s,vt] = linalg.svd(A)
# reorder the last row of vt to 3x3 matrix
HH = vt[-1,:].reshape([3,3])
# test that the matrices are the same (within a multiplicative factor)
print H - HH * (H[2,2] / HH[2,2])
Python test code
26
5/20/2018 Kari Panoramas 02mar10 Opt
27/113
Summary of perspective stitching
Pick one image, typically the central view (red outline)
Warp the others to its plane
Blend27
5/20/2018 Kari Panoramas 02mar10 Opt
28/113
Example
perspective reprojection
common
picture
plane of
mosaicimage
28
5/20/2018 Kari Panoramas 02mar10 Opt
29/113
Using 4 shots instead of 3
perspective reprojection29
5/20/2018 Kari Panoramas 02mar10 Opt
30/113
cylindrical reprojection
Back to 3 shots
surface ofcylinder
30
5/20/2018 Kari Panoramas 02mar10 Opt
31/113
cylindrical reprojection
Back to 3 shots
surface ofcylinder
31
5/20/2018 Kari Panoramas 02mar10 Opt
32/113
Back to 3 shots
perspective reprojection32
5/20/2018 Kari Panoramas 02mar10 Opt
33/113
Cylindrical panoramas
What if you want a 360 panorama?
Project each image onto a cylinder
A cylindrical image is a rectangular array
mosaicimage
x
y
33
5/20/2018 Kari Panoramas 02mar10 Opt
34/113
Cylindrical panoramas
What if you want a 360 panorama?
Project each image onto a cylinder
A cylindrical image is a rectangular array
To view without distortion
reproject a portion of the cylinder onto a picture plane representing the
display screen
mosaicimage
x
y
34
5/20/2018 Kari Panoramas 02mar10 Opt
35/113
2ndreprojection to a plane for display
Imagine photographing the inside of a cylinder that iswallpapered with this panorama
if your FOV is narrow, your photo wont be too distorted
display
FOV
35
5/20/2018 Kari Panoramas 02mar10 Opt
36/113
http://graphics.stanford.edu/courses/cs178/applets/projection.html
36
Demo
5/20/2018 Kari Panoramas 02mar10 Opt
37/113
Changing camera center
Does it still work? synthetic PP
PP1
PP2
37
5/20/2018 Kari Panoramas 02mar10 Opt
38/113
Where to rotate? Nodal point?
http://www.reallyrightstuff.com/pano/index.html
38
5/20/2018 Kari Panoramas 02mar10 Opt
39/113
Many instructions say rotate around
the nodal point wrong!
http://toothwalker.org/optics/
misconceptions.html#m6
Correct: the entrance pupil
the optical image of the physical aperture stop
as 'seen' through the front of the lens system
due to the magnifying effect of the front lens,
the entrance pupil's location is nearer than
that of the physical aperture
39
Rotate around the center of lens perspective
The front and rear nodal points
have the property that a rayaimed at one of them will be
refracted by the lens such that itappears to have come from the
other, and with the same angle
with respect to the optical axis.
5/20/2018 Kari Panoramas 02mar10 Opt
40/113
40
Test for
parallax
http://toothwalker.org/optics/cop.html#stitching
5/20/2018 Kari Panoramas 02mar10 Opt
41/113
41
Wrong center of rotation -> parallax
5/20/2018 Kari Panoramas 02mar10 Opt
42/113
Map 3D point (X,Y,Z) onto cylinder
Cylindrical projection
X
Y
Z
unit cylinder
unwrapped cylinder
Convert to cylindrical coordinates
Convert to cylindrical image
coordinates
42
5/20/2018 Kari Panoramas 02mar10 Opt
43/113
Cylindrical projection
Y
X43
5/20/2018 Kari Panoramas 02mar10 Opt
44/113
Inverse cylindrical projection
X
Y
Z
(X,Y,Z)
(sin!,h,cos!)
44
5/20/2018 Kari Panoramas 02mar10 Opt
45/113
f = 180 (pixels)
Focal length
f = 380f = 280Image 384x300
top-down view Focal length
45
5/20/2018 Kari Panoramas 02mar10 Opt
46/113
Can get a rough estimate by measuring FOV:
Can use the EXIF data tag
might not give the right thing
Can use several images together
find f that makes them matchEtc.
Focal length is (highly!) camera dependant
46
5/20/2018 Kari Panoramas 02mar10 Opt
47/113
Assembling the panorama
Stitch pairs together, blend, then crop
47
5/20/2018 Kari Panoramas 02mar10 Opt
48/113
Problem: Drift
Vertical Error accumulation small (vertical) errors accumulate over time
apply correction so that sum = 0 (for 360 pan.)
Horizontal Error accumulation
can reuse first/last image to find the right panorama radius
48
5/20/2018 Kari Panoramas 02mar10 Opt
49/113
Spherical projection
Convert to spherical coordinatesX
Y
Z
Map 3D point (X,Y,Z) onto sphere
Convert to spherical image coordinates
unwrapped sphere
"
49
5/20/2018 Kari Panoramas 02mar10 Opt
50/113
Spherical Projection
Y
X50
5/20/2018 Kari Panoramas 02mar10 Opt
51/113
Inverse Spherical projection
X
Y
Z
(x,y,z)
(sin!cos",cos!cos",sin"
)
51
5/20/2018 Kari Panoramas 02mar10 Opt
52/113
Full-view Panorama
52
5/20/2018 Kari Panoramas 02mar10 Opt
53/113
Building a Panorama
M. Brown and D. G. Lowe. Recognising Panoramas. ICCV 200353
5/20/2018 Kari Panoramas 02mar10 Opt
54/113
We need to match (align) images
54
5/20/2018 Kari Panoramas 02mar10 Opt
55/113
Detect feature points in both images
55
5/20/2018 Kari Panoramas 02mar10 Opt
56/113
Find corresponding pairs
56
5/20/2018 Kari Panoramas 02mar10 Opt
57/113
Use these pairs to align images
57
5/20/2018 Kari Panoramas 02mar10 Opt
58/113
Matching with Features
Problem 1:
Detect thesamepoint independentlyin both images
no chance to match!
We need a repeatable detector
58
5/20/2018 Kari Panoramas 02mar10 Opt
59/113
Matching with Features
Problem 2:
For each point correctly recognize the corresponding one
?
We need a reliable and distinctivedescriptor
59
5/20/2018 Kari Panoramas 02mar10 Opt
60/113
We should easily recognize the point by looking through
a small window
Shifting a window in any direction should give a large
change in intensity
Harris Corners: The Basic Idea
60
5/20/2018 Kari Panoramas 02mar10 Opt
61/113
Harris Detector: Basic Idea
flatregion:
no change in all
directions
edge:
no change along the
edge direction
corner:
significant change in all
directions
61
5/20/2018 Kari Panoramas 02mar10 Opt
62/113
Harris Detector: Mathematics
Window-averaged change of intensity for the shift [u,v]:
IntensityShiftedintensity
Windowfunction
orWindow function w(x,y)=
Gaussian1 in window, 0 outside62
5/20/2018 Kari Panoramas 02mar10 Opt
63/113
Harris Detector: Mathematics
Expanding E(u,v)in a 2ndorder Taylor series expansion,
we have, for small shifts [u,v], a bilinearapproximation:
where Mis a 2$2 matrix computed from image derivatives:
63
5/20/2018 Kari Panoramas 02mar10 Opt
64/113
Eigenvalues%1,%2of M at different locations
%1and %2are large
64
5/20/2018 Kari Panoramas 02mar10 Opt
65/113
Eigenvalues%1,%2of M at different locations
large%1, small %265
5/20/2018 Kari Panoramas 02mar10 Opt
66/113
Eigenvalues%1,%2of M at different locations
small%1, small %266
5/20/2018 Kari Panoramas 02mar10 Opt
67/113
Harris Detector: Mathematics
%1
%2
Corner
%1and %2are large,
%1 ~ %2;Eincreases in all
directions
%1and %2are small;
E is almost constantin all directions
Edge%1>> %2
Edge
%2>> %1
Flatregion
Classification of
image points using
eigenvalues of M:
67
5/20/2018 Kari Panoramas 02mar10 Opt
68/113
Harris Detector: Mathematics
Measure of corner response:
(k empirical constant, k = 0.04-0.06)
68
5/20/2018 Kari Panoramas 02mar10 Opt
69/113
Harris Detector: Mathematics
%1
%2 Corner
Edge
Edge
Flat
Rdepends only on
eigenvalues of M
Ris large for a corner
Ris negative with large
magnitude for an edge
|R| is small for a flat
region
R> 0
R< 0
R< 0|R|small
69
5/20/2018 Kari Panoramas 02mar10 Opt
70/113
Ha rris Detector Wo rkflow
70
5/20/2018 Kari Panoramas 02mar10 Opt
71/113
Ha rris Detector Wo rkflow
Compute corner response R
71
5/20/2018 Kari Panoramas 02mar10 Opt
72/113
Ha rris Detector Wo rkflow
Find points with large corner response: R>threshold
72
5/20/2018 Kari Panoramas 02mar10 Opt
73/113
Ha rris Detector Wo rkflow
Take only the points of local maxima of R
73
5/20/2018 Kari Panoramas 02mar10 Opt
74/113
Ha rris Detector Wo rkflow
74
5/20/2018 Kari Panoramas 02mar10 Opt
75/113
Harris Detector: Summary
Average intensity change in direction [u,v]can beexpressed as a bilinear form:
Describe a point in terms of eigenvalues of M:measure of corner response
A good (corner) point should have a large intensity change
in all directions, i.e., Rshould be large positive75
5/20/2018 Kari Panoramas 02mar10 Opt
76/113
Harris Detector: Invariant to rotation
Ellipse rotates
but its shape (i.e., eigenvalues) remains the same
Corner response Ris invariant to image rotation
76
5/20/2018 Kari Panoramas 02mar10 Opt
77/113
Partial invariance
Harris Detector: ~Invariant to intensite change
!
Only derivatives are used
=>
invariance to intensity shiftI&I + b
!
Intensity scale:I &a I
R
x(image coordinate)
threshold
R
x(image coordinate)
77
5/20/2018 Kari Panoramas 02mar10 Opt
78/113
Harris Detector:ot
invariant to image scale!
All points will be
classified as edges
Corner !
78
5/20/2018 Kari Panoramas 02mar10 Opt
79/113
We know how to detect points
Next question:
How to match them?
Point Descriptors
?
Point descriptor should be:
1.
Invariant
2. Distinctive79
5/20/2018 Kari Panoramas 02mar10 Opt
80/113
Descriptor overview:
Determine scale (by maximizing DoG in scale and in space),
local orientation as the dominant gradient direction.
Use this scale and orientation to make all further computations invariant to
scale and rotation.
SIFT Scale Invariant Feature Transform
D. Lowe. Distinctive Image Features from Scale-Invariant Keypoints IJCV 2004
5/20/2018 Kari Panoramas 02mar10 Opt
81/113
Descriptor overview:
Determine scale (by maximizing DoG in scale and in space),
local orientation as the dominant gradient direction.
Use this scale and orientation to make all further computations invariant to
scale and rotation.
Compute gradient orientation histograms of several small windows (128
values for each point)
Normalize the descriptor to make it invariant to intensity change
SIFT Scale Invariant Feature Transform
D. Lowe. Distinctive Image Features from Scale-Invariant Keypoints IJCV 2004
5/20/2018 Kari Panoramas 02mar10 Opt
82/113
82
5/20/2018 Kari Panoramas 02mar10 Opt
83/113
Registration in practice: tracking
Camera Module
Real-Time
Tracking
!"#$% '()*$+
time
Current location
83
5/20/2018 Kari Panoramas 02mar10 Opt
84/113
84
Viewfinder alignment for tracking
Andrew Adams, Natasha Gelfand, Kari Pulli
Viewfinder Alignment
Eurographics 2008 http://graphics.stanford.edu/papers/viewfinderalignment/
5/20/2018 Kari Panoramas 02mar10 Opt
85/113
85
Project gradients along columns and rows
5/20/2018 Kari Panoramas 02mar10 Opt
86/113
86
diagonal gradients along diagonals
5/20/2018 Kari Panoramas 02mar10 Opt
87/113
87
and find corners
5/20/2018 Kari Panoramas 02mar10 Opt
88/113
88
Overlap and match the gradient projections
and determine translation
5/20/2018 Kari Panoramas 02mar10 Opt
89/113
89
Apply the best translation to corners
5/20/2018 Kari Panoramas 02mar10 Opt
90/113
90
Match corners, refine translation & rotation
5/20/2018 Kari Panoramas 02mar10 Opt
91/113
System Overview
Camera Module
Real-Time
Tracking
!"#$% '()*$+
,"-. /$+%012%3
4*)-$+
time
Current location Panoramaexpansion
91
5/20/2018 Kari Panoramas 02mar10 Opt
92/113
System Overview
Camera Module
Real-Time
Tracking
!"#$% '()*$+
,"-. /$+%012%3
4*)-$+
Image
Warping
Image
Registration
92
5/20/2018 Kari Panoramas 02mar10 Opt
93/113
Requirements
Robust to illumination changes
Robust to object motion in the scene
Low computational complexity
Image registration
Registration example (including large illumination changes)93
5/20/2018 Kari Panoramas 02mar10 Opt
94/113
Hybrid multi-resolution approach
I.B.
F.B.
F.B.
F.B.
Initial guess
Registration parameters
I.B. Image Based
F.B. Feature Based
94
K. Pulli, M. Tico, Y. Xiong, X. Wang, C-K. Liang, Panoramic Imaging System for Camera Phones, ICCE 2010
5/20/2018 Kari Panoramas 02mar10 Opt
95/113
Progression of multi-resolution registration
Actual
size
Applied
to hi-res
95
5/20/2018 Kari Panoramas 02mar10 Opt
96/113
Feature-based registration
+ +
Convert to spherical
coordinates
Convert from spherical
coordinates
Apply the previous
registration estimate
+ +
Best block cross-
correlation match
Feature Detection
(Harris corners)
Previous estimate
Feature Matching
(spherical coordinates)RANSAC
Validity
check
New estimate
valid
Update search range
invalid
96
5/20/2018 Kari Panoramas 02mar10 Opt
97/113
System overview
97
Camera Module
Real-Time
Tracking
!"#$% '()*$+
,"-. /$+%012%3
4*)-$+
Image
Warping
Image
Registration
5/20/2018 Kari Panoramas 02mar10 Opt
98/113
System overview
Camera Module
Real-Time
Tracking
!"#$% '()*$+
,"-. /$+%012%3
4*)-$+
Image
Warping
Image
Registration
Image
Blending
!"#$% '$#()$*$
Photo by Marius Tico
98
5/20/2018 Kari Panoramas 02mar10 Opt
99/113
Image blending
Directly averaging the overlapped pixels results in ghosting artifacts
Moving objects, errors in registration, parallax, etc.
Photo by Chia-Kai Liang
99
5/20/2018 Kari Panoramas 02mar10 Opt
100/113
Solution: Image labeling
Assign one input image each output pixel
Optimal assignment can be found by graph cut [Agarwala et al. 2004]
100
5/20/2018 Kari Panoramas 02mar10 Opt
101/113
New artifacts
Inconsistency between pixels from different input images
Different exposure/white balance settings
Photometric distortions (e.g., vignetting)
101
5/20/2018 Kari Panoramas 02mar10 Opt
102/113
Solution: Poisson blending
Copy the gradient field from the input imageReconstruct the final image by solving a Poisson equation
Combined gradient field
102
5/20/2018 Kari Panoramas 02mar10 Opt
103/113
Seam finding gets difficult w hen colors differ
No color correction With color correction
103
Y. Xiong, K. Pulli, Fast Panorama Stitching on Mobile Devices, ICCE 2010
5/20/2018 Kari Panoramas 02mar10 Opt
104/113
Color correction in linearized RGB
104
5/20/2018 Kari Panoramas 02mar10 Opt
105/113
105
Dyn am ic programm ing finds go od cuts fast
" Cumulative minimum error surface
" Error surface
Overlapping area in the current composite image
Overlapping area in the current source image
Overlapping areaSource images
Error surf cumulative
Optimal path
5/20/2018 Kari Panoramas 02mar10 Opt
106/113
106
Fast Poisson blending
SIGGRAPH 2009
5/20/2018 Kari Panoramas 02mar10 Opt
107/113
107
Alpha blending
5/20/2018 Kari Panoramas 02mar10 Opt
108/113
p g
After labeling
Poisson blending
108
5/20/2018 Kari Panoramas 02mar10 Opt
109/113
System Overview
Camera Module
Real-Time
Tracking
!"#$% '()*$+
,"-. /$+%012%3
4*)-$+
Image
Warping
Image
Registration
Image
Blending
5($6"$7 %3 5.%3$
!"#$% '$#()$*$
Photo by Marius Tico
109
5/20/2018 Kari Panoramas 02mar10 Opt
110/113
Panorama Visualization
Trivial method:
Show the whole panorama on the screen
Zooming and panning
110
5/20/2018 Kari Panoramas 02mar10 Opt
111/113
No Projection Method is Optimal
Zoom Spherical Projection Perspective Projection
111
5/20/2018 Kari Panoramas 02mar10 Opt
112/113
Solution: Interpolate the Projection Coordinates
Weights are determined
by a sigmoid function ofzoom factor
Zoom
112
5/20/2018 Kari Panoramas 02mar10 Opt
113/113
Slide credits
Fredo Durand
Alyosha Efros
Bill Freeman
Marc Levoy
Chia-Kai Liang
Steve Seitz
Rick Szeliski
Marius Tico
Yingen Xiong