64
Sensors and Sensing Cameras and Camera Calibration, Stereo and Structured Light Todor Stoyanov Mobile Robotics and Olfaction Lab Center for Applied Autonomous Sensor Systems Örebro University, Sweden [email protected] 12.11.2015 T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 1 / 34

Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

  • Upload
    others

  • View
    2

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Sensors and SensingCameras and Camera Calibration, Stereo and Structured

Light

Todor Stoyanov

Mobile Robotics and Olfaction LabCenter for Applied Autonomous Sensor Systems

Örebro University, [email protected]

12.11.2015

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 1 / 34

Page 2: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Outline

1 Camera Models

2 Camera Calibration

3 Color, Infrared and Thermal Cameras

4 Stereo Cameras

5 Structured Light Cameras

6 Practice: Openni and ROS

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 2 / 34

Page 3: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Camera Models

Outline

1 Camera Models

2 Camera Calibration

3 Color, Infrared and Thermal Cameras

4 Stereo Cameras

5 Structured Light Cameras

6 Practice: Openni and ROS

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 2 / 34

Page 4: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Camera Models

Electromagnetic Spectrum

Cameras are passive devices that measure electromagnetic radiation,reflected by objects in the environment.

Conventional cameras detect light in the visible range of theelectromagnetic spectrum: wavelengths between 430nm-790nm.

... but plenty of cameras built for other ranges: e.g., infrared, thermal, UV.

Wavelength (m)Radiation type

Radio Microwave Infrared Visible Ultraviolet X-ray Gamma

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 2 / 34

Page 5: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Camera Models

CCD Sensors, Optics and Shutters

Modern cameras consist of a light sensitive element,a lens and (optionally) a shutter.

The light sensor is usually implemented as a ChargeCoupled Device (CCD) printed on a CMOS chip.

Lenses focus light onto the CCD array.

Mechanical shutters can be used to only expose thechip for a short period of time.

Electronic shutters are often used instead.

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 3 / 34

Page 6: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Camera Models

Digital Images

A digital image I can be thought of asa function f (x,y) : X×Y→ Z, whereX = [0,Px] ∈ N and Y = [0,Py] ∈ Nare pixel coordinates in the imageplane.Depending on the type of image Zcan be:

Binary image if Z = {0,1}Gray scale image if Z ⊂ RColor image if Z ⊂ R3

Each pixel in the image correspondsto a single cell of the CCD array.

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 4 / 34

Page 7: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Camera Models

Digital Images

A digital image I can be thought of asa function f (x,y) : X×Y→ Z, whereX = [0,Px] ∈ N and Y = [0,Py] ∈ Nare pixel coordinates in the imageplane.Depending on the type of image Zcan be:

Binary image if Z = {0,1}Gray scale image if Z ⊂ RColor image if Z ⊂ R3

Each pixel in the image correspondsto a single cell of the CCD array.

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 4 / 34

Page 8: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Camera Models

The Pinhole Camera

A pinhole camera is a simple camera without a lens that projects lightdirectly on an image plane.

The pinhole camera model can be extended to model complex cameras

Some definitions:

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 5 / 34

Page 9: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Camera Models

The Pinhole Camera

The pinhole camera model can be extended to model complex cameras

Some definitions:

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 5 / 34

Page 10: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Camera Models

The Pinhole Camera

principal axis

image plane

camera center

The pinhole camera model can be extended to model complex cameras

Some definitions:

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 5 / 34

Page 11: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Camera Models

The Pinhole Camera

Camera center: the focal point of allrays converging to the camera

Principal axis: by definition this is theZ axis pointing out of the cameracenter

Image plane: the CCD plane wherethe image is acquired

Focal length f : the vector pointingfrom the camera center to the imageplane

principal axis

image plane

camera center

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 6 / 34

Page 12: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Camera Models

The Pinhole Camera

Given a point s = (x,y,z,1)T in worldcoordinate frame (homogeneouscoordinates), we can obtain theprojection of the point to acorresponding pixel (xu,yu) on theimage plane as:

xu

yu

1

fx 0 cx 00 fy cy 00 0 1 0

H

xyz1

(1)

= KHs (2)

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 6 / 34

Page 13: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Camera Models

Lens Distortion

1

In practice, adding a lens to the system adds several different types ofdistortion

radial distortion is due to imperfections of the lens curvaturetangential distortion is due to imperfect alignment of the lens center andthe principle axis

other types of distortion are more difficult to model.

1http://www.uni-koeln.de/~al001/radcor_files/hs100.htmT. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 7 / 34

Page 14: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Camera Models

Modeling Distortion

Going back to Eq. 1, consider the projection of s = (x,y,z,1)T onto the imageplane:

s′ =

x′

y′

1

=

x/zy/z1

(3)

We define r =√

x′2 + y′2 as the radius of the projected point, relative to theprincipal point. The undistorted point s can then be computed as:

s =

x′(1+ k1r2 + k2r4)+2p1x′y′+p2(r2 +2x′2)y′(1+ k1r2 + k2r4)+p1(r2 +2y′2)+2p2x′y′

1

(4)

The undistorted pixel coordinates of s are then obtained as: xu

yu

1

= Ks (5)

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 8 / 34

Page 15: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Camera Calibration

Outline

1 Camera Models

2 Camera Calibration

3 Color, Infrared and Thermal Cameras

4 Stereo Cameras

5 Structured Light Cameras

6 Practice: Openni and ROS

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 9 / 34

Page 16: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Camera Calibration

2D/3D Calibration Patterns

In order to determine the focal lengthand center offsets for the cameramatrix K, the radial distortioncoefficients k1,k2 and the tangentialdistortions p1,p2 cameras arecalibrated.

Calibration from a natural scene is noteasily done, so we use calibrationpatterns

2D patterns: known pattern printed on a plane

rarely 3D pattern: known 3D geometry

2

2ROS camera calibration tutorialT. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 9 / 34

Page 17: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Camera Calibration

Chessboard calibration: basics

The number of chessboard squaresand their size is known in advance.

We fix the world reference frame tothe top-left corner of the board.

All points lie on a plane in worldframe, with z = 0. This means we candrop one column of rotationcoefficients from H

Note: the following slides follow the derivations as shown here3.

3http://ais.informatik.uni-freiburg.de/teaching/ws10/robotics2/pdfs/rob2-10-camera-calibration.pdf

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 10 / 34

Page 18: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Camera Calibration

Chessboard calibration: formulation

For a point (x,y,z,1)T we have :

xu

yu

1

= K

r11 r12 r13 t1r21 r22 r23 t2r31 r32 r33 t30 0 0 1

xyz1

(6)

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 11 / 34

Page 19: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Camera Calibration

Chessboard calibration: formulation

For a point (x,y,z,1)T we have, we set z=0, thus :

xu

yu

1

= K

r11 r12 0 t1r21 r22 0 t2r31 r32 0 t30 0 0 1

xy01

(7)

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 11 / 34

Page 20: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Camera Calibration

Chessboard calibration: formulation

For a point (x,y,z,1)T we have, we set z=0, thus : xu

yu

1

=

fx 0 cx

0 fy cy0 0 1

r11 r12 t1r21 r22 t2r31 r32 t3

xy1

(8)

= H

xy1

(9)

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 11 / 34

Page 21: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Camera Calibration

Chessboard calibration: formulation

For a point (x,y,z,1)T we have, we set z=0, thus : xu

yu

1

=

fx 0 cx

0 fy cy0 0 1

r11 r12 t1r21 r22 t2r31 r32 t3

xy1

(8)

= H

xy1

(9)

H is called the homography matrix. Let:

H = (h1,h2,h3) = K(r1,r2, t) (10)

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 11 / 34

Page 22: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Camera Calibration

Chessboard calibration: formulation

Knowing(h1,h2,h3) = K(r1,r2, t) (11)

we have that r1 = K−1h1 and r2 = K−1h2. We also know that r1 and r2 arecolumns from a rotation matrix, thus they form an orthonormal basis, i.e.rT

1 r2 = 0 and rT1 r1 = rT

2 r2 = 1. Therefore:

rT1 r2 = 0 (12)

hT1 K−TK−1h2 = 0 (13)

and

rT1 r1 = rT

2 r2 (14)

hT1 K−TK−1h1 = hT

2 K−TK−1h2 (15)

hT1 K−TK−1h1−hT

2 K−TK−1h2 = 0 (16)

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 12 / 34

Page 23: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Camera Calibration

Chessboard calibration: solving the problem

Re-formulating equations 13 and 16 and unwrapping the coefficients ofB = K−TK−1 as b = (b11,b12,b13,b22,b23,b33) (B is symmetric) we canformulate

Vb = 0 (17)

where V holds the coefficients from H as in equations 13 and 16.

As we know the relative positions of the points on the pattern, we canobtain V for an image and solve for b.

The parameters of K can be obtained by Cholesky factorization ofB = LLT

Measurements are noisy, so we instead solve a least squares problem tominimize Vb

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 13 / 34

Page 24: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Camera Calibration

Chessboard calibration: distortion parameters

The previous derivations solve the problem for the camera matrix K, butignore distortion.

In order to solve for distortion, we need to formulate the re-projectionerror.

The resulting non-linear optimization problem is usually solved in batch byusing the Levenberg-Marquardt method and linearization around thesolution for K at every iteration.

Fortunately, you don’t typically have to solve the optimization problemyourself. Just use one of the many toolboxes for calibration.

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 14 / 34

Page 25: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Camera Calibration

Chessboard calibration: distortion parameters

The previous derivations solve the problem for the camera matrix K, butignore distortion.

In order to solve for distortion, we need to formulate the re-projectionerror.

The resulting non-linear optimization problem is usually solved in batch byusing the Levenberg-Marquardt method and linearization around thesolution for K at every iteration.

Fortunately, you don’t typically have to solve the optimization problemyourself. Just use one of the many toolboxes for calibration.

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 14 / 34

Page 26: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Color, Infrared and Thermal Cameras

Outline

1 Camera Models

2 Camera Calibration

3 Color, Infrared and Thermal Cameras

4 Stereo Cameras

5 Structured Light Cameras

6 Practice: Openni and ROS

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 15 / 34

Page 27: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Color, Infrared and Thermal Cameras

Color Cameras

The world is colorful!

Color images are obtained by addinga filter designed for specificwavelengths to each pixel of the CCDarray

The filter pattern is called a Bayerfilter.

Typically, we have red, green and bluesensitive pixels.

Raw pixel values often come as astream and camera drivers performde-bayering to obtain thecorresponding RGB vector.

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 15 / 34

Page 28: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Color, Infrared and Thermal Cameras

Color Cameras

The world is colorful!

Color images are obtained by addinga filter designed for specificwavelengths to each pixel of the CCDarray

The filter pattern is called a Bayerfilter.

Typically, we have red, green and bluesensitive pixels.

Raw pixel values often come as astream and camera drivers performde-bayering to obtain thecorresponding RGB vector.

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 15 / 34

Page 29: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Color, Infrared and Thermal Cameras

Color Spaces

Several different models for colors.

RGB systems encode values in equal sizedintervals for red, green and blue.

HSV space encodes the color hue, saturationand value of every pixel with

Different color spaces have advantages indifferent operations.

Color spaces are not equivalent, but we canconvert between representations

H ∈ [0◦,360◦)

S ∈ [0,1]

V ∈ [0,1]

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 16 / 34

Page 30: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Color, Infrared and Thermal Cameras

Infrared and Thermal Cameras

Cameras sensitive to other parts ofthe EM spectrum

IR cameras measure reflected IRlight.

Often used with an IR diode lightsource for night-time securityapplications.

Thermal cameras can detect emittedIR light.

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 17 / 34

Page 31: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Color, Infrared and Thermal Cameras

Infrared and Thermal Cameras

Cameras sensitive to other parts ofthe EM spectrum

IR cameras measure reflected IRlight.

Often used with an IR diode lightsource for night-time securityapplications.

Thermal cameras can detect emittedIR light.

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 17 / 34

Page 32: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Stereo Cameras

Outline

1 Camera Models

2 Camera Calibration

3 Color, Infrared and Thermal Cameras

4 Stereo Cameras

5 Structured Light Cameras

6 Practice: Openni and ROS

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 18 / 34

Page 33: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Stereo Cameras

Stereo Cameras Basics

Stereo cameras have been in use inrobotics since its early days.

The main principle of operation is touse two (or more) cameras withprecisely known relative offsets.

Points in the images are identified andmatched.

3D distance to matching points canbe determined using triangulation.

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 18 / 34

Page 34: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Stereo Cameras

Epipolar Geometry

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 19 / 34

Page 35: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Stereo Cameras

Epipolar Geometry

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 19 / 34

Page 36: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Stereo Cameras

Epipolar Geometry

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 19 / 34

Page 37: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Stereo Cameras

Epipolar Geometry

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 19 / 34

Page 38: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Stereo Cameras

Epipolar Geometry

The line lm1 is the projection of theline between M and C1 onto I2.

Any point m1 on I1 defines a line lm1 :an epipolar line.

The epipolar lines define a plane onwhich m1, m2 and M lie.

The line between e1 and e2 links thetwo camera centers. All epipolar linespass through e1 and e2.

lm1 = Em1 and lm2 = ETm2

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 20 / 34

Page 39: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Stereo Cameras

Epipolar Geometry

The matrix E is determined by the rotation R and translation t betweenthe two camera centers C1 and C2

A stereo camera pair is completely described by a fundamental matrix F,such that:

K1 and K2 are the camera matrices as described previous lecture.F = K−T

1 EK−12

if m1 and m2 are the projections of M onto C1 and C2, then:

mT1 Fm2 = 0 (18)

Knowing F and the pixel coordinates of a corresponding point m1,m2 wecan find the depth along Z in each of the cameras C1,C2

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 21 / 34

Page 40: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Stereo Cameras

Disparity and Rectification

The above setup can be used for triangulationof corresponding points.

For practical reasons stereo cameras are builtwith only a horizontal displacement betweenC1 and C2.

Even in cases where the cameras are notperfectly aligned, images are usually rectifiedto be aligned

Advantage: searching for correspondencescan be done on a line, instead of a plane.

Using the pixel distance d = x− x′, we get thedepth Z = (x− x′)Bf

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 22 / 34

Page 41: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Stereo Cameras

Disparity and Rectification

The above setup can be used for triangulationof corresponding points.

For practical reasons stereo cameras are builtwith only a horizontal displacement betweenC1 and C2.

Even in cases where the cameras are notperfectly aligned, images are usually rectifiedto be aligned

Advantage: searching for correspondencescan be done on a line, instead of a plane.

Using the pixel distance d = x− x′, we get thedepth Z = (x− x′)Bf

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 22 / 34

Page 42: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Stereo Cameras

Disparity and Rectification

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 23 / 34

Page 43: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Stereo Cameras

Disparity and Rectification

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 23 / 34

Page 44: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Stereo Cameras

Disparity and Rectification

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 23 / 34

Page 45: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Stereo Cameras

Identifying Matching Points

Stereo vision algorithms rely on matching corresponding points betweentwo images.

This causes ambiguities in some cases:

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 24 / 34

Page 46: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Stereo Cameras

Identifying Matching Points

Stereo vision algorithms rely on matching corresponding points betweentwo images.

This causes ambiguities in some cases:

Two basic classes of approaches:

Intensity-based:

Feature-based:

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 24 / 34

Page 47: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Stereo Cameras

Identifying Matching Points

Stereo vision algorithms rely on matching corresponding points betweentwo images.

This causes ambiguities in some cases:

Two basic classes of approaches:Intensity-based:

Basic idea: match the information in the intensity values of each line.At each pixel minimize the difference in intensities.

Feature-based:

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 24 / 34

Page 48: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Stereo Cameras

Identifying Matching Points

Stereo vision algorithms rely on matching corresponding points betweentwo images.

This causes ambiguities in some cases:

Two basic classes of approaches:

Intensity-based:Feature-based:

Basic idea: extract information using a group of pixel values and theirspatial distribution.Commonly used are simple features, e.g. edge and corner detectors(Canny, Harris, LoG filters).

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 24 / 34

Page 49: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Stereo Cameras

Stereo Calibration

Procedure for calibrating is similar as for a single camera, relying on anumber of known pointsThe Fundamental matrix F is rank 2, i.e., there are 7 DoFs that need tobe optimized.Typical algorithms are the 7-point and 8-point correspondence algorithms.

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 25 / 34

Page 50: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Stereo Cameras

Limitations

Lack of texture.

Lightning conditions.

Processing becomes more costly for higher resolutions.

Need to verify depth maps using more complicated algorithms.

Quantization effects due to discretisation of pixels.

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 26 / 34

Page 51: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Structured Light Cameras

Outline

1 Camera Models

2 Camera Calibration

3 Color, Infrared and Thermal Cameras

4 Stereo Cameras

5 Structured Light Cameras

6 Practice: Openni and ROS

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 27 / 34

Page 52: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Structured Light Cameras

Structured Light Basics

A structured light camera works by projectinga known light pattern to the scene.

Knowledge of the projected pattern is thenused for triangulation.

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 27 / 34

Page 53: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Structured Light Cameras

Structured Light Basics

An image of the projected pattern is taken at a known distance, storedand used as a virtual camera.

44Figure from [1]T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 28 / 34

Page 54: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Structured Light Cameras

Projected Patterns

The Kinect uses a (pseudo-)randomly distributed speckled pattern.

Each speckle looks differently flat at different distances, due to a speciallens.

The Kinect/PrimeSense camera uses IR light, but visible light is alsopossible.

55Figure from [1]T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 29 / 34

Page 55: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Structured Light Cameras

Projected Patterns

The Kinect uses a (pseudo-)randomly distributed speckled pattern.Each speckle looks differently flat at different distances, due to a speciallens.The Kinect/PrimeSense camera uses IR light, but visible light is alsopossible.

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 29 / 34

Page 56: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Structured Light Cameras

Error Models

Errors for structured light cameras grow with the square of the distance toobjects.Strong quantization effects

5

65Results from [1]6Figure from [1]

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 30 / 34

Page 57: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Structured Light Cameras

Error Models

Errors for structured light cameras grow with the square of the distance toobjects.

Strong quantization effects5

65Results from [1]6Figure from [1]T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 30 / 34

Page 58: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Structured Light Cameras

Limitations

Suffer from illumination variance.

"Pollute" the scene.

Quantization and square distance error can be prohibitive.

... but also many advantages.

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 31 / 34

Page 59: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Practice: Openni and ROS

Outline

1 Camera Models

2 Camera Calibration

3 Color, Infrared and Thermal Cameras

4 Stereo Cameras

5 Structured Light Cameras

6 Practice: Openni and ROS

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 32 / 34

Page 60: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Practice: Openni and ROS

Sample Questions

You are in charge of designing a robot that can map tunnels in an undergroundmine. Your main task is to select a 3D range scanner for this task. The generalproperties of the environment are:

no natural illumination, sparse artificial lightning.

corridors are up to 20m in length and 3m in width.

the walls are not smooth, but may contain interesting natural features.

the vehicles move at speeds of up to 30 km/h.

Your system needs a dense point cloud at high frame rates. Using yourknowledge of current 3D sensing technology, perform an analysis of the pro’sand con’s of the suitable range sensing technologies.

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 32 / 34

Page 61: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Practice: Openni and ROS

Possible Questions

Given a camera fundamental matrix K with fx = fy = 525, cx = 320 andcy = 240. Assume there are no distortion coefficeints on the lens and thecamera is located at the origin. Calculate the pixel projection ux,uy of apoint x = [1.2,4.5,0.6,1]T

The camera has now moved by 0.1m along positive y and 1m alongnegative z. How does this affect the projected point?

Calculate the equation of all points that lie on a particular camera rayux,uy.

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 33 / 34

Page 62: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Practice: Openni and ROS

Practice

Practice task today uses the Asus Xtion camera you have received for thenext lab.

Setup the sensor and write a node that listens to the RGB images andcamera info messages.

Use the camera parameters to undistort the image.

Convert the image to a gray scale image and re-publish it. Use theopencv library.

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 34 / 34

Page 63: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Practice: Openni and ROS

Sensors and SensingCameras and Camera Calibration, Stereo and Structured

Light

Todor Stoyanov

Mobile Robotics and Olfaction LabCenter for Applied Autonomous Sensor Systems

Örebro University, [email protected]

12.11.2015

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 34 / 34

Page 64: Sensors and Sensing Cameras and Camera Calibration, Stereo ...130.243.105.49/Research/mro/courses/sens/2015/lecture4.pdf · Camera Models Outline 1 Camera Models 2 Camera Calibration

Practice: Openni and ROS

References

[1] Kourosh Khoshelham and Sander Oude Elberink.

Accuracy and resolution of kinect depth data for indoor mapping applications.

Sensors, 12(2):1437–1454, 2012.

T. Stoyanov (MRO Lab, AASS) Sensors & Sensing 12.11.2015 34 / 34