1
Exploiting Inertial Planes for
Multi-sensor 3D Data Registration
PhD dissertation
Hadi Aliakbarpour
Faculty of Science and Technology
October 2012, University of Coimbra
2
Introduction
3
Introduction: Problem Statement
This dissertation investigate the problem of multi-sensor 3D data registration using a network of IS-camera pairs.
Target applications: Surveillance, human behaviour modelling, virtual-reality, smart-room, health-care, games, teleconferencing, human-robot interaction, medical industries, and scene and object understanding
4
Introduction: Motivation
Performing 3D data registration and scene reconstruction using a set of planar images is still one of the key challenges of computer vision.
A network of cameras, whose usage and ubiquitousness have been increasing in the last decade, can provide such planar images from different views of the scene.
Recently, IS has been becoming much cheaper and more available so that nowadays most smart-phones are equipped in both IS and camera sensors. 3D earth cardinal orientation (North-East-Down) is one of the outputs of an IS.
How can we benefit from having a network of IS and camera couples, for the purpose of 3D data registration?
5
Introduction: Overall View
6
Main Contributions
A homographic framework is developed for 3D data registration using a network of cameras and inertial sensors. Geometric relations among different projective image planes and Euclidean inertial planes involved in the framework are explored. [AD12a] [AD11c] [AD10b] [AD11b] [AD10a] [AFKD10] [AFQ+11].
A real-time prototype of the framework is developed which is able to perform fully reconstruction of human body (and objects) in a large scene. The real-time characteristic is achieved by using a parallel processing architecture on a CUDA-enabled GP-GPU [AAMD11].
A two-point-based method to estimate translations among virtual cameras in the framework is proposed and verified [AD12a] [AD11a] [AD10a] [AFQ+11].
The uncertainties of the homography transformations involved in the framework and their error propagations on the image planes and Euclidean planes have been modelized using statistical geometry.
7
Additional Contributions
Within the context of the proposed framework, a genetic algorithm is developed to provide an optimal coverage of the camera network to a polygonal object (or a scene).
A method to estimate extrinsic parameters among camera and laser range finder is developed [ANP+09]. A related SLaRF; available to download at http://isr.uc.pt/~hadi is prepared.
8
Methodologies on
Main Contributions
9
Virtual Camera: Concept & Geometry
Mapping: From Scene onto Inertial Planes
10A 3D point X is registered on different Euclidean planes using homographies
11
Extension to a Network of IS-Camera Pairs
A network of IS-camera couples is used to observe the scene from different views
3D Reconstruction Using Inertial Planes: Illustrative example
13
Off-the-plane point Y is subject to parallax and on-the-plane point X with no parralax
An exemplary case: a person is observed by three cameras
Top-view of the registration plane. Area in white is the intersection to the person.
14
Translation Among Virtual Cameras
Knowing the heights of two 3D points (X1 and X2) is sufficient to recover the translation (t) among two cameras
15
Parametric Homography RelationsAmong Different Planes in the Framework
16
Real-time implementation using GP-GPU
17
Parallelized Operations: Virtual Planes Generations
18
Modelization of Uncertainties
The uncertainties of the homography transformations involved in the framework and their error propagations on the image planes and Euclidean planes have been modelized using statistical geometry.
19
Modelization of Uncertainties: examples
• Uncertainty of point μ’X , where mapped from μref to μ’ :
• Uncertainty of point μ(k)X , where mapped from μk-1 to μk :
20
Methodologies on
Additional Contributions
21
Optimization for Sensor Configuration (Camera Placement)
The quality of reconstruction using a camera network depends to mainly three parameters:
1. Number of cameras
2. The quality of the applied background subtraction technique
3. The cameras configurations (e.g. positions)
22
Optimization for Sensor Configuration: Problem Statement
C1C2
X
Y
{W} refπ
e1 e2
e3
e4
e5
An exemplary convex polygon with 5 edges are observed by two camera. The problem is how to arrange cameras to have optimum registration of the polygon with most completeness.
After registering with the present camera configuration: An extra part colored in red is registered as a part of the object!
23
Optimization for Sensor Configuration: Solution
Solution: To use geometry (e.g. normal of the edges etc. ), define some cost functions and applying GA.
24
Integration of a Laser Range Finder
11 Camera LRF
Sensitivity to Illumination
Very high NA
Occlusion handling
Weak Fair
Sensitivity to texture
High NA
Precision in range sensing
Fair Very good
Color sensing Very good NA
LRF is an active sensor which can be used as a complementary sensor to the cameras:
Comparison table
10=
31
)()()(
x
LC
LC
LC tRT
Estimation of the rigid transformation, C T L(α) , among a stereo camera and a LRF
25
Experiments
26
Volumetric 3D Reconstruction: Offline
First virtual plane
Second virtual plane
47’th virtual plane
Sta
tue
Set
up a
nd s
cene
27
Analysis on Translation Estimation: Certainties w.r.t. Noise in IS
Empirical analysis the effects of IS noise to the translation estimation method
Input noise in degrees (roll, pitch and yaw of inertial sensor)
Output uncertainty in cm (on three elements of the estimated
translation vector)
28
Output uncertainty (cm)
Input noise (cm)
Analysis on Translation Estimation: Certainties w.r.t. other Noise
Output uncertainty (cm)
Input noise (pixels)
29
Volumetric 3D Reconstruction using Real-time using GPU: video
30
Uncertainty of Virtual Image’s Points
The uncertainties for pixels of the virtual camera’s image plane are demonstrated by covariance ellipses, where they are scaled 1000 times for clarity.
31
Uncertainty of Inertial plane’s Points
The uncertainties for different registered points on the Euclidean inertial plane, demonstrated by covariance ellipses. The blue and red ellipses stand for points registered by the first and second camera, respectively. For the sake of clarity the covariance values are scaled 500 and 600 times, respectively for the first and second cameras
32
Changes of Uncertainty in Virtual Image Plane
Uncertainties for an exemplary pixel x = [ 450 450 1 ]T where s = [ π/2 -π/2 0 ]T
33
Optimization for Sensor Configuration (Camera Placement)
1200x1200 cm2
34
Estimate of Extrinsic Parameters Among Camera And Laser Range Finder
10=
31
)()()(
x
LC
LC
LC tRT
Reprojection of LRF data on the image (blue points)
+Result
Imag
eR
ange
dat
a
α = 2o
α = 12o
α = 23.2o
(during 6 months)
35
Conclusion & Future Work
36
Conclusion
• We investigated the use of IS for 3D data registration by using a network of cameras and inertial sensors.
• A volumetric data registration algorithm was proposed.
• Normally the volumetric reconstruction of a scene is time consuming due to the huge amount of data to be processed. In order to achieve a real-time processing, a prototype was built using GP-GPU and CUDA
• A method to estimate the translation among cameras within the network was proposed. The certainty of the method has been evaluated in the presence of different noise.
• The issue of sensor configuration, particularly the cameras’ positions in the scene was investigated and a geometric method to find an optimal configuration was proposed using genetic algorithm.
• A method to estimate the extrinsic parameters among camera and LRF was proposed as a step towards applying range data in the framework.
37
Future Work
Integration of range data within the proposed inertial-based data registration framework.
To develop a probabilistic algorithm for fusion of heterogeneous data, capable of dealing with the uncertainty of each sensor node.
To investigate a multi-layer 3D tracking of human/objects. In this future investigation, we will provide contribution to model and recognize the state of scene and to analyse the behavior of small group using probabilistic approaches.
38
Thank you!