Sensor Fusion Multi-Sensor Data Fusion

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Sensor FusionMulti-Sensor Data Fusion

Felix Riegler

8. Mai 2014

Felix Riegler 1/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Felix Riegler 2/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

1 Definition

2 Domains and properties

3 Examples

4 General data fusion methods

5 Stereo vision

6 Conclusion

Felix Riegler 3/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Definition

Felix Riegler 4/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Richardson and March - 1988

Fusion of Multisensor data.

Felix Riegler 5/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Hall - 1992

Multisensor data fusion seeks to combine data from multiplesensors to perform inferences that may not be possible from asingle sensor alone.

Felix Riegler 6/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Starr and Desforges - 1998

Data fusion is a process that combines data and knowledge fromdifferent sources with the aim of maximising the useful informationcontent, for improved reliability or discriminant capability, whileminimising the quantity of data ultimately retained.

Felix Riegler 7/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Breakdown

Input: sensor data from multiple sensors

Process: combining data

Goal: to get better and/or more reliable data

Felix Riegler 8/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Breakdown

Input: sensor data from multiple sensors

Process: combining data

Goal: to get better and/or more reliable data

Felix Riegler 8/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Breakdown

Input: sensor data from multiple sensors

Process: combining data

Goal: to get better and/or more reliable data

Felix Riegler 8/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Domains and properties

Felix Riegler 9/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Domains

Military

Localization/Detection

Navigation/Pathfinding

(Air-)Traffic control

Environment prediction

Robotics

....

Felix Riegler 10/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Domains

Military

Localization/Detection

Navigation/Pathfinding

(Air-)Traffic control

Environment prediction

Robotics

....

Felix Riegler 10/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Domains

Military

Localization/Detection

Navigation/Pathfinding

(Air-)Traffic control

Environment prediction

Robotics

....

Felix Riegler 10/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Domains

Military

Localization/Detection

Navigation/Pathfinding

(Air-)Traffic control

Environment prediction

Robotics

....

Felix Riegler 10/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Domains

Military

Localization/Detection

Navigation/Pathfinding

(Air-)Traffic control

Environment prediction

Robotics

....

Felix Riegler 10/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Domains

Military

Localization/Detection

Navigation/Pathfinding

(Air-)Traffic control

Environment prediction

Robotics

....

Felix Riegler 10/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Domains

Military

Localization/Detection

Navigation/Pathfinding

(Air-)Traffic control

Environment prediction

Robotics

....

Felix Riegler 10/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

3 Properties

Source:

homogeneous

heterogeneous

Goal:

reliability

new data

Representation in a system:

blackboard

module

Felix Riegler 11/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

3 Properties

Source:

homogeneous

heterogeneous

Goal:

reliability

new data

Representation in a system:

blackboard

module

Felix Riegler 11/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

3 Properties

Source:

homogeneous

heterogeneous

Goal:

reliability

new data

Representation in a system:

blackboard

module

Felix Riegler 11/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

3 Properties

Source:

homogeneous

heterogeneous

Goal:

reliability

new data

Representation in a system:

blackboard

module

Felix Riegler 11/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

3 Properties

Source:

homogeneous

heterogeneous

Goal:

reliability

new data

Representation in a system:

blackboard

module

Felix Riegler 11/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

3 Properties

Source:

homogeneous

heterogeneous

Goal:

reliability

new data

Representation in a system:

blackboard

module

Felix Riegler 11/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

3 Properties

Source:

homogeneous

heterogeneous

Goal:

reliability

new data

Representation in a system:

blackboard

module

Felix Riegler 11/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

3 Properties

Source:

homogeneous

heterogeneous

Goal:

reliability

new data

Representation in a system:

blackboard

module

Felix Riegler 11/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

3 Properties

Source:

homogeneous

heterogeneous

Goal:

reliability

new data

Representation in a system:

blackboard

module

Felix Riegler 11/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

3 Properties

Source:

homogeneous

heterogeneous

Goal:

reliability

new data

Representation in a system:

blackboard

module

Felix Riegler 11/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

3 Properties

Source:

homogeneous

heterogeneous

Goal:

reliability

new data

Representation in a system:

blackboard

module

Felix Riegler 11/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Examples

Felix Riegler 12/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Radar

emits electromagnetic waves and detects reflections

surveillance radar

secondary or active radar

+ fire control radar

heterogeneous; reliability and new data; blackboard

Felix Riegler 13/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Radar

emits electromagnetic waves and detects reflections

surveillance radar

secondary or active radar

+ fire control radar

heterogeneous; reliability and new data; blackboard

Felix Riegler 13/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Radar

emits electromagnetic waves and detects reflections

surveillance radar

secondary or active radar

+ fire control radar

heterogeneous; reliability and new data; blackboard

Felix Riegler 13/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Radar

emits electromagnetic waves and detects reflections

surveillance radar

secondary or active radar

+ fire control radar

heterogeneous; reliability and new data; blackboard

Felix Riegler 13/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Radar

emits electromagnetic waves and detects reflections

surveillance radar

secondary or active radar

+ fire control radar

heterogeneous; reliability and new data; blackboard

Felix Riegler 13/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

sound locatisation

two microphones

homogeneous

new data

module

Felix Riegler 14/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

sound locatisation

two microphones

homogeneous

new data

module

Felix Riegler 14/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

sound locatisation

two microphones

homogeneous

new data

module

Felix Riegler 14/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

sound locatisation

two microphones

homogeneous

new data

module

Felix Riegler 14/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

camera + infrared projector

e.g. Kinect

heterogeneous

new data

?

Felix Riegler 15/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

camera + infrared projector

e.g. Kinect

heterogeneous

new data

?

Felix Riegler 15/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

camera + infrared projector

e.g. Kinect

heterogeneous

new data

?

Felix Riegler 15/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

camera + infrared projector

e.g. Kinect

heterogeneous

new data

?

Felix Riegler 15/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

stereo vision

two cameras

homogeneous

new data

module

Felix Riegler 16/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

stereo vision

two cameras

homogeneous

new data

module

Felix Riegler 16/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

stereo vision

two cameras

homogeneous

new data

module

Felix Riegler 16/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

stereo vision

two cameras

homogeneous

new data

module

Felix Riegler 16/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

General data fusion methods

Felix Riegler 17/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

General data fusion methods

Bayesian network

Kalman filter

Fuzzy logic

Monte Carlo methods

Dempster–Shafer theory

Felix Riegler 18/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

General data fusion methods

Bayesian network

Kalman filter

Fuzzy logic

Monte Carlo methods

Dempster–Shafer theory

Felix Riegler 18/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

General data fusion methods

Bayesian network

Kalman filter

Fuzzy logic

Monte Carlo methods

Dempster–Shafer theory

Felix Riegler 18/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

General data fusion methods

Bayesian network

Kalman filter

Fuzzy logic

Monte Carlo methods

Dempster–Shafer theory

Felix Riegler 18/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

General data fusion methods

Bayesian network

Kalman filter

Fuzzy logic

Monte Carlo methods

Dempster–Shafer theory

Felix Riegler 18/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Bayesian network

Design on abstract or raw data?

Felix Riegler 19/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Kalman Filter

tracking, localisation, navigation

suited for many different (sensor-)inputs

recursive state estimation, thus

simple and efficient in many cases

x(t|t− 1) - state at time t with data t-1

P (t|t) - covariant - error estimate

Felix Riegler 20/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Kalman Filter

tracking, localisation, navigation

suited for many different (sensor-)inputs

recursive state estimation, thus

simple and efficient in many cases

x(t|t− 1) - state at time t with data t-1

P (t|t) - covariant - error estimate

Felix Riegler 20/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Kalman Filter

tracking, localisation, navigation

suited for many different (sensor-)inputs

recursive state estimation, thus

simple and efficient in many cases

x(t|t− 1) - state at time t with data t-1

P (t|t) - covariant - error estimate

Felix Riegler 20/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Kalman Filter

tracking, localisation, navigation

suited for many different (sensor-)inputs

recursive state estimation, thus

simple and efficient in many cases

x(t|t− 1) - state at time t with data t-1

P (t|t) - covariant - error estimate

Felix Riegler 20/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Kalman Filter

tracking, localisation, navigation

suited for many different (sensor-)inputs

recursive state estimation, thus

simple and efficient in many cases

x(t|t− 1) - state at time t with data t-1

P (t|t) - covariant - error estimate

Felix Riegler 20/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Kalman Filter

tracking, localisation, navigation

suited for many different (sensor-)inputs

recursive state estimation, thus

simple and efficient in many cases

x(t|t− 1) - state at time t with data t-1

P (t|t) - covariant - error estimate

Felix Riegler 20/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Kalman Filter in a nutshell

Felix Riegler 21/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Fuzzy logic

many-value logic

f → [0, 1]

degree of certainty

especially use in threshold controlled systems

Felix Riegler 22/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Fuzzy logic

many-value logic

f → [0, 1]

degree of certainty

especially use in threshold controlled systems

Felix Riegler 22/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Fuzzy logic

many-value logic

f → [0, 1]

degree of certainty

especially use in threshold controlled systems

Felix Riegler 22/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Fuzzy logic

many-value logic

f → [0, 1]

degree of certainty

especially use in threshold controlled systems

Felix Riegler 22/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Stereo Vision

Felix Riegler 23/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Stereo Vision

stereopsis

PR2 has 2 stereo vision systems

another way for depth recognition

obviously 2 cameras capturing (more or less) the same picture

calibration is problematic

Felix Riegler 24/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Stereo Vision

stereopsis

PR2 has 2 stereo vision systems

another way for depth recognition

obviously 2 cameras capturing (more or less) the same picture

calibration is problematic

Felix Riegler 24/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Stereo Vision

stereopsis

PR2 has 2 stereo vision systems

another way for depth recognition

obviously 2 cameras capturing (more or less) the same picture

calibration is problematic

Felix Riegler 24/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Stereo Vision

stereopsis

PR2 has 2 stereo vision systems

another way for depth recognition

obviously 2 cameras capturing (more or less) the same picture

calibration is problematic

Felix Riegler 24/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Stereo Vision

stereopsis

PR2 has 2 stereo vision systems

another way for depth recognition

obviously 2 cameras capturing (more or less) the same picture

calibration is problematic

Felix Riegler 24/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Anaglyph 3D

Felix Riegler 25/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

2 pictures

Felix Riegler 26/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

3D Reconstruction

Felix Riegler 27/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Standard Geometry

z =b ∗ focallengthxcamL − xcamR

x =xcamL ∗ z

focallength

y =ycamL ∗ zfocalwidth

Felix Riegler 28/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Triangulation

general case

cameras need to be thoroughly calibrated

both intrinsic and extrinsic matrix have to be known

(lens properties + position in relation to the global coordinatesystem)

Felix Riegler 29/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Triangulation

Felix Riegler 30/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Traingulation Calculation

z1 ∗ point1 = camMatrix1 ∗ pointreal

z2 ∗ point2 = camMatrix2 ∗ pointreal

Felix Riegler 31/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

1 image + depth information

Felix Riegler 32/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Conclusion

Felix Riegler 33/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Sensor fusion

combining (multiple) sensor data to get better and/or morereliable data

core ability for humans

used in many domains

(humanoid)robotics

Felix Riegler 34/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Sensor fusion

combining (multiple) sensor data to get better and/or morereliable data

core ability for humans

used in many domains

(humanoid)robotics

Felix Riegler 34/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Sensor fusion

combining (multiple) sensor data to get better and/or morereliable data

core ability for humans

used in many domains

(humanoid)robotics

Felix Riegler 34/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Sensor fusion

combining (multiple) sensor data to get better and/or morereliable data

core ability for humans

used in many domains

(humanoid)robotics

Felix Riegler 34/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Algorithms for sensor fusion

Bayesian networks

Kalman filter

Fuzzy logic

stereo vision - depth recognition

Felix Riegler 35/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Algorithms for sensor fusion

Bayesian networks

Kalman filter

Fuzzy logic

stereo vision - depth recognition

Felix Riegler 35/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Algorithms for sensor fusion

Bayesian networks

Kalman filter

Fuzzy logic

stereo vision - depth recognition

Felix Riegler 35/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Algorithms for sensor fusion

Bayesian networks

Kalman filter

Fuzzy logic

stereo vision - depth recognition

Felix Riegler 35/36

DefinitionDomains and properties

ExamplesGeneral data fusion methods

Stereo visionConclusion

Thank you,Questions?

Felix Riegler 36/36