MIN-FakultätFachbereich Informatik

Fusing Camera and LiDAR to Detect andRecognize Motion

Lukas Wendt

Universität HamburgFakultät für Mathematik, Informatik und NaturwissenschaftenFachbereich InformatikTechnische Aspekte Multimodaler Systeme

12. November 2018

L. Wendt – Fusion Camera/LiDAR 1 / 27

ContentIntroduction Sensor Fusion Motion Conclusion & Future Works References Literatur

1. IntroductionCameraLiDARLiDAR CalculationMotion Detection & Recognition

2. Sensor FusionCompetitiveComplementaryCooperativeDemoFusion

3. MotionMovementRelative to Absolute

4. Conclusion & Future Works5. ReferencesL. Wendt – Fusion Camera/LiDAR 2 / 27

Introduction / MotivationIntroduction Sensor Fusion Motion Conclusion & Future Works References Literatur

I Autonomous Robots/CarsI Multiple Sensors

I CameraI LiDARI ...

I Sensor FusionI Complete View of the WorldI combining advantages of dif. SensorsI Motion Planing

L. Wendt – Fusion Camera/LiDAR 3 / 27

CameraIntroduction Sensor Fusion Motion Conclusion & Future Works References Literatur

I advantageI cheapI passiveI smallI high resolution

I disadvantageI no depth InformationsI fog / rainI intensive Computing

L. Wendt – Fusion Camera/LiDAR 4 / 27

LiDARIntroduction Sensor Fusion Motion Conclusion & Future Works References Literatur

I LiDAR (Light detection and Ranging)I LaserI SensorI advantage

I high distanceI depth InformationI density Information

I disadvantageI expensiveI no color InformationsI active

L. Wendt – Fusion Camera/LiDAR 5 / 27

LiDAR CalculationIntroduction Sensor Fusion Motion Conclusion & Future Works References Literatur

I Calculation Distance

LiDAR Distance

I d = c·t2

L. Wendt – Fusion Camera/LiDAR 6 / 27

LiDAR Calculation (cont.)Introduction Sensor Fusion Motion Conclusion & Future Works References Literatur

I Calculation Density

LiDAR Density

I A = AmplitudeI higher Amplitude = higher Density

L. Wendt – Fusion Camera/LiDAR 7 / 27

Motion Detection & RecognitionIntroduction Sensor Fusion Motion Conclusion & Future Works References Literatur

I Motion DetectionI Detects if an object is moving

I Motion RecognitionI Detects which moving a Object doing

I Gesture Example

L. Wendt – Fusion Camera/LiDAR 8 / 27

Sensor FusionIntroduction Sensor Fusion Motion Conclusion & Future Works References Literatur

I Combination of Sensor DataI RedundancyI Same or different SensorsI Frequency or Resolution loose

Definition„is the combining of sensory data ... so the resulting information isin some sense better than would be possible when these sourceswere used individually“a

aElmenreich 2001.

L. Wendt – Fusion Camera/LiDAR 9 / 27

CompetitiveIntroduction Sensor Fusion Motion Conclusion & Future Works References Literatur

Sensor Fusion Competitive

I Airplane →Redundancy

L. Wendt – Fusion Camera/LiDAR 10 / 27

ComplementaryIntroduction Sensor Fusion Motion Conclusion & Future Works References Literatur

Sensor Fusion Complementary

I Camera in the Front/Back

L. Wendt – Fusion Camera/LiDAR 11 / 27

CooperativeIntroduction Sensor Fusion Motion Conclusion & Future Works References Literatur

Sensor Fusion Cooperative

I LiDAR Cameras Systems in Cars/RobotsI 2 Sensors to increase accuracy

L. Wendt – Fusion Camera/LiDAR 12 / 27

A Short Video Demo

L. Wendt – Fusion Camera/LiDAR 13 / 27

FusionIntroduction Sensor Fusion Motion Conclusion & Future Works References Literatur

Lidar View

Camera View

Fused View

L. Wendt – Fusion Camera/LiDAR 14 / 27

Fusion (cont.)Introduction Sensor Fusion Motion Conclusion & Future Works References Literatur

I First we need to calibrate

Side view of the sensor setup1

1Silva, Roche und Kondoz 2018b.L. Wendt – Fusion Camera/LiDAR 15 / 27

MovementIntroduction Sensor Fusion Motion Conclusion & Future Works References Literatur

First Picture

Second Picture

L. Wendt – Fusion Camera/LiDAR 16 / 27

Movement (cont.)Introduction Sensor Fusion Motion Conclusion & Future Works References Literatur

Colliding Picture

I Possible DirectionI Maybe complete WrongI adding LiDAR Data

L. Wendt – Fusion Camera/LiDAR 17 / 27

Movement (cont.)Introduction Sensor Fusion Motion Conclusion & Future Works References Literatur

Colliding Picture

I represented with a color transitionI green = 0 Speed, blue = negative Speed , red = positive Speed

I speed on AxisI Movement between object

L. Wendt – Fusion Camera/LiDAR 18 / 27

Movement (cont.)Introduction Sensor Fusion Motion Conclusion & Future Works References Literatur

I trigonometric functionsI Arithmetic

Angle calculation Sphere First

L. Wendt – Fusion Camera/LiDAR 19 / 27

Movement (cont.)Introduction Sensor Fusion Motion Conclusion & Future Works References Literatur

Angle calculation Sphere

L. Wendt – Fusion Camera/LiDAR 20 / 27

Movement (cont.)Introduction Sensor Fusion Motion Conclusion & Future Works References Literatur

Angle calculation Sphere Second

L. Wendt – Fusion Camera/LiDAR 21 / 27

Movement (cont.)Introduction Sensor Fusion Motion Conclusion & Future Works References Literatur

Calcualte Points in Cordinate System

L. Wendt – Fusion Camera/LiDAR 22 / 27

Movement (cont.)Introduction Sensor Fusion Motion Conclusion & Future Works References Literatur

I Plot in a GridI (0/0) is the Lidar/Camera Position

Movement between Picture 1 and Picture 2

L. Wendt – Fusion Camera/LiDAR 23 / 27

ConclusionIntroduction Sensor Fusion Motion Conclusion & Future Works References Literatur

I More complete picture of the environmentI Problems with different Frequencies, Resolutions ..I LiDARs are expensiveI We can build a 3D environmentI Possibly Something data lostI Accurate the Interpolation with higher PolynomsI Tests with a real LiDARI exact Positions of detected Object can calculateI calculation are Simple trigonometric and normal arithmetic

L. Wendt – Fusion Camera/LiDAR 24 / 27

QuestionsIntroduction Sensor Fusion Motion Conclusion & Future Works References Literatur

I Thanks for your attentionI Questions ?

L. Wendt – Fusion Camera/LiDAR 25 / 27

ReferencesIntroduction Sensor Fusion Motion Conclusion & Future Works References Literatur

Cho, H. u. a. (2014). “A multi-sensor fusion system for movingobject detection and tracking in urban driving environments”.In: 2014 IEEE International Conference on Robotics andAutomation (ICRA), S. 1836–1843. doi:10.1109/ICRA.2014.6907100.

Elmenreich, Wilfried (2001). An Introduction to Sensor Fusion.Research Report 47/2001. Treitlstr. 1-3/182-1, 1040 Vienna,Austria: Technische Universität Wien, Institut für TechnischeInformatik. (Besucht am 06. 11. 2018).

ouster (2018a). https://www.ouster.io/. (Besucht am06. 11. 2018).

– (2018b). https://www.youtube.com/watch?v=X7TljH5x2kE.(Besucht am 06. 11. 2018).

L. Wendt – Fusion Camera/LiDAR 26 / 27

References (cont.)Introduction Sensor Fusion Motion Conclusion & Future Works References Literatur

Silva, Varuna De, Jamie Roche und Ahmet M. Kondoz (2018a).“Robust Fusion of LiDAR and Wide-Angle Camera Data forAutonomous Mobile Robots”. In: Sensors. (Besucht am06. 11. 2018).

– (2018b). “Robust Fusion of LiDAR and Wide-Angle CameraData for Autonomous Mobile Robots”. In: Sensors. (Besuchtam 06. 11. 2018).

L. Wendt – Fusion Camera/LiDAR 27 / 27