Computer vision for unmanned surface vehicles · Semantic segmentation graphical model (SSM) Observed measurement at i-th pixel. Unobserved pixel label. Prior distribution over pixel

Computer vision for unmanned surface vehicles

Matej Kristan

Visual Cognitive Systems LaboratoryFaculty of computer and information science

University of Ljubljana, Slovenia

Summer Course 2018 Ljubljana – Autonomous carsLjubljana, Slovenia | 12 Jul. 2018

Matej Kristan, [email protected], SSL-AC 2018

Research interests

1. Online density estimationKristan et al., IEEE SMCB 2013 ;Kristan et al., IVC2009; Kristan et al., PR2011 ; Narbutas et al., VC2017;

2. Deep structured networksTabernik et al., CVPR 2018 ; Tabernik et al., ICPR 2016 ; Tabernik et al., CVIU 2015 ; Kristan et al., SCIA 2013 ; Tabernik et al., ICVS 2013 ; Tabernik ICPR 2012

3. Robotic visionBovcon et al, IROS 218 ;Bovcon et al, RAS 2017; Uršič et al., IJRR 2017; Uršič et al., ICRA 2016 ; Mandeljc et al., ICRA 2016 ;Kristan et al., IEEE TCyb 2016;Uršič et al., IJRAS 2013 ;Kristan et al., IMAVIS 2013;Uršič et al., IROS 2012Skočaj et al., EPIROB 2010

4. Visual trackingLukežič et al., IJCV 2018 ; Čehovin et al., ICCV2017;Lukežič et al., CVPR 2017;Lukežič et al., IEEE TCyb 2017Kristan et al., IEEE TPAMI 2016 ;Čehovin et al., IEEE TIP 2016 ; Čehovin et al., WACV2016 ; Kristan et al., ICCV-W 2015 ; Kristan et al., ECCV-W 2014 ; Čehovin et al., IEEE TPAMI 2013 ; Kristan et al., ICCV-W 2013 ; Kristan et al., IEEE SMCB2010 ; Kristan et al., PR 2009 ; Kristan et al., CVIU 2009 ;

0. Industrial R&D

2/39


What are Unmanned surface vehicles (USV)DARPA ACTUV (2016), ~40m

MR Mariner 560 (2013), ~6m Electric USV, ~1-2m

Rolls-Royce concept cargo ship (2016)

3/39


Recently involved in two USV-related projects

Autonomous sailboat Autonomous electric boat

Member of steering committee As principal investigator (PI)

4/39


Autonomous sailboat (2016 -): uTransat challenge

• Mission: Build an unmanned self-powered sailboat that will cross the Atlantic!

• Challenge established in 2005 (ISAE, Toluz, France)

• Partners: Zavod404 and University of Ljubljana (FE, FMF, FPP, FRI, FS)

• Several local supporters (Ocean tec and Justin Yacht Design)

Small size: ≤ 2.4mExclusively wind-poweredNO human interventionSpecified starting and finish region (>5000km)Finish within 4 monthsHas to periodically report position via GPS

~50 ships have tried and failed

5/39

http://www.oceantec.eu/

http://www.justinyachtdesign.com/


uTransat’s about students!

• Shell/sails design, Actuators, Sensory systems, Communication, Power system

• Climate/weather analysis, AI navigation system, Public relations, Funding

6/39


It’s all coming together

• Initially ~100 students, 2 years later ~30 student

• Hardware done 𝛽, Software “done” 𝛽 – now integration starts 𝛽2

• Hopefully soon to see some real sea action!7/39


Actually, there’ll be two boats!

• Major difference in hull design and actuation (hydraulics instead of motors)

• The predicted corridor and AI planning kept the same

8/39


Unmanned surface vehicles (USV)

Project APSIS 2007:

• Small sized (~2m), “low”-cost electric USVs

• Safer renewable energy harvesting

• Hydroelectric plants maintenance (3D barim.)

• Green coastal communities

• Regular water and natural disaster control

Project ViAMaRo 2017-2020: [2 FTE/year; Partners FRI, FE, Harpha d.o.o., Robotina d.o.o.]

• Robust computer vision methods for autonomous water surface vehicles

• The project is focused on “Perception” for autonomy9/39


USV hardware specs

• Size: 2.3 x 0.7m

• Weight: 55 kg

• Drive: electic propulsor (±90°)

• Sensors: GPS (diff.), IMU, compass, US (distance ~7m),

sonar (depth), anemometer (wind), cameras

• Communication: radio(2.4GHz,narr. band)

• Control: NAV (ARM 9), Whitebox (i7 PC, 16GB SSD)

• Batteries: LiPo ; 4h autonomy

[ motor/control + electronics/telemetry + whitebox]

Max. speed 2.5 m/s

10/39


USV system scheme

NAV (ARM9)

• Auto-navigation types• Self-supervised PID• Manual control

WhiteboxCore i7, 3GHz

4 cores

UDP

2.4 GHz Narrow band

Control station

position: ….

velocity: ….

attitude: ….

wind: …

battery: …

depth: ….

range: …

video: ….

USB

1280x720(10fps)

11/39


• Map + sonar keep USV from running aground

USV partially autonomous

• Finely tuned control systems (Vranac&Mozetič, Harpha Sea d.o.o.)

NAV (ARM9)

~1.5m

12/39


• Problem: obstacles not present in maps

• Dynamic obstacles

Challenges: Static vs dynamic obstacles

NAV (ARM9)

~1.5m

13/39


Challenges: Types of sensors

• Laser rangers (e.g., LIDARs)

• Single plane detection (boat tilting)

• Potentially use stronger lasers + multiple planes

• Disturbances due to water spraying, wakes, foam

• Problems: weight, energy consumption, partially submerged objects

14/39


Challenges: Types of sensors

• Camera as energy-efficient information-rich sensor

• Full filed-of-view detection

• Challenge: what does the obstacle look like?

15/39


Which visual property constitutes an obstacle?

• Problem: Obstacles not visually consistent

• Common to all obstacles: Surrounded (or floating on) by water

• Problem: Water not visually consistent across views

Three semantic regionsBottom region is water

16/39


Semantic segmentation graphical model (SSM)

Observed measurementat i-th pixel.

Unobserved pixel label.

Prior distribution overpixel labels.

[Diplaros et al., TNN2007]

Efficient optimization derived by Kristan et al., IEEE TCYB 2016

Co

nd

itio

nal

pri

ors

Hyp

er-p

rio

rsdata term local consistency term

17/39


Application for obstacle detection

Fitting+labelling(core i7 single thread): ~70fps Matlab

18/39


Marine Obstacle Detection Dataset (MODD)

• MODD1 (monocular camera, 12 sequences) 1

• MODD2 (stereo, IMU, 28 sequences – 11675 frames) 2

• Different weather conditions and time of day

Gulf ofKoper

1Kristan, Sulić, Kovačič, Perš, Fast image-based obstacle detection from unmanned surface vehicles, IEEE TCyb 20162Bovcon, Mandeljc, Perš, Kristan. Stereo obstacle detection for unmanned surface vehicles by IMU-assisted semantic segmentation, RAS 2018

19/39


Marine Obstacle Detection Dataset (MODD)

• Obstacles and edge of water manually annotated

1Kristan, Sulić, Kovačič, Perš, Fast image-based obstacle detection from unmanned surface vehicles, IEEE TCyb 20162Bovcon, Mandeljc, Perš, Kristan. Stereo obstacle detection for unmanned surface vehicles by IMU-assisted semantic segmentation, RAS 2018

20/39


SSM segmentation examples

Kristan, Sulić, Kovačič, Perš, Fast image-based obstacle detection from unmanned surface vehicles, IEEE TCyb 2016

21/39


Drawbacks of SSM

• Relies solely on visual information

• Over/under estimates water edge

• Fails in presence of visual ambiguities

[Kristan et al., IEEE TCYB 2016]

IMUUse IMUfor horizonestimation!

22/39


Estimation of horizon line from IMU measurements

• Three differently oriented coordinate systemsFrom calibrationFrom IMU

. . .? ? ht

23/39


IMU-extended SSM (ISSM)

24/39


ISSM vs SSM

Bovcon, Perš, Kristan, Improving vision-based obstacle detection on USV using inertial sensor, ISPA201725/39


Drawbacks of ISSM

• A lot of false positive detections caused by

• sun flares on camera lens,

• sun glitter on the water surface,

• sea foam, …

[Bovcon et al., ISPA 2017]

Use stereofor consistencyverification!

26/39


ISSM with stereo extension: segmentation

• Enforce segmentation consistency in both images!

• Modification of the graphical model [Bovcon et. al, ISPA 2017]

• Compute disparity and align L/R image pixels

Bovcon and Kristan, Obstacle Detection for USVs by Joint Stereo-View Semantic Segmentation, IROS 2018

27/39


ISSM with stereo extension: appearance consistency

Bovcon and Kristan, Obstacle Detection for USVs by Joint Stereo-View Semantic Segmentation, IROS 2018

• Epipolar geometry of rectified stereo images

• NCC template matching with a high threshold

Filtered

Left image Right image

(15 false detections)

28/39


ISSM vs IeSSM

IeSSM:

• Reduced false positives

• Increased true positives

29/39


IeSSM visualization

Bovcon and Kristan, Obstacle Detection for USVs by Joint Stereo-View Semantic Segmentation, IROS 201830/39


Issues remain

• Segmentation fails when driving

towards the sun

• Plan to integrate compass into our

segmentation to predict sun position

• Reflections are a major problem

• Currently exploring “modern” segmentation

approaches – CNNs, e.g., UNet1

1Ronneberger, Fischer, Brox, U-Net: Convolutional Networks for Biomedical Image Segmentation, MICCAI 2015

31/39


• Water surface is approximately flat

• Detect obstacles by 3D outliers of a “plane model”

Detection by stereo reconstruction

32/39


• Disparity computation – compute 3D point cloud by triangulation

• Robust plane fitting (RANSAC)

Detection by stereo reconstruction

3D point cloud

Disparity map

33/39


IMU

Improving plane fitting & detection

• Stereo does not work without texture! – apply IMU constraints

• Apply semantic segmentation (SSM) to identify water pixels

• Verify detections over several time-steps to remove false positives

by 3D “fingerprints”

With SSM

With SSM

34/39


Stereo-based detection

• Close-range detection

works fairly well

• Problems:

• Obstacles > 20m away

• Low obstacles

• Ongoing work:

• A new dataset with

LIDAR included

Muhovič, Mandeljc, Bovcon, Kristan, Perš, Depth Fingerprinting for Obstacle Tracking using 3D Point Cloud, journal submission 2018

35/39


Other ongoing USV-related work

• Developing fast algorithms for long-term tracking1,2

• Camera motion simulators for testing trackers3

1Lukežič, Čehovin, Vojir, Matas, Kristan, FCLT -- A Fully-Correlational Long-Term Tracker, Arxiv 20182Lukežič, Vojir, Čehovin Zajc, Matas and Kristan, Discriminative Correlation Filter Tracker with Channel and Spatial Reliability, IJCV 20183Čehovin, Lukežič, Leonardis, Kristan, Beyond Standard Benchmarks: Parameterizing Performance Evaluation in Visual Object Tracking, ICCV 2017

36/39


Outlook and conclusions

• Two examples of autonomous robotic boats

• Our work on obstacle detection

• Details in papers:

• Several open issues remain:

• Stereo system failures in homogenous regions, reflections

• Driving towards the sun

• Obstacle tracking, re-identification, long-term tracking

1Kristan et al., IEEE TCyb 2016 ; 2Bovcon et al., ISPA2017 ; 3Bovcon et al., RAS 2018 ; 4Bovcon & Kristan , IROS 2018; 5Muhovič et al., prepub 2018 ; 6Lukežič et al., IJCV 2018 ; 7Lukežič et al. arxiv 2018; 8Čehovin et al., ICCV 2017

37/39


Online resources

http://www.vicos.si/Projects/Viamaro

MODD1, MODD2 + reference Matlab code

38/39

http://www.vicos.si/Projects/Viamaro


Thanks

• Janez Perš, Borja Bovcon, Rok Mandeljc, Jon Muhovič, Duško Vranac,

Dean Mozetič, Stanislav Kovačič, Vildana Sulić, Aljoša Žerjal

• Harpha Sea d.o.o.

• Robotina d.o.o.

• Rok Capuder, Zavod404

and uTransat teams

ViAMaRo is a basic research project

funded by Slovenian Research Agency

ARRS under code J2-817539/39

Documents

Computer vision for unmanned surface vehicles · Semantic segmentation graphical model (SSM) Observed measurement at i-th pixel. Unobserved pixel label. Prior distribution over pixel