Autonomous Vehicles –

Sensors and Limitations

Guy Satat

Computational Imaging for Self-Driving Vehicles @ CVPR 2018

Image source: Cognata

Ideal Sensor

Good resolution

(small voxels)

Recovers depth

Good spectral

resolution

Any WeatherDay / Night

Cost

Visible spectrum

360°

Coverage

Recovers speed

Real time

Short Exposure

Sees low reflectivity

targets

© 2018 Guy Satat. All Rights Reserved.

Common Sensors on AVs

• Ultrasonic

• RGB Camera

• Stereo Camera

• LiDAR

• RADAR

Image source: https://www.wired.com/story/self-driving-cars-perception-humans/© 2018 Guy Satat. All Rights Reserved.

RADAR / Ultrasonic / LiDAR

Source

Detector

Target

𝑑 =1

𝑐

𝑇

2

© 2018 Guy Satat. All Rights Reserved.

Ultrasonic Sensor

• Short range (~cm – few meters)

• Low cost

• Mostly for parking assist

• Being replaced by RADAR

Image source: Wikipedia© 2018 Guy Satat. All Rights Reserved.

RADAR

Short Range Long Range

24 GHz 77 GHz

Image source: Wikipedia

© 2018 Guy Satat. All Rights Reserved.

RADAR

Short Range Long Range

24 GHz 77 GHz

Smaller antennas More accurate

Image source: Wikipedia

© 2018 Guy Satat. All Rights Reserved.

RADAR

Short Range Long Range

24 GHz 77 GHz

Smaller antennas More accurate

360 (complements blind spots) Forward facing

Image source: Wikipedia

© 2018 Guy Satat. All Rights Reserved.

RADAR

Short Range Long Range

24 GHz 77 GHz

Smaller antennas More accurate

360 (complements blind spots) Forward facing

Used for:• Lane change assist• Parking assist• Cross traffic monitoring

Used for:• Break assist• Adaptive cruise control

Image source: Wikipedia

© 2018 Guy Satat. All Rights Reserved.

LiDAR

• Raster Scan• Multiple beams / detectors

• MEMS

• Solid state

• Limited optical budget• Eye safety

• Usually NIR (905 / 1550nm)

• Cost

• Low reflectivity targets

• Interference

Image source: Wikipedia

Image source: Wikipedia© 2018 Guy Satat. All Rights Reserved.

SPAD Pixel

Time

Pixel𝑇

1

2

2

3

4

3

2

1

1

2

© 2018 Guy Satat. All Rights Reserved.

PixelPixel

SPAD Camera

Pixel

PixelPixel

Pixel

PixelPixel

Pixel

32x32 pixels

© 2018 Guy Satat. All Rights Reserved.

Depth Sensing

© 2018 Guy Satat. All Rights Reserved.

Camera

• 360 / Rear • Shorter range

• Front facing• Long range

• Traffic lights / signs

• Challenges• Dynamic range: direct sunlight / tunnels

• Night time

Image source: Wikipedia

© 2018 Guy Satat. All Rights Reserved.

Radar

Ultrasonic

Camera

LiDAR

Classification Localization Night Availability Any Weather

© 2018 Guy Satat. All Rights Reserved.

X rays UV IR Microwave Radio Waves

Visible

Wavelength

Visible Light:

• Good resolution (compared to RF)

• Optical contrast

• Non-ionizing radiation

Imaging Across the EM Spectrum

Wavelengths:

• EM Visible ~𝜇𝑚

• EM Radio ~𝑐𝑚

• Ultrasound: > 𝑐𝑚© 2018 Guy Satat. All Rights Reserved.

Wildlife Rehabilitation Center of Minnesota

Optical Contrast

© 2018 Guy Satat. All Rights Reserved.

Other Sensors

• IR / Thermal cameras

• Time gating

• Stereo Cameras

• mmWave RADAR

• Event cameras

Image source: Wikipedia

© 2018 Guy Satat. All Rights Reserved.

Sensor Fusion

RADAR

Perceived Environment

Drive Control

Camera

Perceived Environment

Drive Control

LiDAR

Perceived Environment

Drive Control

Fusion

LiDAR RADAR Camera

Fusion

Drive Control

Perceived Environment

© 2018 Guy Satat. All Rights Reserved.

Learning to Sense

• What sensors do we really need?

• How does sensors spec affect end-to-end performance

Hardware

Algorithms

Computational imaging

© 2018 Guy Satat. All Rights Reserved.

Key Takeaways

• Different sensors for:• Complement limitations

• Different applications

• Ultrasonic, RADAR, LIDAR, Camera

• Sensor Fusion

Images source: Wikipedia© 2018 Guy Satat. All Rights Reserved.