Senses for Safety. - semieurope.omnibooksonline.comsemieurope.omnibooksonline.com/2015/semicon_europa/Imaging/04... · › Eyes On/Off Road › Adaptive Warnings ... Gesture Detection

Senses for Safety.

Driver assistance systems help save lives.

Camera systems for ADAS applications

Dr. Martin Punke

Chassis & Safety | Advanced Driver Assistance Systems

Public

BU ADAS 15 September 2015

3 Dr. Martin Punke © Continental AG

Agenda

Continental 1 Market 2 Challenges: Today 3 Solutions 4 Challenges: Future 5

Public

BU ADAS

Chassis & Safety 22%

Powertrain 19%

Interior 20%

Tires 28%

ContiTech 11% › Since 1871 with headquarters in Hanover, Germany

› Sales of €34.5 billion

› 189,168 employees worldwide

› 317 locations in 50 countries

Continental Corporation Overview 2014

15 September 2015


Sales by division in %

Status: December 31, 2014

Public

BU ADAS

› First automotive supplier in the world to receive a license in Nevada to test automated

driving on public roads

› Supplier of the stereo and mono camera for the current Daimler C-/E-/S-Class supporting

features like Magic Body Control

› Project running with BMW to develop prototypes for highly automated driving

Continental developments

15 September 2015


Public

BU ADAS

ADAS Product Segments

15 September 2015


Functions:

› Adaptive Cruise Control

› Emergency Brake Assist

› Forward Collision Warning

› Blind Spot Detection

› Lane Change Assist

› Rear Cross Traffic Alert

Functions:

› Emergency Brake Assist - City

› Emergency Brake Assist - Urban

› Emergency Brake Assist - Pedestrian

› Crash Imminent Braking

Radar Camera Lidar

Products:

› Long Range Radar (77GHz)

› Short Range Radar (24GHz)

Functions:

› Intelligent Headlamp Control

› Lane Departure Warning

› Lane Keeping Support

› Traffic Sign Assist

› Forward Collision Warning

› Emergency Brake Assist

(Stereo Camera)

Products:

› Mono Camera

› Stereo Camera

Products:

› Short Range Lidar Sensor

› SRLCam (Lidar-Camera Fusion)

SRR

LRR

Public

BU ADAS

ADAS Product Segments

15 September 2015


Functions:

› 2D Surround View

› 3D Bowl Surround View

› Automated Parking

› Trailer Reversing

› Auto Trailer Hitching

› Object/Pedestrian Detection with Warning

› Blind Spot Warning

› Lane Departure Warning

Functions:

› Object Detection with Warning


› Lane Detection

Surround View Rear View Mirror Replacement

Products:

› Surround View Camera

› Surround View ECU

Functions:

› Parking Support (Warning)

› Trailer Reversing

› Auto Trailer Hitching

› Object/Pedestrian Detection with Warning/Braking


Products:

› Intelligent Rear View Camera

Products:

› Mirror Camera

› Mirror Camera - Truck

Public

BU ADAS

Continental Information & Driver HMI Product Segment

15 September 2015


Functions:

› Drowsiness Detection

› Attentiveness (based on Head Pose)

› Eyes On/Off Road

› Adaptive Warnings

Functions:

› 2D Gesture recognition as HMI input, QVGA

resolution , >10 Gesture recognition

› 3D Gesture recognition as HMI input option

Driver Camera (mono) Driver Camera (Multi) Gesture Camera (2D/3D)

Products:

› Mono Camera

Functions:

› Gaze Sector Analyses (180 degree)

› Eye Gaze Interaction

› Eye Opening Gap Measurement

(Input to MicroSleep Detection)

Products:

› Multi Camera System

Products:

› Mono Camera (2D/3D)

Public


9 Dr. Martin Punke, © Continental AG

Agenda


Public

BU ADAS

Driver Assistance 2025 360° predictive safety in all vehicle segments

15 September 2015


Public

BU ADAS

Automotive Camera Applications

15 September 2015


Sensing Camera

Mirror replacement

Viewing Camera

Surround View

Drive Recorder

Interior Driver Camera

Front View ADAS

Blind Spot Camera

Rear View camera

Gesture Detection

Vision System (in-car)

Public

BU ADAS

Business Trend Increasing Vehicle Penetration

15 September 2015


Midsize Cars

Premium Cars

Compact Cars

Technology

Transfer

Public

BU ADAS

Automated Driving for Accident-Free Driving Roadmap & History

15 September 2015

13 © Continental AG

Partially Automated (2016) – Highly Automated (2020) – Fully Automated (> 2025)

AEB City

< 50 kph

Parking

Assist

Automated

Parking

AEB

> 60 kph Driver

Distraction Highway Assist

Stop&Go, Cruising, 0-130km/h

Construction Site

(high-automated)

City

Stop&Go

Highway

(high automated)

Remote

parking

Construction Site

Assist

• ACC Longitudinal/Lateral

• Traffic Jam Assistant

(highway)

• Lateral Collision Avoidance

• Emergency Steering

Assistant

2015-2016

AEB

Pedestrian

Inter-Urban

Autonomous

HISTORY

2002 – PRORETA 1

2006 – PRORETA 2

2007 – DARPA Urban Challenge

2008 – HaveIT

2009 – KoFAS

2010 – interactIVe

2011 – PRORETA 3

2012 – Nevada License

2015 2020 > 2025

Public

BU ADAS

Global Development of Active Safety Regulations & Ratings

14

decided

Expected by

Continental

2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020

AEB & LDWS

Commercial vehicles & buses

AEB & LDWS

Commercial

vehicles & buses

ESC

Light vehicles

ESC

Passenger vehicles

FCW/LDWS

AEB & LDWS

NCAP star-rating

AEB & LDWS

NCAP star-rating

RVS

2016 (10%) -

2018 (100%)

RVS

CIBS & DBS

V2X Device AEB

ESC

Passenger

vehicles

SAS & AEB

Pedestrian

NCAP star-rating

RDP & AEB

Intersection

NCAP star-rating

Le

gis

latio

n

Con

su

me

r

Te

st

Le

gis

latio

n

Con

su

me

r

Te

st

AEB

Passenger vehicles

ESC = Electronic Stability Control; FCWS = Forward Collision Warning System; LDWS = Lane Departure Warning System;

AEBS = Advanced Emergency Braking System; CIBS = Crash Imminent Braking System, DBS = Dynamic Brake Support,

RVS = Rearview Video System, SAS = Speed Assist System, RDP = Road Departure Protection

NCAP = New Car Assessment Program; EC = European Commission; NHTSA = National Highway Traffic Safety

Administration (USA); MLIT = Ministry of Land, Infrastructure, Transport and Tourism (Japan); MLTM = Ministry of Land,

Transport and Marine Affairs (South Korea), IIHS = Insurance Institute for Highway Safety

Public



Agenda

Continental 1 Market 2 Challenges: Today 3

Camera module challenges 3.1

Algorithmic challenges 3.2

Solutions 4 Challenges: Future 5

Public

BU ADAS

Todays ADAS Camera functions

15 September 2015


Light Assist

Lane Assist

Traffic Sign Assist

Mono Object detection - Pedestrians

- Vehicles

Path profile estimation

Stereo Object detection - Pedestrians

- Vehicles

- Generic objects

Public

BU ADAS

Challenges

15 September 2015


System

(Processing, mechanical / optical / electronic integration,

robustness, functional safety, validation, cost)

Camera Module

(Imager, Optics, Packaging)

Machine Vision Algorithms

(Feature detection + classification)

Public

BU ADAS

Camera module challenges

15 September 2015


Public

BU ADAS

› Light Assist: High dynamic range, color discrimination

› Lane Assist: High dynamic range, good SNR at night, color discrimination

› Traffic Sign Assist: High sensitivity / low exposure time, high resolution, colors

› Object detection: High sensitivity, high resolution, high frame rate (+stereo)

› Path profile estimation: High sensitivity, high resolution, high frame rate (stereo)

Challenges for the camera module / system

15 September 2015


Public

BU ADAS

› HDR artefacts all current HDR schemes create

artefacts

› Rolling shutter effects distorted images /

artefacts in LED signs

› LED flicker influence (LED signs, head/tail lights)

can lead to inconsistent signals

Real world technology (image sensor) challenges

15 September 2015


Public

BU ADAS

Algorithmic challenges

15 September 2015


Public

BU ADAS

› Separation of moving and static objects (e.g. street lamps, reflectors)

› Reflectors / reflections Low rate of false positives

› Pulsed light sources

› White balance

Light Assist – Technical Challenges

15 September 2015


Public

BU ADAS

› A lot of structures in the vehicle's environment

look like lane markers

› Finding the correct lane markers is the main

challenge

› Weather conditions (snow, rain, sun glare)

› Variety of lane markers (e.g. Bott’s dots)

Lane Detection – Technical Challenges

15 September 2015


Snow covered surfaces

Top of the concrete wall

looks like a lane marker

Snow covered lane markers

Public

BU ADAS

Traffic sign recognition – Technical Challenges

operating conditions (weather, viewing angle, dirt, sun, snow,...)

variability due to differences between countries

Examples: United Arab Emirates / United States

15 September 2015


Public

BU ADAS

› Wide variety of “normal” objects (vehicles, pedestrians, bicycles)

› Endless variety of generic objects (animals, hazards etc.)

Object Detection – Technical Challenges

15 September 2015


Public

BU ADAS

Pedestrian Recognition – Technical Challenges

15 September 2015


Lighting conditions

Change of pose

Occlusion

Intention Estimation of Pedestrians

Public



Agenda

Continental 1 Market 2 Challenges: Today 3 Solutions 4

Algorithms 4.1

Color 4.2

Stereo vision 4.3

Sensor Fusion 4.4

Challenges: Future 5

Public

BU ADAS

› Clever algorithms

› Color

› Stereo vision

› HDR imager

› Low F# optics

› LED Flicker compensation

› Sensor fusion

Solutions

15 September 2015


Public

BU ADAS

Algorithms

15 September 2015


Public

BU ADAS

› Feature detection (edges, corners, lines, circles, light blobs, objects, etc.)

› Histogram of Oriented Gradients (HOG) pedestrian detection

› Hough transformation detection of geometrical objects (lines, circles) lanes, traffic signs

› Segmentation light intensity, color light assist

› Optical flow motion pedestrian and vehicle detection

› Tracking

› Classification

› Environment model

› 3D construction

› Stereo vision

› Structure from Motion

Basic principles

15 September 2015


Public

BU ADAS

Example: Lane detection

Color processing /

edge detection

Line segmentation

Feature accsociation & tracking

Road model generation

15 September 2015


Public

BU ADAS

› Visualization of relative movement between observer and environment

› Can be used for Structure from Motion approaches

Optical flow

15 September 2015


Public

BU ADAS

Color

15 September 2015


Public

BU ADAS

› Colors of lane markings have country- and situation-specific meaning

› Colors signal dangerous curves, car pool lanes, bikeways

› Colors signal the validity of markers, overrule white markers

› Colors can be used to detect road works

Color is necessary to determine semantics

15 September 2015


Public

BU ADAS

› Colors of lane markings have country- and situation-specific meaning

› Colors signal dangerous curves, car pool lanes, bikeways

› Colors signal the validity of markers, overrule white markers

› Colors can be used to detect road works

Color is necessary to determine semantics

15 September 2015


Public

BU ADAS

Stereo vision

15 September 2015


Public

BU ADAS

Features:

› 6D-Object detection – 3D spatial + 3D flow

› Generic object detection – from road boundaries to pedestrians

› Redundant object detection – improved functional safety

Functions:

› Forward Collision Warning (FCW)

› Lane Departure Warning (LDW)

› Intelligent Headlamp Control (IHC)

› Traffic Sign Assist (TSA)

› Adaptive Cruise Control (ACC) + Stop&Go (speed limited)

› Emergency Brake Assist (EBA) incl. pedestrian

› Road Profile Detection (RPD)

Multi Function Stereo Camera

15 September 2015


Public

BU ADAS

Sensor fusion

15 September 2015


Public

BU ADAS

Features:

› Compact One Box Design with a fusion of the camera and laser sensor

technology

› Color CMOS imager and 905nm Lidar

Functions:

› EBA up to 80kph (Fusion of Camera & Lidar)

› Object recognition, e.g. for the Forward Collision Warning (FCW)

› Lane Departure Warning (LDW)

› Intelligent Headlamp Control (IHC)

› Traffic Sign Assist (TSA)

› Pedestrian Detection (classification)

Multi Function Camera with Lidar

15 September 2015


Public



Agenda


Public

BU ADAS

› Higher resolution

› Better detection in far distance / wider FoV

› Higher night sensitivity

› General performance improvements

› NCAP scenarios

› Wider FoV

› Pedestrian detection

› Bicycle detection

› Higher availability / Redundant systems

› Highly automated driving

Future system requirements

15 September 2015


Regulatory needs

Performance + features

Highly automated

driving

Public

BU ADAS

Future image sensor requirements

15 September 2015


- Sensitivity

- Resolution

- Frame rate

- Power consumption

- Cost

Increased ASIL requirements

Improved HDR schemes

Flicker compensation

Noise compensation

Documents

Senses for Safety. - semieurope.omnibooksonline.comsemieurope.omnibooksonline.com/2015/semicon_europa/Imaging/04... · › Eyes On/Off Road › Adaptive Warnings ... Gesture Detection