IEEE-SA P2020

The IEEE P2020 working group on automotive image quality

December 3rd, 2019

Bastian BaumgartAutonomous Intelligent Driving GmbH

On behalf IEEE P2020 Working Group

P2020 - Defining Image Quality

Algorithmic Performance (AP)Image Quality (IQ)

… influence „this“?How does „this“…

P2020

How does P2020 relate to MaaS?Maas systems will operate primarily autonomousSAE level responsibility on “system provider” (at least for a period of time) Transition of responsibility from Driver to MaaS

SAE Level 1+2 Level 4+5

Scene Driver Car

Scene Driver Car

ADAS

Driver alone, w/o ADAS

Self acting, supported by ADAS

Scene ADS Car

Driver?

ADS drives in known areas; driver to take over?

Scene ADS Car

ADS drives alone everywhere

P2020

The P2020https://site.ieee.org/sagroups-2020/

IEEE P2020

IEEE P2020 Sub Groups

IEEE-SA P2020Working group

[SG #0] – IQ requirements/specification standards

[SG #1] – LED flicker standards

[SG #2] – IQ for viewing standards

[SG #3] – IQ for computer vision standards

Timeline- Draft text by the end of 2019- Comment period until 16th Jan

2020- First version of the standard to be

expected by end 2020

P2020 - Overview & Long Term Objectives

• The next generation of cars will be multi-sensor, multi-modal, multi-camera, AI platforms. The key driver for this dramatic evolution in our vehicles is to increase safety.

• While cameras are crucial for a vehicle to sense and perceive its surroundings, to day there had not been a consistent approach in the automotive industry to measure image quality.

• Existing standards are not covering the needs of automotive imaging:- IEEE-SA P1858 Camera Phone Image Quality (CPIQ) working group - EMVA1288- ISO12233

• Automotive imaging imposes unique challenges due to its varied and distinct landscape of different setups (fisheye, multi-camera, HDR, temperature range, …), which are not adequately addressed in the existing standards.

• Therefore, the IEEE-SA P2020 working group has set the goal of defining the relevant metrics and KPIs for automotive image quality, enabling customers and suppliers to efficiently define, measure and communicate image quality of their systems.

P2020 - Overview & Long Term Objectives #2

• 5.2 Scope: This standard addresses the fundamental attributes that contribute to image and quality for automotive Advanced Driver Assistance Systems (ADAS) applications, as well as identifying existing metrics and other useful information relating to these attributes. It defines a standardized suite of objective and subjective test methods for measuring automotive camera image quality attributes, and it specifies tools and test methods to facilitate standards-based communication and comparison among OEM and Tier 1 system integrators and component vendors regarding automotive ADAS image quality

• 5.4 Purpose: This standard specifies methods and metrics for measuring and testing automotive image quality to ensure consistency and create cross-industry reference points.

• 5.5 Need for the Project: Cameras are being used in greater numbers in automotive applications. Most of these systems have been developed independently, with no standardised calibration or measurement of image quality. Consumers have no standard reference point when using camera enabled systems, and OEM/Tier 1 developers cannot compare camera systems side by side.

• 5.6 Stakeholders for the Standard: Automotive OEMs, Automotive Tier 1 suppliers, image processing software and hardware companies, optics companies, sensor manufacturers, safety certification bodies, end users (drivers).

Key Contacts – P2020 OfficersChair Margaret Belska

+1 408 462 5109mbelska@ieee.org

Vice Chair Dr. Sven Fleck+49 176 6104 0825fleck@smartsurv.de

Secretary Prof. Robin Jenkin+1 408 650 4828rjenkin@nvidia.com

Treasurer Prof. Patrick Denny+353 93 23184 patrick.denny@valeo.com

Communications Robert Stead+44 208 133 5116 rob@sensemedia-events.com

Subgroup #0Image quality requirementsand specification standards

(Terms and Definitions)

Subgroup #0 - Image quality requirementsand specification standards• Coordinates topics relevant to all subgroups

• Glossary to define a common language

• List of (actual) Camera Samples

• Camera Matrix

• List of Use-Cases

• List of Automotive Spectra

Sub Group 0 – Image quality requirements and specification standards

https://www.techstreet.com/ieee/standards/ieee-white-paper?product_id=2021273

Gap Analysis

Subgroup #1LED-Flicker

Sub Group 1 – LED Flicker

Sub-Group 1

Sub Group 1 – Paper

• Deegan, B., LED flicker: Root cause, impact and measurement for automotive imaging applications, Electronic Imaging Symposium 2018, Autonomous Vehicles and Machines Conference 2018, Society for Imaging Science and Technology, DOI: https://doi.org/10.2352/ISSN.2470-1173.2018.17.AVM-146

Sub Group 1 – Psychovisual study

Subgroup #2Image Quality for Viewing

Sub-Group 2 - Image Quality for Viewing

• Developing stable objective metrics initially

• Dynamic Range (Mario Heid - Omnivision) • Dynamic Range Extension and Lowest SNR defined for sensor level

• Noise (Orit Skorka – ON Semiconductor, Vladimir Zlokolica, Valeo)• Moved away from “Low light Sensitivity” at present to concentrate on noise metrics

versus light level. Probably SNR based. Experimental set-up well defined.

• Glare and Flare (Andre Rieder & Darryl Shongedza, Gentex) • Well defined experimental setup and luma metric defined as extension of ISO 18844

to account for source angle. Will extend to account for chroma.

• Resolution (Uwe Artmann, Image Engineering)• Starting development, probably eSFR based

Subgroup #3Image Quality for Machine Vision

Slightly behind schedule w.r.t tooriginal plan

Will be part of first release

Slightly behind schedule w.r.t tooriginal plan

Might be part of first release

Basic discussions startedWon‘t be part of the first

standard document

Sub Group 3 – Time line

Sub Group 3 – Image Quality for Machine Vision

From: https://doi.org/10.2352/ISSN.2470-1173.2019.15.AVM-030

Sub Group 3 – Papers on CDP

• Geese, Marc; Seger, Ulrich; Paolillo, Alfredo, Electronic Imaging. Detection Probabilities: Performance Prediction for Sensors of Autonomous VehiclesDOI: https://doi.org/10.2352/ISSN.2470-1173.2018.17.AVM-148

• Artmann, Uwe; Geese, Marc; Gäde, Max: Contrast detection probability -Implementation and use casesDOI: https://doi.org/10.2352/ISSN.2470-1173.2019.15.AVM-030

Real world Spectra Measurements

24 January 2020 26© Continental AG

Summary and Support• IEEE P2020 is a collaborative effort working towards identifying and

creating fundamental metrics to evaluate image quality for automotive applications.

• Important work that has safety considerations.

• In addition is complex and needs collaboration across disciplines.

• We need your participation!• Meeting fee $100 per meeting

• Three meetings per year (January (variable location), May (Detroit), September (Brussels), May and September in alignment with AutoSens)

• We also need the support of your companies for meetings and to facilitate work

• Trying to keep participation open and free to be inclusive as possible

Optical Quality of Windscreens

Algorithm Performance (AP)

Image Quality (IQ)

Optical Quality (OQ)

… influence this?How does this…

We Need You!

Next meeting:Jan 30/31 2020

OnSemi, San Jose, CA

Thank you!

Backup

Whitepaper Goals

1) Raise awareness that image quality for automotive application is not well-defined, critical metrics for specification are missing, leading to repeated work efforts towards that matter.

2) Raise awareness that the IEEE P2020 working group is trying to remedy these deficiencies to the best extent as possible.

3) Connect with other people already working on similar challenges.

4) Attract more people to help with the IEEE P2020 effort.

• High level steering and projects that are common to all teams

• Camera Matrix (Uwe Artmann – Image Engineering)• Camera systems under IEEE-P2020 can have a huge variation in technical

components, use-cases, architecture and limitations. This makes it a challenge for discussion and communication:

• Same test procedure for a 190° FOV rear-view RGB camera and an IR sensitive RCCC camera with 4K front camera and 50m focus distance?

• Not very likely, but how do we communicate this?

• IEEE-P2020 has many different members and the OEM, Tier1 and supplier perspective can be quite different. We need to have a common ground for internal discussion and external communication.

Sub Group 0 – Image quality requirements and specification standards

• Create a matrix of all relevant options / differentiator between camera systems

• Use this matrix to describe a small number of “typical systems”

• Use the “typical systems” as reference within the standard, whole list in annex of the P2020 standard

• E.g. we can refer to “rear view camera” within the document and the reader can look up in the annex all details about sensor, lens, control level etc.

Camera Matrix

Standard for Characterization

of Image Sensors and Cameras

Release 3.1, December 30, 2016

a

sensor

dis

k-s

hap

ed lig

ht sou

rce

mount

d

D D'

b

Figure 4: a Optical setup for the irradiation of the image sensor by a disk-shaped light source,

b Relative ir radiance at the edge of a image sensor with a diameter D 0, i l luminated by a perfect

integrating sphere with an opening D at a distance d = 8D .

of the setup, which is is defined as:

f # =d

D. (26)

Measurements performed according to the standard require an f -number of 8.

The best available homogeneous light source is an integrat ing sphere. Therefore it is not

required but recommended to use such a light source. But even with a perfect integrat ing

sphere, the homogeneity of the irradiat ion over the sensor area depends on the diameter of

the sensor, D 0, as shown in Fig. 4b [10, 11]. For a distance d = 8 · D (f-number 8) and a

diameter D 0 of the image sensor equal to the diameter of the light source, the decrease is

only about 0.5% (Fig. 4b). Therefore the diameter of the sensor area should not be larger

than the diameter of the opening of the light source.

A real illuminat ion setup even with an integrat ing sphere has a much worse inhomo-

geneity, due to one or more of the following reasons:

R eflect ions at lens mount . Reflect ions at the walls of the lens mount can cause signif-

icant inhomogeneit ies, especially if the inner walls of the lens mount are not suitably

designed and are not carefully blackened and if the image sensor diameter is close to the

free inner diameter of the lens mount .

A nisot r opic l ight sour ce. Depending on the design, a real integrat ing sphere will show

some residual inhomogeneit ies. This is even more the case for other types of light sources.

Therefore it is essent ial to specify the spat ial nonuniformity of the illuminat ion, ∆ E . I t

should be given as the di↵erence between the maximum and minimum irradiat ion over the

area of the measured image sensor divided by the average irradiat ion in percent :

∆ E [%] =Emax − Emin

µE

· 100. (27)

It is recommended that ∆ E is not larger than 3%. This recommendat ion results from the

fact that the linearity should be measured over a range from 5–95% of the full range of the

sensor (see Sect ion 6.7).

6.2 Spect ral Propert ies of Light Source

Measurements of gray-scale cameras are performed with monochromat ic light with a ful l

width half maximum (FWHM) of less than 50nm. For monochrome cameras it is recom-

mended to use a light source with a center wavelength to the maximum quantum efficiency

of the camera under test . For the measurement of color cameras, the light source must

be operated with di↵erent wavelength ranges, each wavelength range must be close to the

maximum response of one of the corresponding color channels. Normally these are the colors

blue, green, and red, but it could be any combinat ion of color channels including channels

in the ult raviolet and infrared.

c Copyright EMVA, 2016 13 of 39

Bayer Pattern Striped Pattern CYGM Pattern RGBW Pattern

Sub-Group 1 :LED Flicker Updates

• Psychophysics study• Goal – correlate proposed flicker metrics with subjective experience of flicker

• Experimental design underway• Pilot study to be performed at face to face meeting

• LED flicker test setup• Basic concept defined, details need to be documented and agreed (e.g. Formalizing dynamic

range, background illumination, testing of different colours etc

• LED flicker vs phase study• Calculate probability of capturing/not capturing a pulse, for a given frequency, duty

cycle and exposure time

• LED flicker white paper in 2019• Goal – a review of root cause and in particular, case studies showing the impact of

flicker on human viewing and machine vision applications. First draft intended for next face to face

Resolution

Follow ideas and concepts of ISO12233:2014 and IEEE-P1858

Established and well known methods like eSFR(slanted edge) and sSFR (Siemens star)

Make an independent document as requirements for automotive industry are similar but different to photography and mobile imaging

Give good definitions, so that specifications and measurement procedures will improve Less discussions within the supply chain

Reduce loop holes in the SFR measurement include metric for sharpening and other processing steps in SFR measurement

outdated

0,00

0,20

0,40

0,60

0,80

1,00

1,20

1,40

1,60

1,80

0,00 0,10 0,20 0,30 0,40 0,50 0,60

contr

ast

LP/px

GoPro

Center Corner [sag] Corner [tan]

nyq

uist

lower sag. than

tan. SFR

high sharpening

Highly distorted images can‘t be analyzed

testchart front view image captured by camera

image scene top down view

camera‘s field of view

180°

basic

advanced

Contrast Resolution Chart

• Film-based transmissive target placed

over a lightbox

• 20 regions

• 0.3D difference

• 95 dB density range (nominal) from

lightest large patch to darkest

• Chart regions defined by their Optical

Density (OD)

• OD is a base-10, logarithmic unit

(Graphic, tone mapped for display)

𝐼0 𝐼 = 𝐼0 ⋅ 10−𝑂𝐷 Courtesy of Rob Sumner, Imatest

Commercial Camera, ISO 100Integration bounds: ±0.05 of target

Patch 1

Patch 6

Region guide (graphic)

Patch 12

Courtesy of Rob Sumner, Imatest

CDP Standard Text Status

Content is growing and 10-Minutes Tasks have been made available for participation

CSP - Color Separation Probability

Many Tasks and Experimental Work has been conducted! See iMeet for the slides!