Title 44pt sentence case

Affiliations 24pt sentence case

20pt sentence case

© ARM 2017

Next-generation automotive image processing with ARM Mali-C71

Steve Steele

ARM Tech Forum Taipei

Director, Product Marketing

Imaging & Vision Group, ARM

July 4th 2017

© ARM 2017 2

Title 40pt sentence case

Bullets 24pt sentence case

bullets 20pt sentence case

Pioneers in imaging and vision

© ARM 2017 4

Title 40pt sentence case

Bullets 24pt sentence case

bullets 20pt sentence case

Automotive ADAS growth

▪ Camera volume rising very rapidly, around 20% CAGR

▪ By 2023 (expected average):

▪ Mid-range (high-volume) cars: three cameras

▪ Luxury sedans: <ten cameras

© ARM 2017 5

Title 40pt sentence case

Bullets 24pt sentence case

bullets 20pt sentence case

Automotive needs: viewability and reliability

Viewability

▪ Dynamic range

▪ Low-light and low-noise performance

▪ Imagery close to human eyes and at times

beyond

Reliability

▪ Low latency

▪ Resistant to random and systematic faults

▪ Imagery feeds into computer vision

engine

Night vision

Pedestrian

protection system

Driver drowsiness

detection

© ARM 2017 6

Title 40pt sentence case

Bullets 24pt sentence case

bullets 20pt sentence case

Automotive needs: viewability and reliability

© ARM 2017 7

Title 40pt sentence case

Bullets 24pt sentence case

bullets 20pt sentence case

Automotive needs: viewability and reliability

© ARM 2017 8

Title 40pt sentence case

Bullets 24pt sentence case

bullets 20pt sentence case

Advanced vision requires centralised processing

ISP per camera sensor

▪ Less flexible

▪ Higher BOM costs

Centralised sensor fusion SoC

▪ Centralised processing

▪ Flexible architecture

Sensor

fusion

SoC

© ARM 2017 9

Title 40pt sentence case

Bullets 24pt sentence case

bullets 20pt sentence case

And there are standards !!

▪ A number of functional safety standards exist:

▪ ISO 26262 – Road vehicles

▪ IEC 61508 – Electrical, electronic programmable

systems

▪ DO 254 – Electronics that fly e.g. in airplanes,

helicopters

▪ …

▪ …

© ARM 2017 10

Title 40pt sentence case

Bullets 24pt sentence case

bullets 20pt sentence case

▪ Relates to electrical and electronic systems used in automobiles

▪ Qualitatively assesses risk of hazardous operational situations

▪ Aims to avoid or control systematic failures and detect or control

random hardware failures, or mitigate their effects

ISO 26262-2011 road vehicles – functional safety

© ARM 2017 11

Title 40pt sentence case

Bullets 24pt sentence case

bullets 20pt sentence case

ISO 26262: goals

Provides an automotive safety lifecycle and supports tailoring the necessary activities during these lifecycle phases

Covers functional safety aspects of the entire development process

Provides an automotive-specific risk-based approach for determining risk classes (Automotive Safety Integrity Levels, ASILs)

Provides requirements for validation and confirmation measures to ensure a sufficient and acceptable level of safety is being achieved

© ARM 2017 12

Title 40pt sentence case

Bullets 24pt sentence case

bullets 20pt sentence case

High-performance ISP for automotive: ARM© Mali™-C71

Ultra-wide dynamic range▪ 24 stops

▪ Every detail captured

Simultaneous support for human display and computer vision▪ Virtual ISPs: <4 real-time camera inputs

▪ Pleasing and natural for human vision; predictable and reliable for computer vision

Every pixel reliable▪ Real-time safety for ADAS applications

▪ Enabling system-level ASIL D and SIL3 compliance

▪ 300+ fault detection circuits, built-in self-test (BIST), CRC on data paths, every pixel

tagged

Designed for safety

Ultra WDR

Multi-camera input

© ARM 2017 13

Title 40pt sentence case

Bullets 24pt sentence case

bullets 20pt sentence case

Ultra-wide dynamic range

Revealed shadows

Enhanced mid-tones

Preserved highlights

0

2^24

2^8

0

0

2^8

Shadows details lost

Mid-tones not revealed

Highlights clipped

As perceived by the human eye

© ARM 2017 14

Title 40pt sentence case

Bullets 24pt sentence case

bullets 20pt sentence case

Dynamic range management: iridix®

▪ More than just the world’s best local tone-

mapping engine…

• Precise model of human retina contrast

adaption

• Precisely simulates illumination by white light

• Basis of Nikon “D-Lighting” (2005)

• Shipped in over 2 billion devices

iridix enables cameras to see like the human eye

© ARM 2017 15

Title 40pt sentence case

Bullets 24pt sentence case

bullets 20pt sentence case

iridix® engine: pixel-by-pixel tone mapping

Contrast in highlights

enhanced

Contrast in midtones

preserved

Contrast in shadows

enhanced

iridix calculates a unique tone curve for each

pixel of each frame

Dyn

amic

ran

ge

© ARM 2017 16

Title 40pt sentence case

Bullets 24pt sentence case

bullets 20pt sentence case

Management of dynamic range

© ARM 2017 17

Title 40pt sentence case

Bullets 24pt sentence case

bullets 20pt sentence case

Management of dynamic range

© ARM 2017 18

Title 40pt sentence case

Bullets 24pt sentence case

bullets 20pt sentence case

Supporting output for human and machine vision

Human display

Computer vision

© ARM 2017 19

Title 40pt sentence case

Bullets 24pt sentence case

bullets 20pt sentence case

Enabling system level ASIL D compliance

▪ 300+ dedicated fault detection circuits

▪ Built in continuous self-test

▪ Fault interrupt controller

▪ Pixel consistency image plane tagging

reliability of every pixel

▪ Detection of sensor and hardware faults

▪ CRC checks to all data paths including

memories and configurations

Safety featuresDeveloped within robust requirements,

tracing and validation framework

▪ Safety manual

▪ Failure modes and effects analysis (FMEA)

▪ Development interface report

Providing support for system level:

▪ SIL 3 / IEC 61508

▪ ASIL D / ISO 26262

▪ With a standards-agnostic approach

Processes

Random

faults

Systematic

faults

© ARM 2017 20

Title 40pt sentence case

Bullets 24pt sentence case

bullets 20pt sentence caseDesigned to enable system-level certification

Safety Element out of Context (SEooC)

ISP has no direct control authority

System integrating ARM’s ISP needs to meet the requirements

• ISO26262 (ASIL D)

• IEC 61508

• SIL3

• Standard product

• Integrated into system (SoC) by our customers

• Specific use cases unknown to ARM

• Has to detect faults, perform self-tests

• Has a responsibility to provide timely, accurate and reliable information to the system

• Required for the system to be safe

• Road vehicles – functional safety

• This flows down to the components of the system

ARM© Mali™-C71 Image Signal Processor

© ARM 2017 21

Title 40pt sentence case

Bullets 24pt sentence case

bullets 20pt sentence case

A complete, high-performing solution

Control the ISP, sensor, auto-white

balance and auto-exposure

Roadmap to complete automotive

software designed for ASIL

compliance

Full set of tools provided

Ecosystem support for tuning and

bringing up specific use-cases and

sensors

1.2 Giga-pixels/sec of processing

performance

Supports up to 4 real-time camera

inputs and 16 camera streams with

a single pipeline

Performance Software Tuning

© ARM 2017 22

Title 40pt sentence case

Bullets 24pt sentence case

bullets 20pt sentence case

Mali-C71 block diagram

Colour/data

plane outputs

Image sensor

sources

© ARM 2017 23

Title 40pt sentence case

Bullets 24pt sentence case

bullets 20pt sentence case

Mali-C71: the next step in automotive vision

First automotive ISP with ultra-wide dynamic range

Supports both human display and computer vision

Enables safer, more comfortable driving

The trademarks featured in this presentation are registered and/or unregistered trademarks of ARM Limited

(or its subsidiaries) in the US and/or elsewhere. All rights reserved. All other marks featured may be

trademarks of their respective owners.

Copyright © 2017 ARM Limited

© ARM 2017

Thank you

謝謝 !