AGL Real-time Architecture Optionsfor Critical Services

Automotive Linux SummitTokyo June 2018

Fulup Ar Foll Lead Architectfulup@iot.bzh

Jun-2018AGL Realtime Architecture Options 2

IoT.bzh Crew

Jun-2018AGL Realtime Architecture Options 3

1St Technical Contributor

Jun-2018AGL Realtime Architecture Options 4

What is RealTime in an OS?

● Realtime means “on time” and not “faster”– Realtime is about predictability

● Typically Realtime address following concerns– 80% => Could my execution code be disturbed ?

● If then:– For how much time ?– How often ?– By who ?

– 20% => How big is my latency● Time lost between an external event, and the time my code can

start to handle it ?● Maximum latency fluctuation ?

Jun-2018AGL Realtime Architecture Options 5

Linux & Soft/Hard Realtime

● Soft realtime● Periodical tasks/events of several milliseconds● Some acceptable unpredictable delays (10/100 ms)● Often implemented by resource controls● Latency of few ms with exceptional unpredictable fluctuation +-10ms

● Hard realtime● Total Latency magnitude of 10/100 us● Predictable and short delays (< 250ms)

● Current vanilla Linux kernel is Soft RealTime● Harder realtime with:

– Preempt-RT– Xenomai

Jun-2018AGL Realtime Architecture Options 6

Which Automotive Apps need RT ?

● Soft Realtime● Data Acquisition● Audio/Video

● Hard Realtime● Cluster● Emergency/Safety signal

Jun-2018AGL Realtime Architecture Options 7

Linux RT Application Impact

● Standard Linux a simple “Ping Flood“ will lags applications.

● Linux network IRQs preempt applications too often and for too long, which significantly increase the latency.

● PREEMPT_RT reduces scheduling latency● Replaces most spinlock by mutexes● Support threaded IRQs● Supports high resolution timers

Jun-2018AGL Realtime Architecture Options 8

Turn “ON” Linux RealTime.

● Objectives● Decrease Application Latencies● Guaranty that high priority tasks will not be bothered by lower

priority ones.● Make sure interrupts cannot lag your critical apps.

● Soft Realtime (Standard Kernel)● Container, CGroups, ...

● Hard Realtime (Kernel must be patched)● PREEMPT-RT● IPIPE+Xenomai

Jun-2018AGL Realtime Architecture Options 9

Preempt_RT vs Xenomai

● Xenomai● Xenomai supports to leagcy RT non-POSIX applications (eg: VxWorks)● Dual Kernel solution brings more performances when no more than 4

cores run RT threads● More confidence on the whole RT application (eg: /proc/xenomai statistics)● Miss some critical Unix development tools (eg: Valgind)

● Preempt_RT● Almost Vanilla Linux (no API/ABI changes)● Continuous testing in OSADL QA farm● No need for extra userspace libraries● Less confidence in app, harder to debug, needs extra code for RT

monitoring

Jun-2018AGL Realtime Architecture Options 10

Xenomai Dual Kernel Mode

Jun-2018AGL Realtime Architecture Options 11

Preempt-RT Architecture

Internal design of the RTLinux system (Image Courtesy - Linux For U)

Jun-2018AGL Realtime Architecture Options 12

Preempt_RT Latency

Source: http://www.emlid.com/raspberry-pi-real-time-kernel Latency (us)

IRQs thread improves latency by removing “disabling of interupts”

Jun-2018AGL Realtime Architecture Options 13

Xenomai & Prempt_RT convergence

● The Linux kernel under Xenomai can use Prempt_RT

● Xenomai 3.x offer dual kernel and native option

● Xenomai latency remains significantly better● Some options (eg: RtNet only run Xenomai)

Jun-2018AGL Realtime Architecture Options 14

Playing Darts with Kernel Patches

● Far more matching kernel version with PREEMPT_RT patches (but harder to port)

● IPIPE patch (arm) ~= 700K (25 000 lines). Better documentation.

● PREEMPT_RT patch ~= 2.8Mo (58 000 lines)

Jun-2018AGL Realtime Architecture Options 15

RT Kernel is only a start● Realtime requires more kernel tuning and clean behaviours on the application side.

● Enable CONFIG_PREEMPT_RT_FULL & CONFIG_HIGHRES_TIMERS to get <1ms precision

● Disable CONFIG_CPU_FREQ !● Might conflic with power management

● There are strict rules to follow and actions to take in the application● Stack pre-faulting● Virtual Memory locking● Fine tuning of threads priorities● malloc() and friends chasing, to avoid page faults (can be difficult with some C++ libraries)● Forbidden usage of system(); popen(); execve() … in runtime.● Monitoring run-away threads (ie, tight loops in RT contexts) to prevent system hanging (and to allow debugging).● clock_nanosleep is your friend, for writing periodic tasks● Carefull initialization parameters of pthread_mutex, default ones do not have PTHREAD_PRIO_INHERIT !● Fancy some LTTng sessions ? (does not work with IPIPE)

● Last but not least● Not everyting can be RT● Providing high priority to some task means than the other will inherit of low priority● Base you flow on lock (semaphore) and not on thread priority● Get rid of any spin lock

Jun-2018AGL Realtime Architecture Options 16

C issues

● malloc()/realloc() do not lead to pagefault always, (though sys_brk() or sys_mmap_pgoff() because of internal memory pool of the glibc

● Thus, a RT ‘leak’ may be hard to reproduce● Using GDB with a breakpoint on malloc() is usually

sufficient● Another technique, less intrusive, is to use Memory

Allocation Hooks of the glibc.● Some companies allow malloc() for initializations, and

always forbid the free() !

Jun-2018AGL Realtime Architecture Options 17

C++ specific issues

● In C++, dynamic allocations are not always explicit

● Example: std::vector growing● In some extra libraries (eg, boost), memory

allocations may be completely out of control (in addition of alien-only-friendly backtraces)

Jun-2018AGL Realtime Architecture Options 18

RT Options inside AGL

Cluster

Carte handling

Localistion management

POI

CAN GPS

Geopositioning Virtual Signal

Multi ECU & Cloud Aware Architecture

Entertainement

CAN-BUS Virtual Signal

Gyro, AcelerometerCAN-BUS

LIN-BUS

Engine-CAN-BUS

ABS

Transport & ACL

Head Unix

Direction Indication

Cloud

Log Analytics

No-SQL Engine

Statistics & Analytics

Transport & ACL

My Car Portal

Paiement

Subcriptions

Preference

Preferences &

Custumisation

MongoDB Engine

Paiement Service

Cluster Virtual Signal

Transport & ACL

Navigation Service

Maintenance Portal

Know Bugs

Maintenances

Service Packs

Soft RealTimeHard RealTime

Jun-2018AGL Realtime Architecture Options 19

AGL RT bindings

● The easiest option to enable RT for AGL● No need to rewrite existing bindings● Could easily support linux with RT-Prempt● A subset of AGL AppFW could be ported to lighter OS (eg:

Zephyr, VxWorks, …)

● Impose to create● Portable AppFw RT transport layer● Add RT definition to application/services● More statistic & debug mechanism

● A bridge from AGL to Automotive Safety Services

Jun-2018AGL Realtime Architecture Options 20

Few References● IoT.bzh AGL publications

● https://iot.bzh/en/publications● Practical Linux RT (ELC)

● https://elinux.org/images/d/d7/Practical-Real-Time-Linux-ELCE15.pdf

● Video Intro RT Linux● https://www.youtube.com/watch?time_continue=1&v=B

KkX9WASfpI