From AGL Supervisor to platform global data collection · Build Develop Deploy Debug Monitor XDS XDS – Quick Overview X(cross) Development System Dedicated to Applications Developers

From AGL Supervisor to platform global data collection

Sébastien DouheretTechnical Director at IoT.bzh<[email protected]>

Automotive Linux SubmitJuly 17 - 19, 2019

Tokyo, Japan

mailto:[email protected]

July 2019From AGL Supervisor to Platform global data collection

2

Who are we ?

● 25 Engineers dedicated to AGL● Location: Lorient/Brittany/France/Europe● Team : strong visibility in Open Source Community● People background : Telecom, Automotive or Marine

domains, Embedded developments (Intel, WindRiver, ...)

https://iot.bzh/en/ http://github.com/iotbzh

LORIENT

https://iot.bzh/en/

http://github.com/iotbzh


3

IoT.bzh New Office in Lorient

Large space (700m2) including Hacking Car Lab


4

Working on ...● Main contributions on AGL :

– Cybersecurity (Secure Application Framework)

– Development tools for AGL

(SDK, X(cross) Development System)

– Integration and board support (Renesas)

– Community Support & Documentation

– 1st AGL technical contributor since 2016

● Connected Boat project

– Consortium with 4 other partners

– Targeting work boats

– Goals : increase Security, predictive maintenance and automation of administrative documents filling

● Conferences / White papers (https://iot.bzh/en/publications)

References: Walt Miner presentationsAGL AMM 2017 https://sched.co/9KB6AGL AMM 2018 https://sched.co/DRAXAGL AMM 2019 https://sched.co/L8W3

https://sched.co/9KB6

https://sched.co/DRAX

https://sched.co/L8W3


5

Agenda

XDS : Apps and Bindings development (Recap XDS - Code / X-Build / Run / Debug)

Monitor a single binding

XDS Supervision (Needs, Architecture and a Demo)

Challenges & Roadmap


6

Agenda






7

Build

Develop

Deploy Debug

MonitorXDS

XDS – Quick OverviewX(cross) Development System

● Dedicated to Applications Developers

– enables development without Yocto skills

● Cross-platform build using AGL SDK

– leverages AGL/Yocto toolchain

● Deploy on development boards orQemu

● Remote debugging from IDE

● Documentation on AGL docs website:https://docs.automotivelinux.org/docs/en/master/devguides/#xcross-development-system-user's-guide

https://docs.automotivelinux.org/docs/en/master/devguides/#xcross-development-system-user's-guide


8

XDS – Reducing complexity

● Easy target access ● console, SSH, …

● Developer uses common IDEs● VS Code, Netbeans, Eclipse...

● Dashboard Web App to manage configurations and trigger actions

● XDS API for CI workflows & specific environments● Command line tool also available (xds-cli)


9

XDS – Key Features● Multi-platform : no dependencies on developer host

(Linux / Windows / MacOS)

● Easy to setupNear-zero install, no admin privileges required

● Application sources remain localCompatibility with existing IT policies (e.g. corporate backup, git, ...)

● Cross toolchain & tools embedded in a container :

● Local : run locally (local subsystem, virtual machine, docker container ...)

● On-premises : run on a local build server● Cloud : SaaS

● Leverage specific OS capabilities where applicable(e.g. Linux for Windows Subsystem, docker)


10

1) Install xds-server on a build server machine

2) Install xds-agent and start-it on your machine

3) Browse XDS dashboard: http://localhost:8800

4) Add a new project

5) Cross build/compile your project using either :

1) Dashboard build page

2) IDE using xds-cli

6) Deploy app

7) Source debug from IDE (xds-gdb)

XDS – Operations

http://localhost:8800/


11

XDS – Web Dashboard and CLI

Web dashboardavailable on

http://localhost:8800

Command Line Interface

http://localhost:8800/


12

XDS – Dashboard (cont.)


13

Agenda






14

Monitor : a simple built in tool

● Monitoring API is built in the binder

● Monitor API is enabled by AppFW in any binder(--monitoring option)

● Monitor only ONE binding● UI is available easily:

http://<host>:<port>/monitoring/

NOTE THE TRAILING SLASH !


15

Build-in Monitor – UI Overview


16

Monitor – outcome● Trace verbs calls

– API calls with args

– Results with data● Trace events content● View internal log calls (AFB_NOTICE et al.)● Filtering available to reduce “noise”:

– Filter per API

– Filter per trace type


17

Agenda






18

AGL Micro-Services architecture


19

Bindings topology agl-demo-platform HH

How to detect abnormal binding exchanges ?

Topology alre

ady

complex

for a si

mple demo !


20

Why Supervision is needed ?

Debugging (bug analysing) and Tunning

Telematic : Black / Save-box logger

2 Main Needs

Systems based on AGL are becoming more & more complex

Simple or traditionnal debug methods no longer adapted

Constructors will need to push some metrics to cloud

1 SolutionXDS Supervision


21

Supervision & Vehicle to Cloud● Data collection for debugging

● Collect system data (cpu, memory usage, …) and AGL binding exchanges (supervision / monitoring)

● Only in development mode

● Tuning● Car testing campaign (offline storage)

● Telematics● Based on acquisition CAN, vehicle signals, GPS, ... ● Externalized vehicle data analysis (failure detection)● Black box, Save box● Push useful data to the could (aggregation, retention policies)● Development and Production mode

Store caching: data saved on-board or in a cloud Time-Serie database


22

Supervision – ArchitectureAPPLICATION

BINDERafb-daemon

BINDING

BINDING

BINDING

http ws

AGL App 1

BINDERafb-daemon

BINDING

BINDING

BINDING

AGL Service 2

APPLICATION

BINDERafb-daemon

BINDING

BINDING

BINDING

http ws

AGL App 2

afs-supervisor

BINDERafb-daemon

BINDING

BINDING

BINDING

AGL Service 1BINDERafb-daemon

BINDING

BINDING

BINDING

AGL Service 3

BINDERafb-daemon

XDS

XDS Supervision

BINDERafb-daemon

HARVESTER

Harvester Service

TimeSeries DB

BINDERafb-daemon

collectd

Low Collector

TARGET HOST


23

Supervision – Target side● afs-supervisor daemon and xds-supervision

service

xds-supervision control afs-supervisor daemon to gather AGL data (events, resquests, ...)

● Low-collector service

collect system wide metrics (CPU, memory load, temperature, …)

● Harvester service

Received and format metrics to storethem in a local or remote Time Serie Database (TSDB)


24

Supervision – Host side

● Time Serie Database POC tested with influxDB

● Graph composer and query processing tools

Grafana draws reports based on influxdb records

● XDS Supervision dashboard : control panel (config / start / stop)and virtualization (Grafana dashboards integration)


25

Low Collector - collectdDeveloped in parternship with a University

● Developed by Marc-Antoine Riou (intership)

collectd integration as an AGL service● reuse collectd source code plugins (≈ 130 plugins available)● run plungins in the Low Collector binding context

Architecture :


26

Low-Collector - collectd : rendering


27

Supervision – Visualize in XDS


28

What about logs? Loki + promtail

● What is promtail ?● responsible for gathering logs and sending them to Loki

● What is Loki ?● Scalable & highly-available log aggregator system

(≈ Prometheus)● main server, responsible for storing logs and processing

queries

● Compatible with Grafana for rendering logs


29

Demo scenario● Example: 2 apps simulate requests to 2 low

level bindings and receive responses and events

Sources : https://github.com/iotbzh/agl-services-mockup

https://github.com/iotbzh/agl-services-mockup


30

Demo...

https://iot.bzh/en/publications/xds-supervision-als-2019.htmlhttps://vimeo.com/iotbzh


31

Agenda






32

Technical challenges

● Loki, promtail powerfull and promising tools but quite recent

● Promtail written in go (go support almost done in IoT.bzh recipe)

● Time Serie Database not initially designed for embedded world (only a challenge for offline mode)

● Local processing in Car / on board (NAND writes)● Collecting too many data = intrusiveness● Tag data (ownership, priority, lifetime)● Security privileges (API V3 and multi-users)


33

Roadmap● Done : XDS Supervision POC (running on HH)

● Beta version (not production ready) need more time and resources to fully integrate supervision – Increase integration/validation of collectd plugins (today only 6

plugins)– reduce again intrusiveness– Improve smooth integration with AGL security– Log analyser specifically for Journalctl (systemd logs)

more investigations about Loki & promtail integration

● Other improvements: – on-board TSDB with edge load balancing algorithm– offline data collecting for Key Performance Indicator analysis


34

3 Things to remember

● AGL System is becoming more & more complexSupervision and log analysis mandatory to debug problem on those systems

● Don’t re-invent the wheel Integration of existing tools and reuse data format to benefit of existing UI viewer or log analyser

● Need more time and ressources Integration, Optimization, make tool robust are time consuming


35

Visiting us in Lorient


36

Q&A

Gulf of Morbihan, south of Brittany, France


37

References● Proof of concept sources

● https://git.automotivelinux.org/apps/agl-service-harvester● https://git.automotivelinux.org/apps/agl-service-xds-monitoring● https://github.com/iotbzh/agl-service-low-collector*● https://git.automotivelinux.org/src/xds/xds-agent

● TSDB and Logger and Rendering Tools● https://fosdem.org/2019/schedule/event/loki_prometheus_for_logs/

● https://www.influxdata.com/● https://grafana.com/● https://d3js.org/

* Not latest version, need Univesity acceptance

https://git.automotivelinux.org/apps/agl-service-harvester

https://git.automotivelinux.org/apps/agl-service-xds-monitoring

https://github.com/iotbzh/agl-service-low-collector

https://git.automotivelinux.org/src/xds/xds-agent

https://fosdem.org/2019/schedule/event/loki_prometheus_for_logs/

https://www.influxdata.com/

https://grafana.com/

https://d3js.org/


38

More References...

● IoT.bzh:

– Website: https://iot.bzh/

– Publications: https://iot.bzh/en/publications

– Github: https://github.com/iotbzh● AGL:

– Website: https://www.automotivelinux.org/

– Documentation: http://docs.automotivelinux.org/

– Sources: https://git.automotivelinux.org/

https://iot.bzh/

https://iot.bzh/en/publications

https://github.com/iotbzh

https://www.automotivelinux.org/

http://docs.automotivelinux.org/

https://git.automotivelinux.org/

Documents

From AGL Supervisor to platform global data collection · Build Develop Deploy Debug Monitor XDS XDS – Quick Overview X(cross) Development System Dedicated to Applications Developers