Breaking up the black box - events.static.linuxfound.org · Breaking up the black box: 1. The problem 2. Insights 3. The tool 4. Feuerlabs Monday, August 27, 2012

Breaking up the black boxErlang on Embedded Systemsby Magnus Feuer, CTO Feuerlabs

Monday, August 27, 2012

Breaking up the black box:

1. The problem

2. Insights

3. The tool

4. Feuerlabs


1. The problem


Case Study: OEM recall


Case Study: OEM recall• Millions of vehicles to a shop



• Many are safety critical




• Billions in cost




• Billions in cost

• Negative brand impact


Fault searching


Fault searching• Software/Hardware/Mechanical?



• Months of uncertainty




• Very little data to go on




• Very little data to go on

• Engineering pressure


Obstacles


Obstacles• Fault recreation is almost impossible



• No way to monitor the vehicles in the field



• No way to monitor the vehicles in the field

• Operating in a black-box environment


Root cause

Source: http://spectrum.ieee.org/green-tech/advanced-cars/this-car-runs-on-code/


http://spectrum.ieee.org/green-tech/advanced-cars/this-car-runs-on-code/


Root cause• The industry is approaching server-level

complexity in embedded system







• Still using embedded tools








• Mercedes Benz navigation system – 20M SLOC








• Mercedes Benz navigation system – 20M SLOC

• 50+ Networked Control Units





It will get worse:


It will get worse:• Software becoming a di!erentiator



• Device Segmentation




• Multicore is coming with threading




• Multicore is coming with threading

• Connected devices are becoming mandatory


2. The insight


Lessons learned in the field


Lessons learned in the field• Realized that we needed to break up

the black box

• This is a distributed system

• Both server and device stack needed


Cascading requirements


Cascading requirements

Staging

Key Revocation

Transfer

Key Rotation

Authentication

Threading

Tra!c O

verride

Rollback

Encryption

Encryption

Simulations vs. H

ardware

Binary Delta Creation

Dependency R

esolve

Watchdog Trigger

Dom

ain Based Unit Testing

Record-R

eplay

Provisioning

Race Conditions

Versioning

Cascading Rollback

Regression w

/ Live Data

Sandboxing

Load Balancing

Tra!c Throttling

Validation

Access Control

Core A!

nity

Rating Plans

Deploym

ent

Pooled Plans

Testing

Package Mgm

t.

Authentication

Third Party KeysValidation

Device/Server keys

Atom

ic Update

Key ChainsR

ollback

Configuration Segmentation Data Budget FOTA Multi-core


3. The tool


Why Erlang?


Why Erlang?• Open Source Software



• Lightweight processes




• Message based




• Message based

• Created by Ericsson to handle telco infrastructure


Erlang usage:

1) http://www.trifork.com/cases/cloud/vocalink2) http://www.drdobbs.com/parallel/using-erlang-to-build-reliable-fault-tol/220600332?cid=RSSfeed_DDJ_HighPerformanceComputing3) http://www.erlang-factory.com/upload/presentations/31/EugeneLetuchy-ErlangatFacebook.pdf4) https://github.com/blog/112-supercharged-git-daemon5) http://www.erlang.org/doc.html


http://www.trifork.com/cases/cloud/vocalink


http://www.drdobbs.com/parallel/using-erlang-to-build-reliable-fault-tol/220600332?cid=RSSfeed_DDJ_HighPerformanceComputing


http://www.erlang-factory.com/upload/presentations/31/EugeneLetuchy-ErlangatFacebook.pdf


https://github.com/blog/112-supercharged-git-daemon


http://www.erlang.org/doc.html


Erlang usage:- Vocalink Real-Time Payment Switch [1]














- Yahoo [2]














- Yahoo [2]

- Facebook chat [3]














- Yahoo [2]

- Facebook chat [3]

- Github [4]














- Yahoo [2]

- Facebook chat [3]

- Github [4]

- Telco (AXD301)[5]













The key factor enabling Erlang


The key factor enabling ErlangEmbedded systems today are as powerful as servers 15 years ago


The key factor enabling ErlangEmbedded systems today are as powerful as servers 15 years ago

1998 Server 2011 Embedded

Compaq ProLiant 7000 Embest SBC6045

Pentium II Xeon 450 Mhz Atmel AT91SAM9G45 400 Mhz

256MB RAM 256MB RAM

6GB disk (140Mbit/sec) 32GB SD card (115Mbit/sec)


Use speed to build robustness


Use speed to build robustness• Telco’s 90s needs are today’s embedded

requirements



requirements

• VMs are no longer too resource intensive



requirements

• VMs are no longer too resource intensive

• High-level languages allow multicore utilization


Why Erlang is better than X


Why Erlang is better than X• Built for communication



• Runtime upgrade



• Runtime upgrade

• Graceful failure



• Runtime upgrade

• Graceful failure

• Enables replay of input data


Solving the OEM recall case


Solving the OEM recall case• Design a probe to be deployed



• Push probe over the air




• Harvest log and capture data OTA





• Replay in lab to recreate fault





• Replay in lab to recreate fault

• Design and deploy fix (if applicable)


Probe deployment 1-2-3


Probe deployment Step 1: Push probe to vehicles


Probe deployment Step 1: Push probe to vehicles

Telematics Server

TPS MAFEngine Controller

Telematics Device

Battery Controller

Monitor

Monitor package

CAN

bus


Probe deploymentStep 2: Fault detection

Telematics Server


Telematics Device

Battery Controller

Monitor

CAN

bus TPS


Probe deploymentStep 3: Capture and report

Telematics Server


Telematics Device

Battery Controller

Monitor

CAN

bus TPS


Inside the vehicle1. Normal operations



Engine Control Unit

CAN Bus

Transmission Control Unit

Throttle Position Sensor

Erlang Probe

CAN Chipset



Engine Control Unit

CAN Bus



Erlang Probe

CAN Chipset


Inside the vehicle2. Capture TPS data

Engine Control Unit

CAN Bus



Erlang Probe

CAN Chipset

Capture DBData + Time stamp


Recreate fault in HQ Lab3. Replay data to test harness




CAN Bus


Erlang Probe

CAN Chipset




CAN Bus


Erlang Probe

CAN Chipset


The end result


The end result• Shortens root cause location time



• Immediate action item for the OEMs




• Identifies QA or design issue




• Identifies QA or design issue

• Fixes installed without customer involvement


4. Feuerlabs


What do we do?


What do we do?• Feuerlabs provides device management servers,

data communication, and embedded development frameworks to implement industrial-level M2M solutions




• Embedded stack, Exosense Device





• Server side, Exosense Server





• Server side, Exosense Server

• Communication - Provide roaming data plans


Feuerlabs’ domain


Battery Controller

CAN

bus


Feuerlabs’ domain


Battery Controller

CAN

bus

Telematics Server

Telematics Device



Exosense ServerExosense Device

Device Application

Server Application

NOC monitoring

Data, RPC

Config, updates, RPC

Sensor Pkg

DAQ Module

JSON Router

SNMPRouter

RPCRPC

ConfigConfig

MonitorMonitorExoport

Exoport


Exosense Device Stack


Exosense Device Stack• MPLv2



• Reference Hardware




• Mixed language apps through DBUS





• Package management w/ upgrades





• Package management w/ upgrades

• Bi-directional encrypted RPCs





Linux Kernel

Erlang

Device Drivers File Systems

Exosense Device Platform

Geospatial

GPS

RPC Mgr

RPC Queue

Alarm Mgr

Data Monitor

i2c

i2c bus

SocketCAN

CAN bus

Backend ServerConnection Scheduler

Backend ServerConnection

Protocol

Encryption

Mobile Communication

Integration

Graphics Mgr

Framebu!er

Busgraph

Touchscreen

Device Mgr

Data Acqusition

Mgr

Config Mgr

Config Database

Config Sync

Queue

Application


Exosense Server Stack


Exosense Server Stack• Device and Tra"c manager



• Configuration management




• Update manager




• Update manager

• Mixed assets




• Update manager

• Mixed assets

• Data plan aware


Conclusion


Conclusion• Connectivity is happening



• Increased complexity is happening




• A new toolset, and mindset, is needed




• A new toolset, and mindset, is needed

• Erlang provides the tools


Questions

Contact:Magnus Feuer Email: [email protected]: +1 949 294 7871


mailto:[email protected]

mailto:[email protected]

Documents

Breaking up the black box - events.static.linuxfound.org · Breaking up the black box: 1. The problem 2. Insights 3. The tool 4. Feuerlabs Monday, August 27, 2012