Presentation 6

Integration of Mobile Apps into the Vehicle HMI

Terminal Mode Deep Dive

Presenter:Matthias Benesch, Daimler AG

Jörg Brakensiek, Nokia Corporation

Overview

� Introduction

� VNC Protocol

� Demonstration – Part I

� Device Attestation

� Audio

� Service Negotiation

� Demonstration – Part II

� Summary

Jörg Brakensiek, Nokia Research Center Palo Alto | June 2, 20102

Introduction – Guiding Principles

Technology properties

• Existing

• Non-proprietary

• Well known

• Stable• Stable

• Do extend only if necessary

Be Application & OS Agnostic

• Support legacy application (no change needed)

• Allow for optimization to automotive

• Can be implemented on existing platforms

Introduction – Terminal Mode Stack

Terminal Mode

RTP & BT(Audio)

VNC(Display & Control)

UPnP(Service Discovery & Initiation)

DeviceAttestation

WLAN USB 2.0 …

Internet Protocol Abstraction Layer

(Service Discovery & Initiation)

BT

VNC Protocol – Overview

Display ControlRFB Protocol

VNC Server

VNC Client

CE DisplayCE Display

HU Display

User Input

Copy CE display to Head-unit display

Receive user input events from the Head-Unit

Consumer Electronics

Device

Automotive Head Unit

Display Control

VNC Protocol – Framebuffer: Concept

Memory

FB

Memory

FB

Sca

lingRFB Protocol

Add

H

eade

r

Rem

ove

Hea

der

VNC Protocol – Framebuffer: Protocol

R G BRGB 5655 6 5

Framebuffer (FB) Update Header

Position of update in source FB• X / Y position

• Width / Height

• Encoding

A R G BARGB 888

8 8 8 8

VNC Protocol – Framebuffer: Messages

Set FB Size, CE Pixel Format

Set Head-Unit Pixel FormatSet

up

Pha

se

Request FB Update

Provide FB Update

Ope

ratio

n P

hase

Full Update

VNC Protocol – Framebuffer: Updates

Incremental Update

Phone

Model

MTU

Size

FB Size

[kByte]

Home Screen Camera Viewer

Full Update Full Update Change Inc. Update

N97 1500 900 4 FPS 3 FPS 22 % 12 FPS

N97 8260 900 7 FPS 5 FPS 24 % 16 FPS

Pro: Reduce fragmentation and data transfer size

Con: Additional Processing for incremental analysis

VNC Protocol – Framebuffer: Optimization

8 8 8 8 A R G B

5 6 5

R G B

Color Conversion

5 6 5

Phone

Model

Color

Format

FB Size

[kByte]

Home Screen Camera Viewer

Full Update Full Update Change Inc. Update

N97 RGB888 900 7 FPS 5 FPS 24 % 16 FPS

N97 RGB565 450 12 FPS 10 FPS 22 % 20 FPS

Pro: Reduce Data transfer size

Con: Color conversion processing at phone and head-unit

FB Request

FB Update

FB Request

FB Update


FB Changed

Wait for Change

Pro: Reduced CPU load

Con: External trigger from Display Driver / UI framework

Phone

Model

Color

Format

No Wait Wait for FB Update

Incremental CPU Load Change Incremental CPU Load

N97 RGB888 70 FPS 100 % 18 % 7 FPS 17 %

N97 RGB565 70 FPS 100 % 15 % 7 FPS 12 %

Application: Performance Monitor


A R G B A R G B A R G B A R G B A R G B

Run-Length Encoding

3 R G B 0 R G B

Phone

Model

Update RAW Encoding RLE Encoding

Update CPU Load Average RL Update CPU Load

N97 Full 7 FPS 70 % 6 pixel/run 19 FPS 85 %

N97 Increment 7 FPS 17 % 6 pixel/run 7 FPS 10 %

Application: Performance Monitor

Pro: Reduced CPU load

Con: Run-Length Encoding processing at phone and head-unit

Cro

p

VNC Protocol – Framebuffer: Extension

• Scaling support

• Display resolution & sizeClient

Display Config.

Memory

FB

Sca

le

• Distance to driver

Virtual FB

Back

Option

+

-

Config.

VNC Protocol – Framebuffer: Extension

Application

Device Status

Key lock Voice input

Device lock Driver Distraction

Dim display Framebuffer rotation

Night mode Framebuffer orientation

VNC Protocol – Input Events: Protocol

Key Events

• Unique key identifier

• Key press & Key release

Pointer Events

• Event position ( x / y )

• Button mask (left, middle, right)• Button mask (left, middle, right)

Phone does not distinguish between local and remote events

Keyboard events defined in /usr/include/keysymdef.h

VNC Protocol – Input Events: Identifier

Specific Device Events Key Identifier

ITU Key 0 (‘0’, ' ‘) 0x3000 0100

ITU Key 1 (‘1’, '.', ‘,') 0x3000 0101

ITU Key 2 (‘2’, ‘A’, ‘B’, ‘C’) 0x3000 0102

ITU Key 3 (‘3’, ‘D’, ‘E’, ‘F’) 0x3000 0103

Phone call start 0x3000 0200

Phone call end 0x3000 0201

Specific Head-Unit Events Key Identifier

Shift Right 0x3000 0001

Shift Left 0x3000 0002

Shift Up 0x3000 0004

Shift Up-Right 0x3000 0005

Shift Up-Left 0x3000 0006

Shift Down 0x3000 0008

Soft key – Left 0x3000 0202

Soft key – Middle 0x3000 0203

Soft key – Right 0x3000 0204

Application key 0x3000 0205

Multimedia – Play 0x3000 0400

Multimedia – Pause 0x3000 0401

Multimedia – Stop 0x3000 0402

Multimedia – Forward 0x3000 0403

Shift Down-Right 0x3000 0009

Shift Down-Left 0x3000 000A

Push 0x3000 0020

Rotate clock wise 0x3000 0050

Rotate anti-clock wise 0x3000 0060

Head-Unit Key 0 0x3000 0300

Head-Unit Key 1 0x3000 0301

Head-Unit Key 255 0x3000 03FF

VNC Protocol – Input Events: ExtensionTouch Head-Unit

Application Dev

ice

Eve

nts

Device

Request Event Mapping

Non-Touch Head-Unit + Knob

Kno

b E

vent

s

Event Mapping

Device Events

VNC Protocol – Input Events: Extension

City:City:

Remove

City:

Show Keyboard

City: Hel Event List

White List:

{ e, u, o, a, s, f, v, m }

Remove Keyboard

VNC Protocol – Driver Distraction

Ok to show, while driving

Not ok to show, while driving


Problem

• Not all content is save to show on the head-unit display while driving.

• Head-unit cannot reason about the content on its

Context Information Structure

X-position of rectangle

Y-position of rectangle

Width of rectangle

Height of rectangle

Context Information Encodingabout the content on its display from the pixel data.

Solution

• Provide Context Information together with the pixel data.

Context Information Encoding

Unique application identifier

Trust Level for application category

Trust Level for content category

Application category

Content category

Driver Distraction Rules followed


Trust Level

No Trust

User Configuration

Self-Registered Application

Registered Application

Application Category

Unknown Application

UI Framework

Phone call application

Media applications

Content Category

Text

Video

Image

Vector Graphics

Application certificateMessaging applications

Navigation

Browser

Productivity

UI less applications

System

3D Graphics

User Interface

Car Mode

Media Audio

Voice Audio

Misc. content

Driver Distraction Rules

Minimum font size required.

No video is shown.

No automatic scrolling text

Maximum feedback time allowed


Ok to show, while driving

Not ok to show, while driving

Application not allowed!

Go back to Home Screen

Audio Input and Output – Overview

Bluetooth

A2DP & HFP

RTP over

UDP/IP

BT Audio Support• BT A2DP for application audio

• BT HFP for phone audio

RTP Audio Streaming• For application and phone audio (bi-directional)

• Audio can be as well voice commands or text-to-speech

Device Attestation – Threats

Fake TM SW

Navigation Application not allowed!

Context

Inform

ation

Filter Device

SMS Navigation

Fake Device

Navigation

Device Attestation – Trust model

Head-unit Manufacturer (HM)

SKHM/PKHM

TrustsCertifies

Device Manufacturer (DM)

CertifiesSKDM/PKDMCertDM = Sig(SKHM, PKDM)

Head-unit (H)

Trust root PKHM

Trusts

Device (D)

CertD = Sig(SKDM, PKD)SKD/PKD

Certifies

DeviceAttestationProtocol

Device Attestation – Attestation Protocol

Random Nonce, Attested software id

Measure requested software componentMeasure requested software componentExtend with IP address and Port numberCreate attestation signature (Quote)

CertDM, CertD, Quote, IP, Port

Verify CertDM with PKHM (� PKDM)Verify CertD with PKDM (� PKD)Verify Quote with PKDSave IP and port

Summary

Terminal Mode consists of a set of basic protocols, enabling user-interface replications

• VNC for display synchronization and event input

• RTP and BT for audio transport

• UPnP for service negotiation

Driver distraction is taken care of in various elements

• Context Information

• Individual Launch and Termination

• Device Attestation

Full specification for download at www.nokia.com/terminalmode

Jörg Brakensiek, Nokia Research Center Palo Alto | June 2, 201027

What does Terminal Mode not Address?

� Providing IP connectivity to the Head-Unit (IP tethering)

� Video and high-end Gaming Use cases for rear-seat entertainment systems

� Provide the CE device access to car data

� Provide the head-unit access to CE device data

� Certification process for automotive applications

Documents

Presentation 6