An Introduction to TTEthernet · The Motivation for Ethernet • Ethernet hardware is low cost. • Ethernet is a well-established open-world standard and very scaleable. • The

www.tttech.com

Ensuring Reliable Networks

Copyright © TTTech Computertechnik AG. All rights reserved. 5/18/2013 / Page 1

An Introduction to

TTEthernet TU Vienna, Apr/26, 2013

Guest Lecture in Deterministic Networking (DetNet)

Wilfried Steiner, Corporate Scientist

[email protected]

www.tttech.com



Reliable Networks from TTTech

What They Have in Common … Boeing 787 NASA Orion

Audi A8 Airbus A380

www.tttech.com



Future Markets for Real-Time

Fault-Tolerant Communication

Requirements on a communication

infrastructure for future markets

•Real-time requirements

•Fault tolerance requirements

•Low cost

•Low power

•Low weight

•Low size

•Consumer acceptance

A system failure potentially leads to

•Loss of life

•Loss of economic assets

•Loss of research results

•Loss of power

•Loss of quality of service (QoS)

•…

•Any bad thing we can think of …

www.tttech.com



Closed and Open World

Communication

Closed World Communication Open World Communication

Performance guarantees:

real-time, dependability, safety

No performance guarantees:

best efforts

High cost Low cost

Standards:

ARINC 664, ARINC 429, TTP,

MOST, FlexRay, CAN, LIN, …

Applications:

Flight control, powertrain, chassis,

passive and active safety, ..

Validation & verification:

Certification, formal analysis, ...

Standards:

Ethernet, TCP/IP, UDP, FTP,

Telnet, SSH, ...

Applications:

Multi-media, audio, video, phones,

PDAs, internet, web, …

Validation & verification:

No certification, test, simulation, ...

We see a market requirement to use the same physical

network for data flows from both worlds.

www.tttech.com



Standard IEEE802.3

Ethernet LAN

Safety-, Time- or Mission-Critical System

Network

Time and space

partitioned OS

Time and space

partitioned OS

Time and space

partitioned OS

Time and space

partitioned OS

Linux

Server

Windows

PC

Windows

PC

F1

F1F1

F2F2

F2F2 F3 F4

F3 F4

F4

F4

Open Networks

Mixed-Criticality Systems

How to share system

resources

and partition critical and

non-critical distributed

functions?

TTEthernet

www.tttech.com



Traffic Classes

TTEthernet provides several traffic classes in

parallel: time-triggered, rate-constrained, and

best-effort

Time-Triggered: dispatch messages according a

predefined communication schedule

Rate-Constrained: enforce minimum duration

between two frames of the same stream

Best-Effort: standard Ethernet communication

paradigm – no temporal guarantees are given

Ethernet IEEE 802.3

Application

Time-Triggered Extension

Layer

3-7

TIME

TT1 TT2 TT2TT1 TT1TT2

30 msec

40 msec 40 msec 40 msec


TT1 TT1RC RC RC RC RCRCBE BE BE BE BE RCBE

Longest Communication Cycle in this Example: LCM(30,40) = 120msec

TTEthernet

www.tttech.com



TTEthernet, a Communication

Infrastructure Highlight: Flexible Integration

and COTS Backward Compatible

TTE TTE

TTE TTE

TTE

FX

FX

FX

FX

FX

CAN

CAN

CAN

CAN

FX

ETH

1 Gbit/sec

100 Mbit/sec

< 10 Mbit/sec

<1 Mbit/sec

TTE

TTETTETTE

TTE

TTPTTP TTP TTP

www.tttech.com



The Motivation for Ethernet • Ethernet hardware is low cost.

• Ethernet is a well-established open-world standard and very

scaleable.

• The OSI reference model gives a well-structured classification of

concepts that can be built on top of Ethernet.

• Existing tools can be leveraged as cost-efficient diagnosis tools.

• As all messages in TTEthernet are standard Ethernet compliant,

existing tools can be leveraged for time-triggered messages as well.

• Standard web servers can be leveraged for maintenance and

configuration.

• Engineers learn about Ethernet at school.

Ethernet compatibility enables the usage of technology

that is established, tested, and verified.

www.tttech.com



Outline

Prerequisites for Safe and Deterministic Communication

• Asynchronous vs. Synchronous Communication

• Clock Synchronization and Fault-Tolerant Clock Synchronization

• Formal Verification Activities

Utilization of Safe and Deterministic Communication

• Time-Triggered Communication

• Constraints in Multi-Hop Networks

• Integrated communication for mixed-criticality systems

• Combined Time-Triggered / Rate-Constrained / Best-Effort Communication

• Tooling Overview

Summary

www.tttech.com



Outline











Summary

www.tttech.com



NIC

SWITCH

NIC

NICNIC

NIC SWITCH

NIC

NIC

NIC

NIC

SWITCH

NIC

NIC

X

X

Asynchronous Communication

Transmission Points in Time are not predictable

Transmission Latency and Jitter accumulate

Number of Hops has a significant impact

Usually solved by High Wire-Speeds & Low Utilization

and/or Priorities

Problem of ``Indeterminism’’ remains

Ethernet = Asynchronous

Communication

X

www.tttech.com



Adding Clock Synchronization

to Ethernet

TTE

1588

1588

Eth

TTE

TTE

Eth

TTE

TTETTE

TTE

TTE

TTE

Eth

Time Master

IN 1

Enabler for Synchronous Operation:

Synchronized Global Time

Communication Schedule

www.tttech.com


Page 13

Quality of Clock Synchronization: Precision

In an ensemble of clocks, the precision is defined as the

maximum distance between any two synchronized non-

faulty clocks at any point in real time.

Perfect Clock Early Clock Late Clock

www.tttech.com



Time-Triggered Operation

Time-Division Multiple-Access Communication

Composable network

Complexity reduction and faster integration

Fault tolerant communication system

send

receive

receive

send

receive

receive

receive

receive

send

t1

t1

t1

t2

t2

t2

t3

t3

t3

Node A

Node B

Node C

time Slot

www.tttech.com



NIC

SWITCH

NIC

NICNIC

NIC SWITCH

NIC

NIC

NIC

NIC

SWITCH

NIC

NIC

Synchronous Communication

X

X

Exactly one order of messages Mi

(in contrast to PERM(Mi) in async. comm)

Synchronous Communication (TT)

www.tttech.com



Example: 1,000 Frames

(Industrial-Sized)

X

12

3

4

5

6

Time-Triggered Only

1 2

Dataflow Links are enumerated

on the x-axis

…

www.tttech.com



Single-Master

Synchronization

1588

1588

Eth

Eth

Eth

IN 1

IN 1

IN 1 IN 1

IN 1IN 1

IN 1

IN 1

IN 1

Time Master

constant and/or dynamic

www.tttech.com



Transparent Clock and Permanence

0

306

receive 5

302

45

302

45

70

302

8010

306

302

302

receive

receive

0

302

dispatch

ES 102

ES 106

Switch 201

Switch 202

Switch 203

306

302permanenceSwitch 203

max_transmission_delay (=120)

permanence_delay (120 – 10 = 110)

dispatch

send

send

0 5 10 15 20 25 30 35 45 50 55 60 65 75 80 85 90 95

105

110

115

40 70 100 120 130

135

140

145125

150

302

send 5

5

306

send

permanence_delay (120 – 80 = 40)

max_transmission_delay (=120)

SM 1 SM 2 SM 3

SC 1

SM 4

SC 2

SM 5

CM 1

SM 6

www.tttech.com



Synchronization Services

Per

fect C

lock

Real Time

Co

mp

ute

r T

ime

Slow Clock

Fast Clock

R.int

Me

ssa

ge

Exch

an

ge

R.int

Me

ssa

ge

Exch

an

ge

Clock Synchronization Service

Startup/Restart Service

Clock Synchronization Service is

executed during normal operation mode

to keep the local clocks synchronized to

each other.

Startup/Restart Service is executed to

reach an initial synchronization of the

local clocks in the system.

Integration/Reintegration Service is

used for components to join an already

synchronized system.

Clique Detection Services are used to

detect loss of synchronization and

establishment of disjoint sets of

synchronized components.

www.tttech.com



Single-Master Clock

Synchronization

TTE

1588

1588

Eth

TTE

TTE

Eth

TTE

TTETTE

TTE

TTE

TTE

Eth

Time Master

IN 1

Enabler for Synchronous Comm.:

Synchronized Global Time

Communication Schedule

www.tttech.com



Fault-Tolerant Clock

Synchronization

TTE

1588

1588

Eth

TTE

TTE

TTE

Eth

TTE

TTE

TTE

TTE

TTE

TTE

TTE

Eth

Time Master

Time Master

Time Master

IN 1

IN 1

IN 1IN 1

IN 1

IN 1

Fault-tolerant synchronization services

are needed for establishing a safe

global time base

www.tttech.com



Step 1: ALL Synchronization Master Dispatch

IN Frames at the SAME Scheduled Point in Time Compression

Master

Synchronization

Master 5

Synchronization

Master 4Synchronization

Master 3

Synchronization

Master 2

Synchronization

Master 1

IN 1

IN 2 IN 3 IN 4

IN 5

SM1Dispatch

Permanence SM1 SM2

SM2

SM5

SM5

SM3

SM4

SM3

SM4

t_0 t_1,

t_2

t_4,

t_5

Acceptance Window

(of SM 2/5)

CM

CM

Reference Point

Precision

...

www.tttech.com



Step 2: Compression Master Dispatch Compressed

IN Frame back to Synchronization Masters/Clients

Compression

Master

Synchronization

Master 5

Synchronization

Master 4Synchronization

Master 3

Synchronization

Master 2

Synchronization

Master 1

IN CIN CIN CIN CIN C

SM1Dispatch

Permanence SM1 SM2

SM2

SM5

SM5

SM3

SM4

SM3

SM4

t_0 t_1,

t_2

t_4,

t_5

Acceptance Window

(of SM 2/5)

CM

CM

Reference Point

Precision

...

www.tttech.com



TTEthernet Clock

Synchronization i

Algorithm Specification

www.tttech.com



TTEthernet

Clock Synchronization ii

www.tttech.com



Other Synchronization Safety

Mechanisms

Controlled and autonomous late integration

• Synchronous operation will be reached when a sufficient number of ECUs

is powered-up.

• Remaining ECUs may power up at arbitrary times and will join synchronous

operation.

Controlled and autonomous re-integration

• ECUs that drop out of the synchronous operation will autonomously re-

integrate

after recovery.

Controlled and autonomous system-wide reset

• In the extremely unlikely event that the synchronous time-base is lost, the

system is configurable to automatically execute a controlled system-wide

restart.

Synchronization robustness against EMI

• Synchronization is configurable to continue operation without receiving

synchronization messages for a parameterized number of re-synchronization

intervals.

www.tttech.com



Formal Verification Activities

CoMMiCS

TTEthernet Executable Formal Specification

• Using symbolic and bounded model checkers sal-smc and sal-bmc

• Focus on Interoperation of Synchronization Services (Startup, Restart, Clique

Detection, Clique Resolution, abstract Clock Synchronization)

Verification of Lower-Level Synchronization Functions

• Permanence Function (sal-inf-bmc + k-induction)

• Compression Function (sal-inf-bmc + k-induction)

Formal Verification of Clock Synchronization Algorithm

• First time by means of Model Checking (sal-inf-bmc + k-induction)

Re-use of the Formal Models to prove:

• Layered clock-rate correction algorithm (sal-inf-bmc + k-induction)

• Layered clock-diagnosis algorithm (sal-inf-bmc + k-induction)

Verification and minor corrections of the “Sparse Timebase” Concept

• Distributed computations without

explicit coordination (PVS)

Work has mostly been done in the context

of the Marie Curie CoMMiCS project FP7 (FP7/2007-2013) project no. 236701

www.tttech.com

Ensuring Reliable Networks References

B. Dutertre, A. Easwaran, B. Hall, W. Steiner, “Model-based analysis of Timed-Triggered Ethernet,”

Proceedings of the 31st IEEE/AIAA Digital Avionics Systems Conference (DASC 2012), IEEE 2012,

Recipient of “Best in Session” and “Best in Track” awards

W. Steiner, G. Bauer, B. Hall and M. Paulitsch, “Time-Triggered Ethernet: TTEthernet,”

In Time-Triggered Communication, R. Obermaisser, editor, CRC Press, 2011

W. Steiner and J. Rushby, “TTA and PALS: Formally Verified Design Patterns for Distributed Cyber-

Physical Systems,” Proceedings of the 30th IEEE/AIAA Digital Avionics Systems Conference (DASC

2011), IEEE 2011, Recipient of “Best in Session” and “Best in Track” awards

W. Steiner and B. Dutertre, “Layered Diagnosis and Clock-Rate Correction for the TTEthernet Clock

Synchronization Protocol, ” Proceedings of the 17th IEEE Pacific Rim International Symposium on

Dependable Computing (PRDC 2011), IEEE Computer Society, 2011

W. Steiner and B. Dutertre, “Automated Formal Verification of the TTEthernet Synchronization Quality,”

Proceedings of the 3rd NASA Formal Methods Symposium (NFM 2011), Springer Lecture Notes in

Computer Science, 2011

W. Steiner and B. Dutertre, “SMT-Based Formal Verification of a TTEthernet Synchronization Function,”

Proceedings of the 15th International Workshop on Formal Methods for Industrial Critical Systems

(FMICS 2010), Lecture Notes in Computer Science 6371 Springer, 2010,

pp. 148-163


www.tttech.com



Outline











Summary

www.tttech.com




(Industrial-Sized)

X

12

3

4

5

6

Time-Triggered Only

1 2


on the x-axis

…

www.tttech.com



A

B C

H

G

F

D

E

Physical Topology Dataflow Path Virtual Link

End-To-End (E2E) TT

Dataflow

offset_AD

offset_DE

offset_EF

TT frames can be scheduled on each communication link.

The communication schedule needs to satisfy constraints

as discussed in the following.

www.tttech.com



Contention-Free Constraints i

Definition

• A sender or relaying instance will dispatch a new frame only after

the previous frame has been processed.

• In a pure time-triggered network, the term processed refers to the

transmission of the previous frame.

• In a mixed time-triggered / event-triggered network, the term

processed can be relaxed as the previous time-triggered frame may

get delayed by an event-triggered frame in transition.

Cluster Cycle

no “overlaps”

www.tttech.com



End-to-End Constraints

Definition

• The end-to-end transmission constraints are derived from the application

and assumed to be provided by the user.

• They describe the worst-case maximum and optionally also worst-case

minimum allowed latency for a frame x.

• In general we assume that the bounds specified will be the same for all

receivers of the frame x.

Cluster Cycle

…

Cluster Cycle

bound

www.tttech.com



Path-Dependent Constraints

Definition

• Within the dataflow path of a frame x the dispatch points in time of two

adjacent edges will be well-timed.

• This means that the dispatch point in time of a succeeding edge will be

scheduled only after it was received from the preceding edge.

A

B C

H

G

F

D

E

Physical Topology Dataflow Path Virtual Link

offset_AD

offset_DE

offset_EF

e.g., slot = 5

www.tttech.com



Bounded-Memory Constraints

Definition

• The restrictions of switch memory generates another implementation-

imposed set of constraints.

• The memory size required to prevent buffer overflows in the switch can

also be expressed in terms of time.

Cluster Cycle Cluster Cycle

bound

e.g., slot = 5 e.g., slot < 8

www.tttech.com



Simultaneous Relay

Constraints

Definition

Though, not conceptually a

requirement, there may me an

implementation-derived

requirement in the switches to

dispatch a frame x on all ports

simultaneously.

Cluster Cycle

Cluster Cycle

~ same points

in time

www.tttech.com



Application-Level Constraints i

Definition

• Application-level dependency constraints describe requirements that span

multiple frames x_i.

• E.g. x_1 has to be dispatched 17.3 ms before x_2.

That’s the main complexity driver !

www.tttech.com



Application-Level Constraints ii

Physical Part

Cyber Part

Interrupts can be generated by

a synchronized time reaching

scheduled points in time.

In several safety-relevant and safety-critical systems,

synchronized time is a fundamental building block.

Physical Process

Sensor

NIC

SwitchSwitch

CPU

Capture

Sensor Value

Task Schedule

Calculate

Control Value

Switch

Operate

Actuator

Frame Schedule

1

2

3

NIC Switch

Switch Switch

Switch NIC

4

5

6

Control

NIC

CPU

Actuator

NIC

CPU

Scheduled Events

on the Timelinea b c d e f g h i

12

3 45 6

“4 shall be sent x ms

after 3 is received”

www.tttech.com



Example: 100 Frames

12

3

4

5

6

Highlighted Constraints: path-dependent,

simultaneously dispatch,

application-level

www.tttech.com



Emerging Benefits of using TT

Consistent Distributed Computing Base

Unification of Interfaces—Temporal Firewalls

Composability

• Independent Development of ECUs

• Stability of Prior Services

• Constructive Integration

• Replica Determinism

Scalability

Transparent Implementation of Fault Tolerance

www.tttech.com



Emerging Benefits of using TT

Consistent Distributed Computing Base

Unification of Interfaces—Temporal Firewalls

Composability

• Independent Development of ECUs

• Stability of Prior Services

• Constructive Integration

• Replica Determinism

Scalability

Transparent Implementation of Fault Tolerance

www.tttech.com



Outline











Summary

www.tttech.com



Mixed-Criticality Systems

Standard IEEE802.3

Ethernet LAN

Safety-, Time- or Mission-Critical System

Network

Time and space

partitioned OS

Time and space

partitioned OS

Time and space

partitioned OS

Time and space

partitioned OS

Linux

Server

Windows

PC

Windows

PC

F1

F1F1

F2F2

F2F2 F3 F4

F3 F4

F4

F4

Open NetworksHow to share system

resources

and partition critical and

non-critical distributed

functions?

TTEthernet

www.tttech.com



TTEthernet for Mixed-Criticality

Systems

Enables robust partitioning of all computing and

networking resources in one system

•Fault-tolerant distributed clock

•Hard real time communication

(µs jitter, fixed latency)

•host critical controls, video, audio, LAN, …

In parallel, two types of Ethernet communications:

Synchronous (TDMA-style) Communication: TT

Asynchronous (event-triggered style): RC + BE Ethernet IEEE 802.3

Application

Time-Triggered Extension

Layer

3-7

TIME

TT1 TT2 TT2TT1 TT1TT2

30 msec



TT1 TT1RC RC RC RC RCRCBE BE BE BE BE RCBE

Longest Communication Cycle in this Example: LCM(30,40) = 120msec

TTEthernet

www.tttech.com



TTEthernet Dataflow:

Rate-Constrained Traffic

Switch/RouterReceiver

Sender

Rate-Constrained Traffic (RC)

min. duration min. duration min. duration

www.tttech.com



Mixed Traffic on Ethernet –

RC Accumulated Jitter

Time Triggered

Rate Constrained

Best Effort

TTEthernet Switch

1

2

4a

3a

2a

1a

3b

2b

1b

4b

00:01

00:10

00:02 00:11

www.tttech.com



Mixed Traffic on an Ethernet –

RC Accumulated Jitter

Time Triggered

Rate Constrained

Best Effort

TTEthernet Switch

1 2

4a

3a

2a

1a 3b

2b

1b

4b

00:02 00:11

00:01

00:10

TT has lowest

latency and

lowest jitter

RC potentially

queue-up in

switch memory

RC frame delivery

is guaranteed, but

potentially has high

latency and jitter

TT is dispatched according

synchronized time

TT is forwarded

according

synchronized time

www.tttech.com




BE Buffer Overflow

Time Triggered

Rate Constrained

Best Effort

TTEthernet Switch

4

3

2

1

3

2

1

www.tttech.com




BE Buffer Overflow

Time Triggered

Rate Constrained

Best Effort

TTEthernet Switch

4

3 2 1

3

2 1

Best-effort frame delivery

(standard Ethernet traffic)

is NOT guaranteed !

Rate-constrained frame

delivery (standard Ethernet

traffic) is guaranteed !

www.tttech.com



Integrated Dataflow Example

TT TTTT TT TTTT TT

3ms cycle

2ms cycle

3ms cycle 3ms cycle

2ms cycle 2ms cycle 2ms cycle

6ms Cluster Cycle

RC BE BE BE RC BE t

Sender

1 Switch/RouterReceiver

Sender

2

TT TT TT

3ms cycle 3ms cycle 3ms cycle

BE BE BE t

TT TT TT

2ms cycle 2ms cycle2ms cycle

BE BE RC BE

t

Dataflow – Integration

- Time-Triggered (TT)

- Rate-Constrained (RC)

- Standard Ethernet (BE)

TTEthernet Switches are non-preemptive

store-and-forward switches using priorities

www.tttech.com



Integration Options

When two (or more) messages compete for relay to the same

outgoing port, the switch has to serialize these messages.

Typically, a priority mechanism will be used.

Priority is easy, when there is a clear “winner” in terms of priority.

If there are messages of same priority the messages will be serviced

according FIFO.

What happens if there is a

low-priority message (L) in

relay, when a high-priority

message (H) becomes ready

for relay?

H

L

H L

L H

H L

Preemption:

Timely Block:

Shuffling:

real-time

Contention:

Implemented in current

versions of TTEthernet

Implemented in early (academic)

versions of TT-Ethernet

www.tttech.com




(Industrial-Sized)

X

12

3

4

5

6

Time-Triggered Only Time-Triggered

+ Event-Triggered

1 2


on the x-axis

…

TT

TT

TT

TT

RC

RC

RC

RC

RC/BE frames are also integrated

during TT phases.

www.tttech.com




(Industrial-Sized)

X

12

3

4

5

6

Time-Triggered Only Time-Triggered

+ Event-Triggered

1 2


on the x-axis

…

TT

TT

TT

TT

RC

RC

RC

RC

RC/BE frames are also integrated

during TT phases.

www.tttech.com



TTETools Requirements Data Flow Overview

System

Specification

XML

Network

Configuration

XML

TTEBuild –

Network Configuration Plug-in Device Config. Generation

TTEPlan Network Config. (Schedule) Generation

(currently TTE-Demo Scheduler) This stores the “schedule“ (TT,

RC, ET configs). Who sends

what at what time (TT) at what

rate (RC) on what route?

High-level communication reqs.

Senders, receivers, virtual links,

sync domains, fault-tolerance

requirements, etc.

TTEBuild Basic Image Generation

Device

Configuration

XML

Device

Configuration

XML

Image Image

This is a truthful, human readable

XML representation of the binary

tables in the switches and end

systems.

This is the binary image for a

switch or end system, ready

for download. Images for multiple

devices in the system may be

collected in a download database

www.tttech.com



Outline











Summary

www.tttech.com



Summary and Conclusion

Cyber-physical systems become more and more complex with an increasing demand on resources.

Determinism is a key concept to manage complexity and to ensure system safety.

The integration of applications with mixed-criticality requirements, so that they share resources, allows cost-effective architectures for real-time and safety-critical systems.

Ethernet is a good basis for an integrated communication infrastructure.

Enabling Ethernet with time-triggered services (TTEthernet) generates a deterministic communication infrastructure for mixed-criticality systems that allows synchronous and asynchronous communication.

The synchronized global time protects highly critical dataflows from less critical or uncritical ones.

www.tttech.com



Books on Time-Triggered

Technology

www.tttech.com

E n s u r i n g R e l i a b l e N e t w o r k s

w w w . t t t e c h . c o m


Wilfried Steiner, Senior Research Engineer

[email protected]

Documents

An Introduction to TTEthernet · The Motivation for Ethernet • Ethernet hardware is low cost. • Ethernet is a well-established open-world standard and very scaleable. • The