The System-Level Simplex Architecture

Stanley Bak

Olugbemiga Adekunle

Deepti Kumar Chivukula

Mu Sun

Marco Caccamo

Lui Sha

Building Reliable Software

• Use best practices from industry:– Software review– “Safe” programming languages – Extensive testing

• Cost:– $100-$1000 per line of code

• And worst of all…

“Reliable” Software Still Has Bugs!

• In 2007, 12 F-22s were going from Hawaii to Japan

• After crossing the IDL, all 12 experienced multiple system crashes– No navigation– No fuel subsystems– Limited

communications– Rebooting didn’t help

• Formal Verification?– F-22 has 1.7 million

lines of code

Our Contribution

Our contribution is threefold:

• We present the System-Level Simplex Architecture that provides reliability for large, safety-critical systems.

• We formalize and verify our architecture in an AADL model, which can be immediately instantiated for applications requiring reliability.

• We demonstrate the System-Level Simplex Architecture in an Inverted Pendulum control system, and empirically verify its safe functionality in spite of controller/OS/middleware bugs.

Trends in Cross-Layer Systems

• Reliability is a cross-layer property

• Other examples of cross-layer properties include security, and real-time

Hardware

Operating System

Software

Trends in Security

• Security protocols at the software-level (SSL) trust the operating system and hardware is secure

• An operating system can be compromised by a kernel rootkit or VMBR

Solution: Use the hardware for security checks (Secure Boot, TPMs)

Hardware - TPM

Operating System - SELinux

Software - SSL

Trends in Real-Time Systems• A real-time software application requires the operating system be

aware of real-time requirements and the hardware be predictable• A real-time operating system can use a real-time scheduling

algorithm, but can do nothing in the face of unpredictable hardware

Ideal Solution: Design hardware predictably (ASICs, deterministic hardware)

Practical Solution: Enforce predictable behaviour (bus monitoring and cutoff)

Hardware – Deterministic Processor

Operating System - VxWorks

Software – Flight Controller

Trends in Reliability• Designing 100% correct, complex control software is

infeasible

• Operating systems can provide isolation (microkernel) and power through abstractions, but are often large, complex, and unverified

Ideal Solution: Design hardware reliably; verify all software

Practical Solution: Reject OS abstractions? Accept failure?

Hardware – ???

Operating System – MINIX 3 (microkernel)

Software – Complex Flight Controller

Trends in Reliability• Designing 100% correct, complex control software is

infeasible

• Operating systems can provide isolation (microkernel) and power through abstractions, but are often large, complex, and unverified

Ideal Solution: Design hardware reliably; verify all software

Practical Solution: System-Level Simplex

Hardware – System-Level Simplex

Operating System – MINIX 3 (microkernel)

Software – Complex Flight Controller

System-Level Simplex

• System-Level Simplex works components off the shelf (COTS) is compatible with existing engineering practices (triple modular redundancy)

• First, develop a simple, safe controller in hardware (on a Field Programmable Gate Array [FPGA])

• Next, develop a complex controller that can take advantage of the power of software (COTS processor + hardware)

• Then use the complex controller when possible, but switch to the simple one to preserve system liveliness

PhysicalComponents

Motherboard

CPU

MemoryRam

Field Programmable Gate Array (Xilinx ML505)

sensors actuatorssensors

actuators

Complex Controller

Decision Module

Safety Controller

PCIe Bus NorthBridge

Bus

FrontSideBus

LogicalMapping

Proof of Safety

• AADL is an architecture description language designed for real-time, embedded systems– Used by European Space Agency, Rockwell-Collins,

Lockheed Martin, Airbus, and others

• Systems can be instantiated from an AADL Model

• Safety properties of a model can be proven using model checking

System-Level Simplex Model

• We provide a System-Level Simplex AADL Model generator to generate an initial architecture design

• This model is modified as the design evolves

• The final AADL design can then be checked for violation of System-Level Simplex requirements

System-Level Simplex: Inverted Pendulum Testbed

Overview

Complex Controller

CPU with Linux OS

PCIe Bus

IO Module

Analog Input FPGA

Safety Controller DecisionModule

A/D Converter D/A Converter

Analog Output

Bus moduleType

checker

Inverted PendulumAn inverted pendulum is an unstable system that tries to maintain an upright rod by moving the base along a track (video)

We used the Quanser Q4 IP04 inverted pendulum for our testbed

The pendulum tells us the angle and track position which we convert to a digital signal with an A/D Converter (ADS7812P)

We output the digital voltage for the motor to use, which is converted to an analog using an A/D converter (DAC714P)

Hardware Components

Our hardware components run on a Xilinx ML505 Field Programmable Gate Array (FPGA)

The safety controller code can be generating in Matlab given the physical properties of the inverted pendulum

The decision module switches controllers when the pendulum is in danger of collapse. We can compute this state region with a Lyapunov stability function.

Decision Module

Recoverable Region:

Based on the dynamics of the controlled physical system, we can derive a stability envelope. Here, any state inside the red region is recoverable if we use the safety controller.

Safe Region:

When the state is in the green region, the system can tolerate aggressive action without immediately losing stability (we can use the complex controller).

FPGA

Safety Controller Envelope Calc

A/D D/A

Bus module Typechecker

PCIe Bus

Software

State Space

Software Components

Our complex controller runs on a x86 PC with Linux RK (a real-time Linux variant)

The software components are interfaced with the FPGA through the PCIe bus

Communication occurs through memory mapped I/O, where sensor and actuation values are viewed as memory on the FPGA

The complex controller is a modified version of the safe controller with various bugs to test the System-Level Simplex design

Implementation Results

By introducing bugs in the complex controller, we were able to verify that the System-Level Simplex Architecture protected the system from several potential failures

Conclusions and Future Work• We proposed a system-level simplex architecture

which provides reliability at the lowest (hardware) level

• We provide an AADL architecture generator and checker

• We demonstrated the System-Level Simplex Architecture on an Inverted Pendulum Control System, and empirically verified its functionality.

• The System-Level Simplex Architecture has fostered a funded collaboration with John Deere applied towards autonomous tractor control