RTAI: a Linux extension to hard real time applications

RTAI: a Linux extension to hard real time applications Politecnico di Milano - Italy

RTAI: a Linux extensionRTAI: a Linux extensionto hard real time applicationsto hard real time applications

Dipartimento di Ingegneria Aerospaziale,Dipartimento di Ingegneria Aerospaziale,Politecnico di Milano,Politecnico di Milano,

via La Masa 34, 20156, Milanovia La Masa 34, 20156, Milano, Italy, Italy

Paolo MantegazzaPaolo Mantegazza

([email protected])([email protected])


Presentation overviewPresentation overview

Operational view of the hard real time definition.Operational view of the hard real time definition.

Appropriate perspective of why custom off the shelf (COTS) solutions Appropriate perspective of why custom off the shelf (COTS) solutions are usable for high demanding local/distributed real time systems.are usable for high demanding local/distributed real time systems.

Advantages in adopting a free open source distributed real time Advantages in adopting a free open source distributed real time operating system.operating system.

The RTAI solution:The RTAI solution:

overview of the core system (RTHAL/ADEOS) and services;overview of the core system (RTHAL/ADEOS) and services; description of the implementation and features of RTAI middleware description of the implementation and features of RTAI middleware

layer (net_rpc);layer (net_rpc); presentation of RTAI-Lab: a comprehensive middleware tool to presentation of RTAI-Lab: a comprehensive middleware tool to

develop and monitor local/distributed RTAI applications.develop and monitor local/distributed RTAI applications. DIAPM and non-DIAPM RTAI applications.DIAPM and non-DIAPM RTAI applications.

Concluding remarks.Concluding remarks.


Soft and hard real timeSoft and hard real time

Computer Science literature definitions:Computer Science literature definitions:

a a soft real timesoft real time (SRT) system is characterized by its ability to (SRT) system is characterized by its ability to execute a task according to a desired time schedule on the execute a task according to a desired time schedule on the average (average determinism)average (average determinism)

video displayvideo display

an an hard real timehard real time (HRT) system (HRT) system embodiesembodies guaranteed timing, guaranteed timing, cannot miss deadlines and must have bounded latencies (strict cannot miss deadlines and must have bounded latencies (strict determinism)determinism)

plant controlled by a digital controllerplant controlled by a digital controller

Strict determinism is possible only with Strict determinism is possible only with dedicated CPUdedicated CPU, i.e. DSP and , i.e. DSP and DSP like microcontrollers, having a guaranteed interrupt latency in the DSP like microcontrollers, having a guaranteed interrupt latency in the range of a single/few execution cycles.range of a single/few execution cycles.


Hard real time using COTS hardwareHard real time using COTS hardware

GPCPUs have non deterministicGPCPUs have non deterministic

latencies and jitter due to:latencies and jitter due to: virtual memory and MMU;virtual memory and MMU; multi level caches;multi level caches; bus arbitration from bus arbitration from

intelligent I/O subsystems;intelligent I/O subsystems; high depth piped execution high depth piped execution

and speculation.and speculation.

GPCPUs can meet high GPCPUs can meet high requirements at requirements at unprecedented unprecedented cost/performance figurescost/performance figures (mega/giga FLOPS (mega/giga FLOPS capabilities).capabilities).

Even higher performances using Even higher performances using low cost Multi Processors and low cost Multi Processors and networked GPCPUs.networked GPCPUs.

Is it viable the solution of adopting COTS hardware, like general Is it viable the solution of adopting COTS hardware, like general purpose CPUs (GPCPUs), for HRT systems?purpose CPUs (GPCPUs), for HRT systems?


High precision profiling machine exampleHigh precision profiling machine example

MIMO controller at 10 KHz with MIMO controller at 10 KHz with some form of adaptivity, force some form of adaptivity, force control and active vibration control and active vibration suppressionsuppression

loop foreverloop forever

acquire data (sensors/commands)acquire data (sensors/commands)

compute control actioncompute control action

output data (actuators/logs)output data (actuators/logs)

wait for the next samplewait for the next sample

(latency computation)(latency computation)

Worst case latencies in the range of Worst case latencies in the range of few few s if the controller is run s if the controller is run standalone on a GPCPU.standalone on a GPCPU.

Typically the controller has to work Typically the controller has to work cooperatively in a prioritised fully cooperatively in a prioritised fully preemptable multitasking execution preemptable multitasking execution environment, e.g. for data environment, e.g. for data visualisation, monitoring, logging visualisation, monitoring, logging and communication.and communication.

Such a mix can lead to non Such a mix can lead to non deterministic latencies (jitter) up to deterministic latencies (jitter) up to 30 30 s (30% of the sampling period).s (30% of the sampling period).

A system having such a jitter can be A system having such a jitter can be still considered HRTstill considered HRT??


Hard real time specificationsHard real time specifications

The finishing and tolerances of our machined items verify our top The finishing and tolerances of our machined items verify our top expectations, around the clock, seven days a week.expectations, around the clock, seven days a week.

A comparison against any equivalent dedicated DSP implementation A comparison against any equivalent dedicated DSP implementation will show no difference in results.will show no difference in results.

Our Our real time specifications must be related to the actual system real time specifications must be related to the actual system bandwidthbandwidth and only indirectly to the sampling rate. and only indirectly to the sampling rate.

Our Our 10 KHz controller10 KHz controller will likely be controlling a will likely be controlling a 1 KHz bandwidth1 KHz bandwidth system for which a worst case 3% timing uncertainty will produce system for which a worst case 3% timing uncertainty will produce measurements and command errors that are guaranteed to be measurements and command errors that are guaranteed to be filtered, down to an acceptable noise level, by the filtered, down to an acceptable noise level, by the inherent system low inherent system low pass attenuationpass attenuation..

No clear cut between hard and soft real time but simply a continuous No clear cut between hard and soft real time but simply a continuous transition from low to high bandwidths.transition from low to high bandwidths.


Hard real time using FOSSHard real time using FOSS

Real time operating systems (RTOS) running on GPCPUs will afford Real time operating systems (RTOS) running on GPCPUs will afford equivalent performancesequivalent performances because these are dominated by the because these are dominated by the architectural features of GPCPUs.architectural features of GPCPUs.

At this point FOSS RTOSes can have an edge, with the added At this point FOSS RTOSes can have an edge, with the added advantage of full code sources availability.advantage of full code sources availability.

Is it possible to meet the required HRT specifications within an existing Is it possible to meet the required HRT specifications within an existing truly FOSS operating system such as Linux?truly FOSS operating system such as Linux?

Linux is a non preemptable UNIX like OS, with superb performances as Linux is a non preemptable UNIX like OS, with superb performances as a general purpose computational/desktop/server OS, but with limited a general purpose computational/desktop/server OS, but with limited soft and hard real time capability. However, it offers a very good base soft and hard real time capability. However, it offers a very good base to which HRT can be added.to which HRT can be added.

This goal is achieved easily setting a This goal is achieved easily setting a strict separation of hard and non strict separation of hard and non hard real time processes, while affording efficient communication and hard real time processes, while affording efficient communication and synchronization services to allow an effective interaction between the synchronization services to allow an effective interaction between the two environmentstwo environments. RTAI follows such a concept.. RTAI follows such a concept.


RTAI core system: RTHAL/ADEOSRTAI core system: RTHAL/ADEOS

A A Real Time Hardware Abstraction Real Time Hardware Abstraction LayerLayer (RTHAL) is installed that (RTHAL) is installed that gathers all the pointers to the time gathers all the pointers to the time critical kernel internal data and critical kernel internal data and functions into a single structure so functions into a single structure so that they can be dynamically that they can be dynamically substituted by RTAI when HRT is substituted by RTAI when HRT is required. Linux is changed to use required. Linux is changed to use what pointed in RTHAL for its what pointed in RTHAL for its operation.operation.

The The Adaptive Domain Environment Adaptive Domain Environment for operating Systemsfor operating Systems (ADEOS) (ADEOS) nanokernel is installed that nanokernel is installed that implements a pipeline scheme into implements a pipeline scheme into which every domain (OS) has an which every domain (OS) has an entry with a predefined priority. entry with a predefined priority. RTAI is the highest priority domain RTAI is the highest priority domain which always processes interrupts which always processes interrupts before the Linux domain, thus before the Linux domain, thus serving any hard real time activity serving any hard real time activity either before or fully preempting either before or fully preempting anything that is not hard real time.anything that is not hard real time.

RTAI is integrated into Linux through a RTAI is integrated into Linux through a kernel patchkernel patch and a and a series of series of add on programs add on programs (loadable modules) expanding the Linux kernel to (loadable modules) expanding the Linux kernel to hard real time.hard real time.

The patch can be RTHAL or ADEOS based.The patch can be RTHAL or ADEOS based.


RTAI schedulersRTAI schedulers

Fully preemptable.Fully preemptable.

Can be scheduled from within interrupt handlers so that Linux has no Can be scheduled from within interrupt handlers so that Linux has no chance of delaying any RTAI hard real time activity.chance of delaying any RTAI hard real time activity.

Allow to symmetrically work inter-intra both user and kernel space by Allow to symmetrically work inter-intra both user and kernel space by using the same set of APIs everywhere.using the same set of APIs everywhere.

Optimised for Optimised for kernel kernel uniprocessor (UP), uniprocessor (UP), kernel kernel symmetric symmetric multiprocessors (SMP)multiprocessors (SMP), kernel, kernel multi uniprocessor (MUP) multi uniprocessor (MUP), integrated , integrated UP/MP kernel/user space processing (LXRT)UP/MP kernel/user space processing (LXRT)..

Possibility of choosing between a base Possibility of choosing between a base periodic timingperiodic timing, with a fixed , with a fixed assigned time resolution tick, and an assigned time resolution tick, and an arbitrary timingarbitrary timing, allowing , allowing scheduling a task at the resolution of the available clock by firing a scheduling a task at the resolution of the available clock by firing a oneshot timer at the time instant imposed by the highest priority task oneshot timer at the time instant imposed by the highest priority task waiting on the timed list.waiting on the timed list.

Different scheduling policies: FIFO, Round Robin, Early Deadline First.Different scheduling policies: FIFO, Round Robin, Early Deadline First.


User space User space interruptsinterrupts

RTAI servicesRTAI services

MailboxesMailboxes

Binary/Binary/resource resource

semaphoresemaphore

BitsBitssynchronizatisynchronizati

onon

FIFOFIFOcommunicationcommunication

ss

TaskletsTaskletsand timersand timers

Distributed Distributed servicesservices(net_rpc)(net_rpc)

Distributed Distributed servicesservices(net_rpc)(net_rpc)

Direct Direct intertask intertask messagesmessages

User space User space interruptsinterrupts

Basic task Basic task managememanageme

ntnt


Distributed RTAI: net_rpcDistributed RTAI: net_rpc

Small (20 KBs) and effective real time middleware layer to allow native Small (20 KBs) and effective real time middleware layer to allow native usage of any of the RTAI APIs on distributed local/remote control nodes.usage of any of the RTAI APIs on distributed local/remote control nodes.

The integration is provided by just adding a node/port identifier in front The integration is provided by just adding a node/port identifier in front of any RTAI function call. Thus any application can be run on a single of any RTAI function call. Thus any application can be run on a single machine or on many networked machines without changing a single line machine or on many networked machines without changing a single line of source code.of source code.

Lean implementation of the RPC idea.Lean implementation of the RPC idea.

The scheme uses a stub (buddy, proxy, agent) task to execute the The scheme uses a stub (buddy, proxy, agent) task to execute the remote APIs.remote APIs.

Stubs and ports are statically connected to guarantee maximum Stubs and ports are statically connected to guarantee maximum determinism.determinism.

All services are executed in a strict synchronous All services are executed in a strict synchronous request_service/return_result moderequest_service/return_result mode, asynchronous usage is also possible, asynchronous usage is also possible..

Port requests/releases are not real time operation.Port requests/releases are not real time operation.


Distributed RTAI: net_rpcDistributed RTAI: net_rpc (cont.) (cont.)

Ports are owned by a task and a task can own multiple ports having Ports are owned by a task and a task can own multiple ports having different identifiers.different identifiers.

Capability of dynamically expanding the services at run time.Capability of dynamically expanding the services at run time.

Actual available support of low level net service through UDP, with Actual available support of low level net service through UDP, with dedicated connections and switches.dedicated connections and switches.

Sharing Linux networking possible but forgetting real time Sharing Linux networking possible but forgetting real time performances.performances.

Ethernet UDP protocol is non strictly deterministic but using a Ethernet UDP protocol is non strictly deterministic but using a dedicated high speed hardware guarantees acceptable performances dedicated high speed hardware guarantees acceptable performances for low to medium bandwidth, say up to 2 KHz, distributed controllers.for low to medium bandwidth, say up to 2 KHz, distributed controllers.

Higher bandwidth at an increased cost (GigaEthernet or SCI adapter) Higher bandwidth at an increased cost (GigaEthernet or SCI adapter) or using the free open source project RTNet.or using the free open source project RTNet.

Security?Security?


RTAI-LabRTAI-Lab

Provides a Provides a common structured framework to design, build, run and common structured framework to design, build, run and monitor locally/remotely any suite of RTAI-based single/multitasking monitor locally/remotely any suite of RTAI-based single/multitasking real time applicationsreal time applications, either specifically coded in a high level , either specifically coded in a high level procedural language (C/C++), or automatically generated by procedural language (C/C++), or automatically generated by proprietary packages (MATLAB/Simulink/Real-Time-Workshop), and/or proprietary packages (MATLAB/Simulink/Real-Time-Workshop), and/or fully open source projects such as SCILAB/Scicos/CodeGen.fully open source projects such as SCILAB/Scicos/CodeGen.

Allows two separate systems, the host and the target(s), to Allows two separate systems, the host and the target(s), to communicate for changing/tuning parameters on the fly and/or communicate for changing/tuning parameters on the fly and/or monitoring/logging signals.monitoring/logging signals.

The host is the machine where RTAI-Lab is executing in soft real time; The host is the machine where RTAI-Lab is executing in soft real time; the targetthe target((ss)) is/is/are the machineare the machine(s)(s) where the local/distributed where the local/distributed generated code runs in hard real time.generated code runs in hard real time.


Matlab caseMatlab case

Simulink modelSimulink model(model.mdl)(model.mdl)

Real-Time WorkshopReal-Time Workshopbuild processbuild process

1) Target Language1) Target LanguageCompiler - rtai.tlcCompiler - rtai.tlc

(model.c, model.h)(model.c, model.h)

2) Template Makefile2) Template Makefile- rtai.tmf- rtai.tmf

(model.mk)(model.mk)

Real timeReal timeexecutableexecutable

(model)(model)

Making processMaking process

make, gccmake, gcc

RTAI-Lab Graphical User InterfaceRTAI-Lab Graphical User Interface


DIAPM control applications (1)DIAPM control applications (1)

Tracking control of a flexible Tracking control of a flexible manipulatormanipulator

Structural flexibility of the links.Structural flexibility of the links.

Integrated active damping of structural Integrated active damping of structural vibrations.vibrations.

Few sensors (angular potentiometers at Few sensors (angular potentiometers at the joints and strain gauges along the the joints and strain gauges along the links) and actuators (direct drive links) and actuators (direct drive brushless motors) to limit weight and brushless motors) to limit weight and costs.costs.



Fatigue test systemFatigue test system

High flexibility in programming High flexibility in programming arbitrarily complex coordinated multi-arbitrarily complex coordinated multi-load histories.load histories.

Load signals generation, test results Load signals generation, test results logging, display and recordinglogging, display and recording

Optimisation of test conditions and Optimisation of test conditions and controller parameterscontroller parameters



MPWM control of the MPWM control of the vibrationsvibrations of a large space of a large space structurestructure

Modular beam-like (19 m) plastic tubes-Modular beam-like (19 m) plastic tubes-made (75 Kg) truss suspended from the made (75 Kg) truss suspended from the ceiling by means of six soft springs.ceiling by means of six soft springs.

Equipped with six pairs of on/off air jet Equipped with six pairs of on/off air jet thrusters operating within the horizontal thrusters operating within the horizontal plane, and six colocated capacitive plane, and six colocated capacitive accelerometers.accelerometers.

Multi PWM deadbeat predictive control Multi PWM deadbeat predictive control law.law.



Active control of Active control of combustion combustion instabilities on a Rijke instabilities on a Rijke tubetube



Reduction of vibrations Reduction of vibrations transmitted to helicopter transmitted to helicopter airframes using active airframes using active gearbox strutsgearbox struts

Agusta A109 mockupAgusta A109 mockup

Active gearbox struts provided with Active gearbox struts provided with surface bonded piezoelectric patchessurface bonded piezoelectric patches

Feedforward multi-input multi-output Feedforward multi-input multi-output FXLMS control algorithm.FXLMS control algorithm.


Non-DIAPM control applicationsNon-DIAPM control applications

Marble and granite automatic cutter machineMarble and granite automatic cutter machine

Ball bearing rings shape controlBall bearing rings shape control

Laser cutting machineLaser cutting machine

Robot controlRobot control

Universal measuring machinesUniversal measuring machines

Electric power conversionElectric power conversion

Electroerosion machinesElectroerosion machines

Plastic deformation machinesPlastic deformation machines


Concluding remarksConcluding remarks

Since its very beginning RTAI has grown in an impressive way in Since its very beginning RTAI has grown in an impressive way in robustness, services and programming tools.robustness, services and programming tools.

RTAI is a FOSS development effort with a team of developers RTAI is a FOSS development effort with a team of developers worldwide.worldwide.

RTAI numbers many academic and industrial applications and its RTAI numbers many academic and industrial applications and its usage is more and more expanding.usage is more and more expanding.

RTAI might still lack some of the bells and whistles of commercial RTAI might still lack some of the bells and whistles of commercial RTOSes, but it does not fear any performance challenge.RTOSes, but it does not fear any performance challenge.

Bugs are always there: with FOSS, if nobody cares, you are free to Bugs are always there: with FOSS, if nobody cares, you are free to loose some sleeping and fix bugs yourselves (feeling proud to loose some sleeping and fix bugs yourselves (feeling proud to contribute your fixes back to help the community).contribute your fixes back to help the community).

Documents

RTAI: a Linux extension to hard real time applications