Real-Time Linux Evaluation NASA Glenn Research Center Kalynnda Berens [email protected] Richard Plastow [email protected]

Real-Time Linux Evaluation

NASA Glenn Research Center

Kalynnda [email protected]

Richard [email protected]

SAIC @ NASA Glenn Research Center 2

Mission Success Starts With Safety

SAS, 2004

Research Overview

Compare real-time Linux variantsAre lower-cost alternatives good enough?

Provide projects with data to make informed decisions on operating systems

Use a standardized test approachAcross all real-time Linux variantsCompatible with previous VxWorks testing

Hardware differences prevent timing comparisons



SAS, 2004

Variants Chosen

RTLinux (free version) V3.2 pre3 RTLinux Pro (commercial) V2.0 RTAI V24.1.11 Kurt Linux 2.6.7 Kernel Jaluna and/or MontaVista



SAS, 2004

Embedded Emulators

Micro/Sys SBC2590100 MHz pentium32 MB ram768K Flash Disk

Embedded Planet EP405Power PC 200 MHz16 MB ram



SAS, 2004

58 Test Cases

User Space Kernel Space I/O use Remote Procedure Capabilities Inter-process Communication Stability



SAS, 2004

User Space Tests Memory

Shared Allocation Leaks Violation

Process Spawning Scheduling Deadlock Overhead

Stability Program Crashes Error Handling Stress

CPU Cache Memory Stack Scheduler



SAS, 2004

User Space Task Creation

User Space Task Creation Ave

635307.0

1016435.0

1395012

0.0

200000.0

400000.0

600000.0

800000.0

1000000.0

1200000.0

1400000.0

1600000.0

nan

ose

con

ds

RTLinuxPro

RTAI

RTLinuxFree



SAS, 2004

User Space Program Forking

Fork Time Average

19006487 19035921

4530189

0

2000000

4000000

6000000

8000000

10000000

12000000

14000000

16000000

18000000

20000000

na

no

sec

on

ds

RTLinuxPro

RTAI

RTLinux Free



SAS, 2004

Kernel Space Tests All User Space Tests Real Time Function

calls Real Time System

calls Configuration Timing

Priority Inversion Load/Unload Modules Real Time I/O Task Starvation Scheduler Accuracy



SAS, 2004

Timing Jitter - Harmonic

HarmonicMean Timing Jitter vs CPU Load

23016.124885.7 24101.6 23550.6

13242.7

32486.3

-5103.0 -4793.3-7074.8

-5498.9-6871.7

-5258.5

5214.3

-8717.4

3102.7

-602.9 320.3-1196.4

-15000.0

-10000.0

-5000.0

0.0

5000.0

10000.0

15000.0

20000.0

25000.0

30000.0

35000.0

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

CPU Load (%)

sig

ma

(ns) RTLinuxPro

RTAI

RTLinux Free



SAS, 2004

Timing Jitter - NonHarmonic

NonHarmonicMean Timing Jitter vs CPU Load

23016.1 27141.1

111081.4

363644.4 360855.2

-498891.4

-5103.0 -8143.9 -6688.7 -6314.6 -6049.8 -5517.45214.3

-107563.0

-255901.1

157828.8

24122.8 12544.4

-600000.0

-500000.0

-400000.0

-300000.0

-200000.0

-100000.0

0.0

100000.0

200000.0

300000.0

400000.0

500000.0

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

CPU Load (%)

sig

ma

(ns) RTLinuxPro

RTAI

RTLinux Free



SAS, 2004

Context Switch TimesAverage Context Switch Times

20709.653

1936292.353

26812.096

0

500000

1000000

1500000

2000000

2500000

nan

ose

con

ds

RTLinuxPro Context Sw itch

RTAI Context Sw itch

RTLinuxFree Context Sw itch



SAS, 2004

Hardware InterruptsHardware Interrupts Average Time

8546.065065

4810.214

10153.184

0

2000

4000

6000

8000

10000

12000

RTLinuxPro Interrupts

RTAI Interrupts

RTLinux Free Interrupts



SAS, 2004

Software InterruptsSoftware Interrupts Average

65283.585

16930.026

87545.728

0

10000

20000

30000

40000

50000

60000

70000

80000

90000

100000

RTLinuxPro Interrupts

RTAI Interrupts

RTLinux Free Interrupts



SAS, 2004

Kernel Task Creation

Kernel Task Creation Average

4915227.433

1015750.465

109564.0

0

1000000

2000000

3000000

4000000

5000000

6000000

nan

ose

con

ds

RTLinuxPro Task

RTAI Task

RTLinux Free Task



SAS, 2004

I/O Tests Remote Procedures

Serial Port Parallel Port Ethernet

UDP TCP/IP

USB File System Interrupts

Start a task on another machine

Start a task in user space

Access a remote database



SAS, 2004

RTAI Hard drive Write Times

IDE Hard Drive Write Performance

0

500

1,000

1,500

2,000

2,500

3,000

3,500

4,000

4 8 16 32 64 128 256 512 1024

Transaction Size (kB)

Tra

nsf

er S

pee

d (

kB/s

)

64

128

256

512

1024

File Size (kB)



SAS, 2004

RTLinuxPro Hard drive Write Times

IDE Hard Drive Write Performance

0

500

1,000

1,500

2,000

2,500

3,000

3,500

4,000

4,500

4 8 16 32 64 128 256 512 1024

Transaction Size (kB)

Tra

nsf

er S

pee

d (

kB/s

)

64

128

256

512

1024

File Size (kB)



SAS, 2004

Inter-Process Communication

Shared memory space First-In-First-Out (FIFO) buffers Mutexes Tokens Critical sections



SAS, 2004

Intertask Messages

Intertask Message Average Time - 1 Byte Message

38923.7 38835.4 38746.5

19922.0 19782.3 20457.8 20072.6

31021.7 31726.6 31368.6 31197.5

38518

15000

20000

25000

30000

35000

40000

45000

1024 2048 8192 16384

Buffer Size

na

no

se

co

nd

s

RTLinuxPro

RTAI

RTLinux Free



SAS, 2004

Semaphores

12288.955130941146610483.12

15686

6683

19519.68

42784

17056

0

5000

10000

15000

20000

25000

30000

35000

40000

45000

Semaphore Get from another Task

"RTLinuxPro Average"

"RTLinuxPro Max"

"RTLinuxPro Min"

"RTAI Average"

"RTAI Max"

"RTAI Min"

"RTLinuxFree Average"

"RTLinuxFree Max"

"RTLinuxFree Min"



SAS, 2004

Semaphores

3967.109

8019

39392866.04

9335

2612

6147.872

30304

5952

0

5000

10000

15000

20000

25000

30000

35000

Semaphore Get and Release


"RTLinuxPro Max"

"RTLinuxPro Min"

"RTAI Average"

"RTAI Max"

"RTAI Min"


"RTLinuxFree Max"

"RTLinuxFree Min"



SAS, 2004

Semaphore

3963.608

20309

39192791.671

10159

2662

6621.92

35072

5952

0

5000

10000

15000

20000

25000

30000

35000

40000

Semaphore Release and Get


"RTLinuxPro Max"

"RTLinuxPro Min"

"RTAI Average"

"RTAI Max"

"RTAI Min"


"RTLinuxFree Max"

"RTLinuxFree Min"



SAS, 2004

Working with projects?

Results are available for project use Potential Applications

Projects will have “hard data” upon which to base a decision to use/not use a real-time Linux operating system

Lessons learned in working with the operating Systems are documented



SAS, 2004

Pass the information to them! Availability

Current and final results available through SARP

Final results will be also be submitted to a Linux journal

Barriers to researchSome OS software is a moving targetTest case adaptation can be time consuming

for other Linux-based OS’s



SAS, 2004

Conclusions The three real-time Linux variants are

Stable Support many processors Require a learning curve

Documentation was often not complete Good support on user forums (answers in < 24 hrs)

RTLinux Pro was easy to install and run, with good documentation

VxWorks timing comparison not possible Testing done on borrowed hardware



SAS, 2004

Which Real-time Linux is best? Depends on your system requirements RTLinux (free and Pro)

Best context switch times Best kernel task creation times Least timing jitter for non-harmonic tasks at high CPU usage Pro – best documentation, easiest to install

RTAI Best interrupt response Best intertask messaging (single and 500 bytes) Least timing jitter for harmonic tasks at high CPU usage Allows user-space “soft real-time” tasks

Documents

Real-Time Linux Evaluation NASA Glenn Research Center Kalynnda Berens [email protected] Richard Plastow [email protected]