Hardware/Software Co-design of Schedulers for

Real Time Systems

Jorge Ortiz

CommitteeDavid Andrews, Chair

Douglas NiehausPerry Alexander

Presentation OutlineBackground

Prior work in hybrid co-designFPGA hardware/software co-designScheduling

RTFPGA Project OverviewDesign approach & functionalityDesign implementation plan

Scheduler ImplementationHybrid co-design for KURT-Linux event schedulerSystem properties

Testing and AnalysisFunctionality validation of designData Stream testingConclusions and future work

Contributions

DesignMitchell Trope, Sweatha RaoImplementationSweatha Rao, Jorge OrtizTestingMitchell Trope, Sweatha Rao, Jorge Ortiz

Background

Prior Work in Hybrid Co-design

Average-case enhancementsWorst-case scenariosPlatform architectures for real time systemsSystems that support real-time constraints

System Co-DesignParallel systems

Multi-processor, FPGA, System-on-Chip, ASIC, Reconfigurable systems

Hybrid systemsExploit recursive patternsIncrease quality of serviceMeet real-time constraints

Real time operating systemsHardware support for the operating systemOS function migration

FPGA Hardware/Software Co-design

Desired system propertiesTimeliness, concurrency, liveness, interfaces, heterogeneity, reliability, reactivity, predictability and safetyAverage-case, worst case scenarios

System flexibility and performanceReconfigurableNo loss in performance

Scheduling

How to allocate which resources to whom and for how longHandling real-time eventsDesire for finer granularity for servicing interruptsOverhead costs due to interrupt handling and event schedulingDesign trade-offs

RTFPGA Project Overview

Project Motivation

Provide higher resolution and finer granularity for event handlingMinimize overhead processingMigrating key functions into hardwareTime keeping hardware supportHardware/software co-design of event scheduler for real-time hybrid operating system

Design ApproachMigrate shadow timer into FPGAImplement scheduler bookkeeping operationsProvide processing for enqueuing and dequeuing eventsAllow entering, sorting and deletion of eventsEnable access to all queued eventsMove scheduling algorithm into the hardware

Design Functionality

FPGA stores event information in local queueRun scheduling algorithmIdentify the next event to runSends event time to a match registerFree-counting hardware clock provides interrupt generation

Design Implementation PlanInterface ported KURT-Linux to FPGA componentsCreate FPGA-mapped registersCreate basic event queue storage, interface and functionalityImplement event schedulerDeliver FIQ interrupts to CPUSolve concurrency issues of hardware-based implementation

Scheduler Implementation

Hybrid Co-design for Kurt-Linux Event Scheduler

Forward event requests from OS to hardware through command registerIdentify request type and event information:execution time and reference pointerService request on event schedulerRaise interrupt flag upon execution timeKURT-Linux runs appropriate ISR with retrieved event information

Data Flow Chart in the RTFPGA Scheduler

Request to FPGASend request to schedulerIdentify requestAdd or delete in Block RAM storageSchedule events in parallelQueue expired events for CPU to read

RTFPGA Scheduler Modules

RTFPGA Scheduler Modules

Register MapperInterface to all registersInterface to internal functionality in the FPGA board Modular and portableTwo operational blocks - read and writeTiming, event information, scheduler commands, debugging information.

RTFPGA Scheduler Modules

Memory ManagerControl storage and handling of Block RAM-implemented event queueBlock RAM Address generation for event additionService Scheduler requestsDirty bits indicate usage of Block RAM addresses

RTFPGA Scheduler Modules

Block RAMDual-read/write port synchronous Block RAM FPGA storage memoryFirst port for Memory Manager requestsSecond port for continuously polling information in BRAM addresses for scheduling and searching purposes

RTFPGA Scheduler Modules

UtimeShadow of KURT Utime clock implementationSet for microsecond resolutionProvide match register for next scheduled event timeCreate FIQ interrupts to CPU

RTFPGA Scheduler ModulesQueue Minimum

Earliest Deadline First algorithm Reads event values output by Block RAM Output information for next event to be scheduledSend appropriate control signals

EDF pseudo-code

STARTSET next_event = max valueLOOPIF last deleted value = next_event

GOTO STARTIF last expired event = next_event

GOTO STARTIF no elements in queue THEN GOTO STARTELSE

READ new data from BRam event queueIF data = event (dirty bit is set)

READ scheduled timeIF scheduled time < next_event

next_event = scheduled timeEND IF

END IFEND IFGOTO LOOP

RTFPGA Scheduler Modules

Queue DeleteContinuously receive polled information from Block RAMPerform a deletion using linear search & comparisonFound or lost signal sent back to Memory ManagerCheck for concurrency with interrupt generation and event queue popping

RTFPGA Scheduler Modules

FIFO Block RAMFIFO queue for expired events that have not been read yet by CPUKeep track of number of expired eventsDe-queue until empty

Inter-Module Interfaces

System Properties

High interdependency of modulesConcurrency of deletions and interrupts solved by linear approachHighly cohesive scheduling algorithmTight coupling due to system requirements

Testing and Analysis

Functionality validation of design

Initial setup bootstraps FPGA timer shadowCheck for:

Event additionEvent deletionEvent scheduling

Event Addition Timing Diagram

Event Deletion Timing Diagram

EventSchedulingTimingDiagram

Integrated Scheduler Test1. Initial Set up2. Add Event1, Add Event2,

Add Event3, Add Event4, Add Event5

3. Find Minimum4. Delete Event2, Delete

Event35. Find Minimum6. Read current time from

FPGA timer registers (jiffy, jiffy_u)

7. Write event_time = jiffy + offset

8. Write ref_ptr = loop variable

9. Repeat the above three steps for loop variable = 0 to 2

10. Wait on jiffy + large offset

11. WHILE fifo_ref_ptr != x"00000000" (queue not empty)

12. Read fifo_ref_ptr(confirm all the three timers expired)

13. END WHILE14. Read jiffy & jiffy_u

Data Stream Kernel Interface Tests

0

1000

2000

3000

4000

5000

0 2 4 6 8 10 12 14 16 18 20 22

Execution time delay (us)

Num

ber o

f eve

nts

With Hardw are Components

Without

Timer Execution Delay for Software RTOS and Hardware/Software RTOS

Data Stream Kernel Interface TestsThe Software-based RTOS showed expected better performance

Lag between CPU and FPGA timers of 2-4 microsecondsFIQ handler schedules an IRQ, downgrading the interrupt priority

Data Stream Kernel Interface Tests[root@george dski]$./dskihists -i 900 -f 4 -h 1 -n 100Waiting for 900 seconds...Name of the histogram: HIST_TIMER_EXEC_DELAY Number of events logged: 10000 Minimum Value: 0 Maximum Value: 7137329 Lower Bound: 0 Upper Bound: 100

Num of Buckets: 102

Stability and Bounded Execution Time

Maximum execution delay for a software-based timer was in the range of millions of microsecondsIn the hardware version, all events in all tests were executed within the time delay range of 23 microsecondsHardware delay attributed to startup delays, worst case scenario checking, interrupt servicing, communication costs across ports

Event handling under different loads

Increase of 6.4% for events handled with a zero microsecond delay1.7% increase for all events handled within one microsecond12.6% increase after the serial port polling delay between 10-12 microseconds

FPGA Event handling

0

20

40

60

80

100

120

1 3 5 7 9 11 13 15 17 19 21 23 25

Time in microseconds

% o

f Eve

nts

Han

dled

10K events100 K Events

Device Utilization (V2P)

EDF scheduling functionLogic utilization: 11% SLICES, 17% of LUT, 22% Block RAM, Gate Count of 405,637Device utilization: 24% SLICES

Total FPGA SLICE Utilization83% or2510 out of 3008

Conclusions and Future Work

System behaviorWith Hardw are Components Aggregate

performance characteristics of both systems are comparableRobust base for hardware based event scheduler correctness, behavior and performance

0

2000

4000

6000

8000

10000

12000

1 3 5 7 9

11

13

15

17

19

21

23

Execution Delay (us)

Nu

mb

er

of

Ev

en

ts

Without

System characteristics

Modular and Portable, IP encapsulatedFlexible and reconfigurableCOTS Linux environmentTested in different architectures, ADI Engineering's 80200EVB and the Xilinx Virtex-II Pro

Future Work

Thread SchedulingHardware-based ThreadsHybrid ThreadsSmarter scheduler functionalityOnly necessary interruptions to CPU

Questions