121

The design of D.A.R.T.S.: A dynamic debugger for multiprocessor real-time applications.

Dr.ir.M.Timmerman, ir. F.J.A. Gielen. Dept. of Computer and Information Science, Royal Military Academy, Renaissance AV. 30, B-1040 Brussels (Belgium).

Abstract The correctness of execution of a real-time application depends upon the timing properties of the system.The DARTS functions implement a set of debugging func- tions which have a minimum and predictable influence on the temporal logic of the application. They pro- vide the programmer with crucial information on the temporal logic of the system (direct control flow), they perform watch functions on variables and control the invariant relations which may exist between application and system level objects. (indirect dataflow)

I. INTRODUCTION Real-time programming and embedded systems were the subject of a revolutionary evolution during the last few years. From small assembly programs running on 8-bit microprocessors they grew into fullgrown software engineering environments, using high level languages and multiprocessor RISC/CISC platforms to run their application. The complexity of today's real-time appli- cations clearly shows the need of a powerful set of tools that accompany the basic building bloc of a real-time system: the real-time operating system. Most of the focus in real-time system research has been directed to structured design and coding. However, since studies show that debugging software takes about half of de- velopment time, it is obvious that powerful debuggers are needed. Despite this reality, debug ing remains the black art of real-time programming [lf. This paper is the result of a research project (D.A.R.T.S. which is

tion Science of the Royal Military Academy in Brus- sels,Belgium. The D.A.R.T.S.- project (Debug Add on for Real-Time System) concentrates on specifying, de- signing and implementing a multiprocessor real-time de- bugger which can be integrated with existing real-time operating systems.

going on at the Department of Computer an d Informa-

11. THE DEVELOPMENT ENVIRONMENT.

A. The Hardware Platform.

host and target are single board computers on a VME- bus. (Fig.1) Multiple Workstations are connected by the departments LAN. Communication between the hosts and targets can either use a serial line, an Ethernet or the VMEbus (shared RAM).

B. The Software. The host system runs Unix System V while the tar- gets run MTOS-UX, a multiprocessor real-time operat- ing system [2]. At present work is going on to make an ORKID compliant interface for MTOS-UX. ORKID is the Open Realtime Kernel Interface Definition and has been proposed as a standard by the VFEA (VME- bus to Futurebus Extended Architecture) International Trade Organisation[4] (VITA). The ORKID subcom- mittee met with the IEEE POSIX P1003.4 Real-time Extensions Working group. The ORKID specification is accepted as a working paper and is assigned to the P1003.4 POSIX Working Group. ORKID functions and services are mapped onto POSIX specifications while ORKID interface specifications are being changed to match the syntax of the POSIX documents in structure and format. These services and facilities (like system calls and naming conventions) will execute the kernel functions outlined in the ORKID document, but with POSIX syntax. The host side of the DARTS system is written under X-Window and we use the TLI (Trans- port Level Interface) interface as the communication in- terface. (TLI is an I S 0 service definition ( IS0 8072)). The communication protocol between host and target is UDP/IP while the application level of D.A.R.T.S. is tailored after L.D.P., the Loader, Debugger, Protocol from the TCP/IP protocol suite.(Fig.2) The final ver- sion of D.A.R.T.S should run Unix System V4.0 on the host. an O.R.K.I.D. comdiant kernel on the target and standard transport services between host and target.

to support commuii'ication

Because our debugger must be able to work with exist- ing and future real-time products we decided to con- struct a real-time workstation which is built around standard hardware and software. Basically, the work- station is a host target cross development system. Both

oo18-9499/92$03.00 0 1992 IEEE

122

w

HOST SBC #1

DARTS

x11 LDP

TCP UDP I

i IP I

I I

WJ VME

TARGET TARGET TARGET #1 #2 #3

I

SYSTEM V U MASTER 0 SHARED RAM

I

M T O S - U X multiprocessor

i-) SHARED RAM

SHARED RAM

I

SHARED RAM

VMEbus

UNM HOST

Figure 1: The hardware platform.

MTOS TARGET I

I I

I 4

DEPARTEMENTAL LAN

Figure 2: The software model.

123

111. FUNCTIONAL ANALYSIS. A. State Of The Debugging Art. A survey of existing debuggers on classical software en- gineering platforms showed us that these tools basically offer the same functionalities. They provide the pro- grammer with a set of tools which allow efficient con- trol flow debugging and data flow debugging. They were designed for the debugging of the correctness of the logi- cal result of the execution of non-real time) application

to the real-time market, software vendors ported their, functionally unmodified, source code level debuggers to the real-time world. Real-time systems, however, show other characteristics and have other needs which are not covered by classical debuggers and although source code level debuggers have proven their merrits, they are not sufficient for real-time applications[2]. Eg.: we use a highly optimized compiler for the application code. If we want to use the source code debugger, we have to recompile the application with the ”no optimize” com- piler flag. Obviously this operation will have an impact on the temporal logic of the system.

tasks. Ever since high level \ anguages made their way

B. Real-time Properties Complicate Debug- ging. Despite the critical nature of real-time software most of the research in debugging has been directed to non real-time programs. The following properties of real- time applications complicate the debugging cycle of a product .: a) Logical dependence: While the logical execution of every individual task may be correct, the execution of the overall multitasking ap- plication may be incorrect. This is due to the fact that the tasks are logically dependent and work towards a common goal. Realtime applications are typically mul- titasking applications. b) Asynchronuous nature: Real-time tasks are driven by asynchronuous events of nondeterminate nature. The system state is funda- mentally time dependent but has to remain predictable even though (infinitely) many possible input states ex- ist. This makes formalizing debugger specifications very difficult. c) Timing properties: Since the correctness of execution of a real-time appli- cation depends upon the timing properties of the sys- tem, the implementation of a set of debugging functions must not degrade the target’s application. In extreme cases the debugging process may not influence anything a t all. The ”target under debug” must still be able to comply with the real-time requirements of the applica- tion. Above all the temporal logic of the application has to remain unchanged. d) System level coupling: A real-time application is layered on top of a real-time operating system and intensively uses its system ser- vices. A real-time operating system is an object ori- ented program: its semantic model defin,es a set of ob-

jects and a number of operations allowed on the object. The active objects are tasks and interrupt service rou- tines. Passive objects are memory objects, semaphores, mailboxes ... etc. The operations on the objects are de- fined in the manager for that type of object. (e.g. P() and V() operations on counting semaphores are defined within the semaphore manager.) The application level creates and uses system level objects which are man- aged by the operating system. So in much cases the real-time application is ”closely coupled” to the real- time operating system. For example, different kernel scheduling a1 orithms may lead to tasks being sched- uled differentfy, which may lead to possible erroneous behaviour of the overall application. This means that a real-time debugger has to be aware of the nature of our real-time operating system or even be part of it.

IV. INFORMAL REQUIREMENTS. A. Non Real-time Debugging Functions. The real-time multiprocessor debugger has to operate in a host target cross development system. Therefore the debugger has to be network based and distributed among the host and the multiple targets. It is not ab- solutely necessary that the debugger presents the code in source format but it must have access to the symbol table. In order to support classical debugging features the system has to implement the following set of debug- ging functions, which are the result of previous research and publications. [6,7] - Data transfer commands which transfer data between host and targets (Read, Write) - Task control commands control the state of the target system (Start, Stop, Resume, Suspend) - Management commands used to control resources in the target (Query, Allocate, Free) - Breakpoints and watchpoints. Although these functions are ”classical” their implemen- tation has to take the real-time nature of the system into account. E.g. The breakpoint mechanism must not halt the target.

B. Real-time Debugging Functions (D .A.R.T.S .). The real-time debugging functions must not alter the temporal logic of the application and they have to pro- vide the programmer with time related state informa- tion on the application and the operating sytem. In theory the first design objective can only be achieved if DARTS runs on a dedicated processor or on dedi- cated hardware. But even this is a necessary but insuf- ficient condition because DARTS needs access to objects which reside on the target system. Therefore it will read shared RAM via the VMEbus. So even though from the software point of view the temporal logic remains unchanged, underlying hardware mechanisms such as VMEbus arbitration cycles and VMEbus interrupt ser- vicing can interfere with the timing of the application. On the other hand, it will not always be possible to run DARTS on a dedicated processor due to space lim- itations in the workstation. Whenever DARTS shares

1 24

resources with the real-time application their implemen- tation has to be designed for bounded execution and minimal interference with the application. The pro- grammer must be able t o predict the exact influence of the debugging functions on the overall application. Therefore the execution of any DARTS function must be one hundred percent predictable.

V. DYNAMIC ASSERTIONS AT OPERATING SYSTEM LEVEL.

A. Selecting DARTS Implementation. The basic assumptions which led to the selection of DARTS implementation techniques are: a) The logical correctness of every subtask of the real- time application is confirmed and tested by non-real time debug functions within the limitations, inherent to those tools. (e.g. Tasks which have special 1 / 0 require- ments cannot be completely debugged without context .) b) The remaining real-time bugs are inherent to the runtime behaviour of the set of tasks. c) The close coupling to the real-time operating system implies that a lot of real-time defects can be detected by monitoring system level objects. (event monitor- inglinterprocess communcations monitoring, scheduler monitoring) d) The uncertainty principle: Any debugging function, which shares resources with the real-time application, changes the temporal logic of the system. In order to provide relevant information, the influence of the de- bugging functions must be predictable. The Method of Dynamic Assertions is particulary use- full for this kind of applications. Dynamic assertions in- sert statements into a program which monitor its state. Several dynamic assertion monitors have been succes- fully implemented by researchers. However they pro- cess source code and significantly degrade performance, which makes them less for real-time applications[3]. Our implementation of dynamic assertions will process exe- cutable code at the operation system level and will allow on-line software modification of a running application.

B. Description Of The Real-time OS. We have selected an ORKID compliant real-time kernel for our system. ORKID defines a standard program- ming interface to real-time operating systems. This in- terface consists of a set of standard system calls oper- ating on objects of standard types. ORKID does not dictate any implementation dependencies such as the scheduling algorithm. The ORKID specification covers basic real-time kernel functionality for embedded sys- tems: - Task control facilities. - Queues for multiple task priority management. - Semaphores. - Clock and Timer controls. - Memory management. - Interrupt Support. - Event Flags.

- Exception handling. By specifing DARTS as an extension to ORKID we stay at a high level of abstraction which makes the specifi- cation of DARTS system independent. However, since the POSIX P1003.13 study group is still working on the ORKID to POSIX mapping, we will continue our work on DARTS with MTOS-UX for 680x0 and 88x00 micro- processors until the POSIX Embedded Profiles (AEP’s) become available: APE-1: Single Processor/Single memory map systems. APES: Distributed Multiprocessor systems. APE-3: Networked Multiprocessor systems.

MTOS-UX provides a rich set of facilities and ser- vices and can be considered as a functional superset of ORKID. The standard object types defined by ORKID are shown in table 1.

- Tasks:

- Regions:

- Pools:

- Semaphores:

- Queues:

- Events:

- Note-pad:

- Clock: - Timers:

single threads of program execution in a node memory areas for dynamic allocation of variable sized segments memory areas for dynamic allocation of fixed sized buffers mechanisms used for synchronization and to manage resource allocation among tasks inter task communication mechanisms with implied synchronisation task specific event ex- ceptional with asynchronous exception service routines task specific integer locations for sim- ple, unsynchronised data exchange current date and time software delays and alarms

Table 1: ORKID objects. ,

All objects are named in the kernel space by an identi- fier. tid : task-id kernel defined task identifier rid : region-id kernel defined region identifier pid : part-id kernel defined pool identifier sid : sem-id kernel defined semaphore identifier. qid : queue-id kernel queue identifier nid : node-id system defined node identifier.

The standard operations on the objects are part of the corresponding object manager. The task manager is responsible for the following operations: task creation, task deletion, start and stop a task, suspend and re- sume, set task priority .... etc. Our debugger associates a DARTS monitor with every objectmanager of the real- time operating system. They are system level execution monitors which use the dynamic assertion method to report status and timing information to the host. Table 2 shows the DARTS which have to be available. Three levels of DARTS functions will be supported: - execution monitoring by time-stamping. - system object monitoring a t symbolic level. - dynamic assertions: conditional monitoring.

125

This mechanism will allow direct control flow monitor- ing and indirect data flow monitoring.

ORKID OPERATIONS DARTS MONITOR

scheduler scheduler monitor task manager task monitor region manager region monitor pool manager pool monitor semaphore manager semaphore monitor queue manager queue monitor event manager event monitor

Table 2: DARTS monitor

The programmer must have complete freedom in select- ing the kind of assertions that are active in the monitor. (e.g. only monitor suspend and resume operations on a task with a specific (tid)). Any DARTS monitor must be capable of being turned ON or O F F completely. Fi- nally the programmer must be able to specify the kind of information which will be reported if a monitor is as- serted: e.g. calling parameters, a time stamp on return values.

VI. CONFIGURING DART MONITORS INTO A SYSTEM.

A. Software Architecture Of A DART. a) Using Exception handlers. Since our systems are 680x0 based we can take advantage of the exception vector frame of the microprocessor for the implementa- tion of a DARTS monitor. Basically a DARTS module consists of an Exception handler and a task. The ex- ception handler is necessary because we insert DARTS monitors into the code as A-emulators (fig.3). The ex- ception handler is responsible for handling code substi- tion and starting the DARTS task whenever the address of the A-emulation is executed. The DARTS task han- dles the assertion and replies to the host. The priority of the DARTS task can be dynamically adjusted to min- imize interference with the real-time application. The main disadvantage is that the overhead of an exception handler can become important. b) Using No Operation Instructions.(NOP) This is a more general method, which is independent of the tar- et processor. The method was introduced by Bemmerl

B 1 who uses them in combination with a hardware mon- itor. Before the kernel is assembled to the object code we modify the appropriate source modules as follows.: At selected places in several object managers we will in- sert a few NOP instructions which will reserve enough place for a ” M 0 V E ” or a ” J S R instruction. Each inser- tion will be marked with a label which will be exported from the module. e.g. XREF D$SCHEDl

B. Static And Dynamic DARTS Configura- tion [9]. A DART monitor can be inserted into the operating system code in two different ways. a) The static method is an off-line method which uses a kernel preprocessor on the host for the insertion of the DARTS. DARTS monitors, contained in the DARTS li- brary, are activated by replacing NOPs with a branch or jump to the DARTS entry point. Labels, defining where the kernel code is to be modified, and the corresponding instructions are grouped in a collection file, the syntax of which is very much like that of an assembler: label, assembly-language instruction(s), data and symbols:

PATCHLAB: JSR DARTS 1

Preprocessing the kernel is actually a two-phase pro- cess. During the first phase, statements from the collec- tion file are transformed in a lower level syntax: labels (defining where to write) are passed without modifica- tion in the output flow; assembly-language statements and their arguments (defining what to write) are trans- lated in the corresponding op-code, and raw data, with symbol declaration and relocation directives.

PATCHLAB: 4EB9 0000 0000 . SYMBOL DARTSl . RRELOC 0002 DARTSl 0011

The corresponding translators, built with standard UNIX tools such as lex and yacc, are obviously pro- cessor dependent: those that we developed up till now are preasm68K, for the 68000 family of processors and preasm88K for the 88100 RISC processor (all from Mo- torola). They only implement the useful subset of the full instruction set, but they may easily be extended. During the second phase, the preprocessor itself, prep, which is a specialised tool for manipulating object files available on the host in COFF format (Common Ob- ject File Format), will patch the kernel code, modify- ing and expanding symbol and relocation info tables as needed; prep is processor independent. The modified kernel then passes through the UNIX linking loader Id, which extracts the activated modules from the DARTS library (fig.4). b) The dynamic method uses on-line software modifica- tion techniques. Conceptually the same operations as for the static case are executed but the DARTS con- figuration data are processed by an On line Software Modification Module (OSMOD) instead of the kernel preprocessor. This module supports the following algorithm: - ask the target, for a memory segment. - build an absolute image of DARTS function from the library. - download the DARTS modules

D$SCHEDl NOP

126

Manual mos m tSRC> -

a. before: line n-1: line n: line n+l : line n+2:

b. after: line n-1: line n: line n+2:

Definitions

A-emulation:

DARTS task:

Opcode Opcode Opcode Opcode

Opcode SAXXX Opcode

Exception Handler. Code substitution Wake up DARTS- task RTE

Assert Report Sleep

Figure 3: Code space on the target after the insertion of a DARTS monitor using A-emulation.

I I

Figure 4: Static configuration of DARTS.

127

- add an entry in the exception vector table for the A- emulator exception handler.(Only, if you are using ex- ceptions .) - Obtain exclusive access to the A-emulation, MOVE or JSR insertion address. - insert A-code, MOVE or JSR instruction. - release address

VII. THE FIRST DARTS PROTOTYPE.

A. DARTS Primitives [7].

The first implementation of DARTS primitives are ex- ecution monitors for the MC680XO series of micropre cessors. We have built a simple library with DARTS functions that act as software probes from within the MTOS-UX/68K kernel. The library is designed as a shareable library, which is completely put into the tar- get’s memory. During the elaboration phase the DARTS initialization function allocates a memory buffer. The buffer is structured as a chain of linked records of vari- able or fixed length. These records contain the raw debug data which will be transmitted to the host sys- tem by the DARTS reporter task. The user can choose two different buffer management strategies. The first one uses flow control on the buffer and variable length records. While the DARTS functions write data into the buffer, the DARTS reporter starts reading it and transmits the data to the host. The priority of the DARTS reporter has to be fine-tuned for minimal in- terference with the application eg. it could run on a

control on the buffer but has fixed length records. The DARTS always continue to send data into the circular buffer each time their address is executed. The DARTS reporter can either be periodically or conditionally ac- tivated. Both strategies have a different influence on the temporal behaviour of the system and it is the users responsibility to select the appropriate buffer manage- ment scheme for debugging his application.

dedicated processor). The secon 6 strategy uses no flow

B. New Application Domain: Porting real- time kernels to complex hardware[9]. MTOS-UX has been designed to support multiprocessor applications. Typically, the devices are tightly coupled single-board computers; that is, they share a blackplane and parts of global memory (fig.5). Because the kernel is reentrant it can be run on all CPU’s simultaneously, without interference. Whenever a system integrator has to port a real-time operating system to such an architec- ture, he can always rely on the local debugger monit,or and a serial port on every seperate single-board com- puter for the debugging. For the DARTS project we had to port MTOS-UX to the MVME188. This is a VMEbus module with four clusters of MC88100 RISC processors closely coupled with MC88200 cache MMU units. Ev- ery cluster contains one MC88100 and two MC88200. The four clusters are integrated in a HYPERmodule mezzanine module plugged into the main board via a multiplexed M-bus (fig.6). The M-bus is an enhanced

microprocessorbus, which has additional signals to sup- port multiple CMMU’s and fault checking. The board resident firmware monitor only operates in single pro- cessor mode. This means that once you have forked the four processor clusters, you can only have physical ac- cess to one processor at a time. The only way t o obtain information about the state and the flow of MTOS-UX on every single processor is the use of DARTS execution monitors. The software probes proved their usefulness for efficiently debugging this kind of complex systems.

C. Analysis Of The Scheduler Resynchro- nization. One of the most critical elements of a real-time kernel that has an important influence on the deterministic behaviour of a real-time application is the scheduling algorithm. The existing gap between the available ker- nels and the research results forces the application pro- grammer to look for a compromise between rewriting his own kernel and designing his application around a software component which is not optimized for his ap- plication. In the first case he will spend more time in developing and debugging his home-made system soft- ware than with his application. In the latter case he will have to use lots of tricks to get around the deficiencies of a poor product, which will make the maintenance and the upgrading of the application very hard and time- consuming. Because current real-time kernels very often offer a complete set of powerful1 system calls and come with comfortable and user friendly development tools, we believe that a good possible solution is to rewrite the scheduler while taking into account the results of advanced research in the area of real-time scheduling algorithms. The function of a scheduling algorithm is to determine, for a given set of tasks, whether a schedule l e . the

exists such that the timing, precedence, and resource constraints of the tasks are satisfied, and to calculate or determine the schedule if there exists one. Due to the intensive research activities in this area a large set of algorithms have been elaborated and tested. They span the complete spectrum of application needs start- ing from the simple static priority scheduler to the so- phisticated distributed dynamic deadline schedulers for hard real-time systems. MTOS-UX uses an advanced distributed priority sched- uler which integrates transparant CPU scheduling and automatic load balancing in a global priority scheme. Load balancing algorithms however do not consider the timing constraints of tasks. While rewriting the MTOS-UX scheduler we observed that it was possible to obtain state information about the scheduler from the application level. In MTOS, ev- ery service call and clock interrupt is a synchronization point for task scheduling. This made it very difficult to observe task migration and load balancing operations in a multiprocessor system. If you want to visualize an active task by sending a message to the system con- sole, you actually execute a service call which influences scheduler synchronization. So the only way to observe the internal states of the scheduler (which task is on

sequence and the time periods) for executing t (I; e tasks

YYEbus system

SBC 11 sac g2 GLOBAL RAY

Figure 5:Tightly coupled single-board computers on a common backplane.

VNEbus board: MVMEISB

M - bus

Figure 6: The board resident debug monitor of the MVME188 only operates in single processor mode.

which processor and in what state?) is to insert execu- tion monitors a t decision points of the scheduler. The execution monitor reports the scheduler queues for both local and global tasks and always returns the decision of the scheduler (i.e. what is the running task on this processor).

VIII. Conclusion And Future research. We have proposed a dynamic debugging technique, which gives us the possibility to trap errors specific to real-time applications. The DARTS monitors, which use the dynamic assertion method, can be used as ac- tivity monitors. In that case the DARTS gives the user crucial information on the temporal logic of the system (direct control flow). DARTS monitors can also perform watch functions on variables and control the invariant relations which may exist between application and sys- tem level objects (indirect data flow). In both cases the implementor has to take the basic design rules into con- sideration - the debugger is an extension of the real-time operating system. - influence on the timing behaviour of the application must be minimal. The futur research in the DARTS-project will concen- trate on the following topics: - specification of debugger semantics. - determination of the temporal behaviour of DARTS and their influence on the application. Although it is clear that DARTS monitors do not ad- dress all the aspects of real-time debugging, we are sure that its exsistence means a significant contribution to real-time programming and to the reduction of software development and maintenance costs.

129

Refer e nc es BEMMERL T. , Realtime high level debugging in host/target environments., Euromicro Symposium, Sept 86.

RIPPS D. , A Guide to Real-time programming., Prentice Hall Inc., Englewood Cliffs,NJ,1989.

DE BAERE J., Operating System Test Specifica- tions., UMI Dissertation Service, Ann Arbor (Michi- gan 48106),1987.

VITA, ORKID; open kernel Interface Definition VITA. , software sub commit tee, 1990.

WELLES C., R F C 909: Loader Debugger Protocol., 1984.

WARD R. , Debugging C ., 1986.

J.F. EL FOULY, Design of host resident DARTS tools: preasm, prep., Technical report R.M.A., Brus- sels, 1991.

VASSEUR N., DARTS functions for MTOS- UX/SSK, Technical report R.M.A., Brussels, 1991.

GIELEN F. Esoterica of implementation. The MTOS-UX/88K port to the MVME 188, Technical report R.M.A., Brussels, 1991.