Introduction to Inter-Processor Communication (IPC) for KeyStone and Sitara™ Devices

1

PresenterPresentation NotesHello everyone, thank you for joining this training presentation on Introduction to Inter-Processor Communication (IPC) for Keystone and Sitara Devices. The IPC software package is designed to hide the lower layer hardware complexity of the multi-core devices and help users to quickly develop applications for data transfer between the cores or devices. It also provides a common API interface across all supported platforms, and maximize application software reuse.

Agenda • IPC 3.x Overview

– MessageQ Architecture – Modules

• Library and daemon

• RPMsg and Resource Table

• Examples

• Benchmarking

• For More Information

2

PresenterPresentation NotesIn this presentation, we will discuss the latest IPC 3.x release messageQ software architecture. MessageQ is the most recommended messaging API in IPC, then we will talk about the modules in the IPC package.

After checking out the IPC feature offerings, we will go through the procedure of how to build the IPC libraries for SYS/BIOS, for High level Operating system like Linux, and daemon that runs on linux. By default the IPC libraries and daemon are not pre-built and you need to build for a specific platform that you will be working on.

Then, we will briefly look at the RPMsg framework including remoteproc and rpmsg Linux kernel drivers. RPMsg is the key component to allow host control and communicate with remote processors, and the resource table is the construct to define the remote processor resource, so remoteproc driver can properly configure and bring up remote processors.

In addition, we will take a look at the IPC examples that are bundled with IPC release. These examples demonstrate the IPC configuration, API usage and data flow, users can learn from it and take it as a starting point to develop own application.

Finally we will talk about the benchmarking of IPC messageq, the latency of transferring data with IPC API.

IPC 3.x Overview: Architecture

3

PresenterPresentation NotesLet’s get started with the IPC 3.x release feature overview, messageq software architecture. For the purpose of this training, we will be referring to 66AK2H as keystone device and AM57x as Sitara device. IPC supports quite a few devices, such as, OMAPL138, OMAP54XX, DRA7XX, 66AK2G, 66AK2E, TCI663x,, TDA3XX. In terms of IPC build, some of the device names are interchangeable, for example 66Ak2H and TCI6638, AM57x and DRA7XX.

Multicore Processing

4

TI’s KeyStone and Sitara AM57x devices are built to meet the complex processing needs of modern embedded products. While the number of diverse processors and rich connectivity peripherals on these devices provides flexibility, it also challenges software developers to fully exploit the multicore processing potential.

PresenterPresentation NotesThis slide shows the system block diagram of 66AK2H and AM572x devices. As you can see these devices includes multiple processors (A15, M4, DSP…), and they also include rich connectivity peripherals, such as, HyperLink, SRIO, PCIE…(+ slide text).

Why IPC? To shorten the software development cycle and accelerate the time to market, TI provides a solution for inter-processor communication between homogenous and heterogeneous processor on the device with various transports.

5

AM57x

AK2H

AK2H AK2H

PresenterPresentation NotesIn order to shorten…

The two block diagrams in the slide illustrate the data flow and transports of IPC for inter-processor communication and inter device communication. For inter processor communication, shared memory is the most used communication transport. TI also provides queue manager subsystem (QMSS) transport for certain devices that greatly reduce the message passing latency. For inter device communication, Serial RapidIO is the only transport that is supported on keystone device currently. In the training slides, we will not cover the details of each transport, user can refer to TI Processor SDK user guide to understand the implementation and configuration.

IPC 3.x Overview • IPC 3.x is an evolution of the IPC product in the TI Processor SDK (Software

Development Kit): – Abstracts the lower layer of processor fabric connection – Provides a set of modules and APIs to facilitate communication:

• Between cores in a multi-processor environment (inter-processor) • To other threads on same processor(inter-process) • To peripherals (inter-device)

• IPC can be used to communicate in a multi-processor environment with: – Other threads on the same processor – Threads on other processor running SYS/BIOS – Threads on other proccessor running an HLOS (e.g., Linux, QNX)

6

PresenterPresentation NotesNow let’s look at the latest IPC 3.x release comes from TI’s Processor SDK RTOS software package. The Processor SDK is a unified software platform for TI embedded processors providing easy setup and fast out-of-the-box access to benchmarks and demos. It consolidates all platform specific software into a single software component. The IPC 3.x is an evolution of …

http://processors.wiki.ti.com/index.php/Category:PROCESSOR-SDKhttp://processors.wiki.ti.com/index.php/Category:PROCESSOR-SDK

Current IPC Releases

7

IPC releases newer than IPC 3.40 are now distributed as part of various Processor SDKs.

See the Processor SDK for RTOS download pages for a list of supported platforms.

PresenterPresentation NotesIn the past, IPC release can be downloaded and installed as a standalone package. Now to ensure the Processor SDK modules compatibility, the IPC release newer than IPC 3.40 are distributed as part of …

Although the Processor SDK RTOS packages are provided to target a various of supported platforms, the IPC release included in the packages are the same release with no built-in IPC libraries, that requires users to define and build for the platform after the IPC is installed.

http://processors.wiki.ti.com/index.php/Processor_SDK_RTOS_Getting_Started_Guide

IPC Open Source Project

8

IPC is an open source project currently managed with git and maintained at http://git.ti.com/ipc/ipcdev. Full source browsing, including all changes, are available at http://git.ti.com/cgit/cgit.cgi/ipc/ipcdev.git.

PresenterPresentation NotesIPC is an open source …

http://git.ti.com/ipc/ipcdevhttp://git.ti.com/cgit/cgit.cgi/ipc/ipcdev.git

IPC MessageQ Software Architecture: AK2H

9

Device 1

Linux IPC MessageQ

TransportLinuxQmss

Device 2 ARM Cortex-A15 Linux Thread/Process A

DSP X

IPC MessageQ

TransportSrio

ARM Linux

Linux IPC MessageQ

TransportLinuxSrio

ARM Linux Thread/Process B

Linux IPC MessageQ

TransportLinuxQmss

TransportLinuxSrio

SYS/BIOS IPC MessageQ

TransportQmss TransportSrio

DSP X

TransportRpmsg TransportRpmsg

Linux Kernel

PresenterPresentation NotesThe slide shows the IPC messageQ software architecture for 66AK2H device. As we can see from the diagram, IPC are supported with Linux and SYS/BIOS, and message passing can be through various of transports (SRIO, QMSS, Rpmsg). Rpmsg and Qmss are the two transports for inter processor communication. SRIO is the transport that can used to pass message between devices. IPC supports registering two different transports (primary and secondary) for an application in order to pass message within processors and within devices.

Device 1

IPC MessageQ Software Architecture: AM57x

10

DSP X / IPU M4

Linux IPC MessageQ SYS/BIOS IPC MessageQ

TransportRpmsg TransportRpmsg

Linux Kernel

ARM Cortex-A15 Linux Thread/Process A

PresenterPresentation NotesThe slide shows the IPC messageQ software architecture for 66AK2H device. As we can see from the diagram. Rpmsg is the only transport supported between host A15 Linux and slave DSP and IPU M4 processors.

IPC 3.x Overview: Modules

11

PresenterPresentation NotesWe have now checked the high level functions of IPC release. Let’s look at the individual modules provided by the IPC software package. IPC consists of a set of APIs, modules to facilitate the message passing as we have discussed.

IPC SYS/BIOS Modules

12

Ipc

MessageQ

Notify MultiProc

SharedRegion

GateMP

NameServer

HeapMemMP HeapBufMP

ti.sdo.ipc ti.sdo.utils

List

PresenterPresentation NotesThe IPC modules for SYS/BIOS are listed in the two categories: ti.sdo.ipc and ti.sdo.utils which corresponds to the IPC directory: packages/ti/sdo/ipc and packages/ti/sdo/utils.

The ti.sdo.ipc provides the main IPC modules including the ipc, MessageQ, HeapMemMP…Here the ipc module in the ti.dos.ipc box is responsile for IPC initialization and managing processor synchronization.

The ti.sdo.utils provides the utilities Multiproc, Name server, List, that are used by the main IPC modules.

We will explain more details of these modules and utilities in next few slides.

IPC HLOS Modules

13

Ipc

MessageQ

NameServer

GateMP

MultiProc

PresenterPresentation NotesTo reduce the complexity of APIs, IPC for high level operating system provides less modules than in SYS/BIOS, it has ipc, Message…

IPC Modules Defined • Ipc initializes and synchronizes IPC.

• MessageQ sends and receives messages.

• NameServer manages the distributed name/value database.

• Notify sends and receives event notifications.

• MultiProc manages processor identification.

• SharedRegion manages shared memory address translation.

• GateMP (Multi-Processor Gate) enforces local and remote context protection.

• Heap*MP allocates multi-processor memory.

• ListMP creates lists of objects, implemented as a doubly-linked list.

14

PresenterPresentation NotesHere are the definitions for each IPC modules. Let’s dive into the highlighted modules to see the module APIs, configuration, priority, data structure, data flow and function call flow.We will look into the ipc modules.

Ipc Module • Ipc is used to initialize IPC and synchronize with other processors.

• An application that uses IPC APIs--such as MessageQ--must include the Ipc module header file and call Ipc_start() before any calls to IPC modules.

• Two startup protocols: – Ipc.ProcSync_ALL option indicates all processors start at same time. – Ipc.ProcSync_PAIR option indicates host processor starts first.

• Configuration: – Ipc.procSync property configures startup protocol. – When using Ipc.ProcSync_ALL, Ipc_attach is called internally from Ipc_start,

application should never call Ipc_attach.

15

PresenterPresentation NotesIpc is used …

The Ipc.procSync property configures the start up protocols – either Sync_all or Sync_pair depends on application scenario.

Note when using

MessageQ Module • MessageQ module supports the structured sending and receiving of variable length messages.

• Each message queue has one reader and can have many writers. A thread may read from or write to multiple message queues. Timeouts are allowed when receiving messages.

• Reader: The single reader thread calls MessageQ_create(), MessageQ_get(), MessageQ_free(), and MessageQ_delete().

• Writer: Writer threads call MessageQ_open(), MessageQ_alloc(), MessageQ_put(), and MessageQ_close().

• Priorities: – MessageQ_NORMALPRI = 0 – MessageQ_HIGHPRI = 1 – MessageQ_URGENTPRI = 3

16

PresenterPresentation NotesMessgeQ is the recommended module for message passing. Message module supports…Internally the highest priority queue will be first processed.

MessageQ Header typedef struct { Bits32 reserved0; /*!< reserved for List.elem->next */ Bits32 reserved1; /*!< reserved for List.elem->prev */ Bits32 msgSize; /*!< message size */ Bits16 flags; /*!< bitmask of transport id, priority */ Bits16 msgId; /*!< message id */ Bits16 dstId; /*!< destination queue id */ Bits16 dstProc; /*!< destination processor id */ Bits16 replyId; /*!< reply id */ Bits16 replyProc; /*!< reply processor */ Bits16 srcProc; /*!< source processor */ Bits16 heapId; /*!< heap id */ Bits16 seqNum; /*!< sequence number */ Bits16 reserved; /*!< reserved */ } MessageQ_MsgHeader;

17

PresenterPresentation NotesThis slide shows the IPC MessageQ header data structure. Application can define its own message payload but the header must be included as part of the message. The messageQ header is 32 byte fields that include the information about the message that is transferred. Users can call certain APIs to configure the messageQ header for various applications.

Using MessageQ

18

MessageQ_open MessageQ_alloc MessageQ_create

IPC NameServer

MessageQ_Msg

MessageQ_ MsgHeader

Data

MessageQ_put

Transport_put Transport Medium

MessageQ_put

Transport_isr

MessageQ_get

MessageQ_Msg

MessageQ_ MsgHeader

Data

MessageQ_Msg

MessageQ_ MsgHeader

Data

QueueId

MessageQ_Handle

Msg List

Processor X (Writer)

Processor Y (Reader)

QueueId

MessageQ Data Movement

PresenterPresentation NotesThe slide shows the data and function call flow of MessageQ. The left is the processor - writer. The right is the processor – reader.When the application starts, the messageq reader first call MessageQ_create to create a queue with a name string for the writer to send message. The name string will be stored in the NameServer. The MessageQ_create returns a MessageQ_Handle, MessageQ_get() using the MessageQ_Handle receives messages from the created queue. After messageQ is created, the reader start to wait for new message by calling MessageQ_get.

When the queue is created, the writer calls MessageQ_open with the name string that matches with the name of the created object to get the queue Id.

The message was allocated by calling MessageQ_alloc. After the message is ready, it can then be sent out by calling MessageQ_put().

MessageQ_put underneath will call Transport_put to send the data out via a transport layer.

When there is new data available in receive buffer at the reader side, the Transport_isr will be triggered and store the message into the internal MessageQ list, according to the queueID and priority parsed from the messageQ header, the message will be stored in different priority queues, so that the reader’s MessageQ_get will be unblocked and receive the new message that is designated to from the writer.

This completes a message passing cycle.

GateMP • A GateMP instance can be used to enforce both local and remote context protection.

• Entering a GateMP can prevent preemption by another thread/process running on the same processor and simultaneously prevent a remote processor from entering the same gate. GateMPs are typically used to protect reads/writes to a shared resource, such as shared memory.

IArg key;

/* Enter the gate */

key = GateMP_enter(gateHandle); …

/* Leave the gate */ GateMP_leave(gateHandle, key);

19

PresenterPresentation NotesGateMP instance is designed to enforce both local and remote context protectionmessageQ internal has context protection, for other applications that needs to ensure the context and critical data section not being corrupted, GateMP may be needed. GateMPs are typically used to protect reads/writes to a shared resource, such as shared memory.

Besides the GateMP initialization/creation functions, the two APIs for entering and leaving the Gate is listed here.

GateMP Ladder Diagram

20

GateMP Creator

GateMP_open

Failure: Gate not created

GateMP_create

GateMP Handle GateMP_enter

GateMP_open

GateMP Handle GateMP_enter

GateMP_leave

GateMP_leave GateMP_enter

GateMP_close

GateMP_leave

GateMP_delete Initialized gate

GateMP GateMP Opener

Blocked gate

Entered gate

PresenterPresentation NotesThis slide shows the GateMP call flow. The GateMP call flow is pretty straight forward.The black solid line represents an initialized gate.The red line represents a blocked gate.The green line represents that a thread enters a gate.

Before a Gate is created, if the GateMP opener tries to open a Gate not being created, the GateMP_open will fail.When a gate is created, GateMP will return a handle. Note when you create a gate, shared memory is initialized, but the GateMP object is created in local memory. Only the gate information resides in shared memory.

Once an gate instance is created on a processor, the gate can be opened on another processor to obtain a local handle to the same instance.

The GateMP module uses a NameServer instance to allow a remote processor to address the local GateMP instance, it uses a user-configurable string value as an ID instead of a potential dynamic address value.

When a thread has entered a gate, the caller has exclusive access to the data protected by this gate.

GateMP_leave() may only called by a thread that has previously entered this gate via GateMP_enter()GateMP_close() frees a GateMP object stored in local memory.

MultiProc Module • IPC MultiProc module assigns unique ID to each processor in system

• Cluster implementation to support multi-device systems – Each device must have mutually exclusive base cluster ID – All devices must have a MultiProc mapping such that each processor ID is unique

• A system config can be simple, as shown in this example: /* describe the processors in the system */ var MultiProc = xdc.useModule('ti.sdo.utils.MultiProc'); //MultiProc.baseIdOfCluster = 0;

//MultiProc.numProcessors = 2; MultiProc.setConfig(“DSP1", ["HOST", “DSP1"]);

• Or a system config can be complex, as shown on the following slide …

21

PresenterPresentation NotesMultiProc module is a utility in IPC to manager processor IDs in a device or cluster.Cluster is implemented to support multiple devices in a system, and the cluster config parameters include:baseID of clusternumProcessors.numProcsInCluster

IPC Multi ….A systm configuration can be as simple as shown in the slide that has two processors only, Host A15, and DSP.A system can be also complex as the following slide shows.

MultiProc Mapping

22

Device A MultiProc Mapping numProcessors = 23 numProcsInCluster = 9 baseIdOfCluster = 0

Device B MultiProc Mapping numProcessors = 23 numProcsInCluster = 9 baseIdOfCluster = 9

Device C MultiProc Mapping numProcessors = 23 numProcsInCluster = 5 baseIdOfCluster = 18

Proc ID: 0

Proc ID: 1

Proc ID: 2

Proc ID: 3

Proc ID: 4

Proc ID: 5

Proc ID: 6

Proc ID: 7

Proc ID: 8

ARM Linux DSP RTOS

Proc ID: 9

Proc ID: 10

Proc ID: 11

Proc ID: 12

Proc ID: 13

Proc ID: 14

Proc ID: 15

Proc ID: 16

Proc ID: 17

ARM Linux DSP RTOS

Proc ID: 19

Proc ID: 20

Proc ID: 21

Proc ID: 22

DSP RTOS

Proc ID: 18

ARM Linux

PresenterPresentation NotesThe slide shows a complex system that has a multiProc module defined withtotal number of processors 23 and The number of processors 9 in a cluster.The based ID of cluster is device dependent, can be 0, 9, 18 as shown in the picture.

Library and daemon

23

PresenterPresentation NotesNow that we have taken a look at the IPC module offerings, let’s move on to look at how to build the IPC libraries, and daemon which maintains processor-wide state on the HLOS.

IPC Library Build: products.mak • The current IPC release supports multiple devices. The IPC libraries for each device are

NOT pre-built.

• IPC release contains a products.mak file that specifies the necessary paths and options to build IPC for the various supported OS.

Edit products.mak and set the following variables:

• Variables used by both Linux-side and BIOS-side build scripts: – PLATFORM (Optional) identifies the device to build.

• Variables used by Linux-side build scripts: – TOOLCHAIN_INSTALL_DIR identifies the path to the ARM Linux cross-compiler toolchain. – TOOLCHAIN_LONGNAME defines the long name of the device toolchain (e.g., arm--linux-gnueabihf). – KERNEL_INSTALL_DIR identifies the location of the Linux kernel installation. – DRM_PREFIX (Optional) – CMEM_INSTALL_DIR (Optional)

• Variables used by BIOS-side build scripts: – XDC_INSTALL_DIR identifies the path to the TI XDCTools installation (e.g., c:/ti/xdctools_). – BIOS_INSTALL_DIR identifies the path to the TI SYS/BIOS installation (e.g., c:/ti/bios_). – ti.targets. identifies the path to the TI toolchain for the device.

(e.g., c:/ti/CCS/ccsbase/tools/compiler/c6000_)

24

PresenterPresentation NotesIPC release supports a variety of devices. The supported devices are listed in the products.mak file.Many build variables in the products.mak file need to be configured before building the IPC libraries for a specific device.These variables listed in products.make covers both Linux and BIOS build, include tool chain paths, Linux kernel paths, XDC, and BIOS installation paths etc.

IPC Library Build: products.mak example

DEPOT = /home/user/ti PLATFORM = DRA7XX DESTDIR = /home/user/ti/ipc_3_42_00_02/generate TOOLCHAIN_LONGNAME = arm-linux-gnueabihf TOOLCHAIN_INSTALL_DIR = /home/user/gcc-linaro-4.9-2015.05-x86_64_arm-linux-gnueabihf TOOLCHAIN_PREFIX = $(TOOLCHAIN_INSTALL_DIR)/bin/$(TOOLCHAIN_LONGNAME)- KERNEL_INSTALL_DIR = /home/user/ti/ti-processor-sdk-linux-am57xx-evm-02.00.02.11/board-support/linux-4.1.18+gitAUTOINC+01c1359baa-g01c1359 CMEM_INSTALL_DIR = /home/user/ti/ludev XDC_INSTALL_DIR = $(DEPOT)/xdctools_3_32_00_06_core BIOS_INSTALL_DIR = $(DEPOT)/bios_6_45_01_26 CCS = /home/user/ti/ccsv6_1_2/ccsv6/tools/compiler ti.targets.elf.C66 = $(CCS)/ti-cgt-c6000_8.1.0 ti.targets.arm.elf.M4 = $(CCS)/ti-cgt-arm_5.2.5 gnu.targets.arm.A15F = $(CCS)/gcc-arm-none-eabi-4_8-2014q3

25

PresenterPresentation NotesThe slide shows an example product.mak file to build IPC libraries and daemon for DRA7XX which is also applicable to AM57x devices.

As you can see from the slide, we have different toolchain defined for C66, M4 and A15 cores. Also we specified the XDC, and BIOS version.

IPC Library Build Commands • Linux-side build:

– The IPC package provides a GNU makefile (ipc-linux.mak) to configure the Linux-side build, using the options and component paths set in the products.mak file. To configure the build, issue the following command:

make -f ipc-linux.mak config

– Or to specify a platform:

make -f ipc-linux.mak config PLATFORM=66AK2G

– And then make; sudo make install

• SYS/BIOS-side build: – After editing products.mak, issue the following command:

make -f ipc-bios.mak all 26

PresenterPresentation NotesThe slide show the commands to build Linux and SYS/BIOS side IPC libraries

For Linux side build, the IPC package provides a GNU make file using the options and component paths set in the products.mak file

IPC Daemon • IPC provides system-wide services across multiple applications, and utilizes low-level

system hardware (e.g. interrupts and shared memory). To facilitate these services, IPC uses a user-space daemon (LAD) and several Linux kernel device drivers.

• At startup, the daemon creates a FIFO (named pipe) to listen for connection requests from other user-mode clients. When a connection request comes in, the daemon opens a dedicated FIFO for sending responses to the client.

• LAD resides on the target filesystem (typically in /usr/bin/) and typically starts running in the background during kernel bootup. LAD logs the IPC state in the file lad.txt in the /var/tmp/LAD/ directory.

27

PresenterPresentation NotesThe IPC user space daemon, called LAD, is built along with IPC libraries.the daemon creates a FIFO to listen for connection requests from other user-mode clients. When a connection request comes in, the daemon opens a dedicated FIFO for sending responses to the client.

Since the LAD log file stores the IPC state from the beginning, usually it’s the 1st check point for debugging a IPC application.

IPC Daemon Initialization Initializing LAD... [0.003995] opening FIFO: /tmp/LAD/LADCMDS [546.947163] Retrieving command... [546.947209] LAD_CONNECT: [546.947217] client FIFO name = /tmp/LAD/1216 [546.947223] client PID = 1216 [546.947233] assigned client handle = 0 [546.947285] FIFO /tmp/LAD/1216 created [546.948337] FIFO /tmp/LAD/1216 opened for writing [546.948371] sent response [546.948378] DONE [546.948383] Retrieving command... [546.948436] Sending response... [546.948469] Retrieving command... [546.948481] LAD_MULTIPROC_GETCONFIG: calling MultiProc_getConfig()... [546.948491] MultiProc_getConfig() - 5 procs [546.948497] # processors in cluster: 5 [546.948502] cluster baseId: 0 [546.948513] ProcId 0 - "HOST" [546.948519] ProcId 1 - "IPU2" [546.948525] ProcId 2 - "IPU1" [546.948530] ProcId 3 - "DSP2" [546.948535] ProcId 4 - "DSP1" [546.948540] status = 0 [546.948545] DONE [546.948549] Sending response... [546.948669] Retrieving command...

28

PresenterPresentation NotesThe slide shows an lad log exampleDuring LAD initialization, it will create a FIFO and then wait for request (commands) from client, that is the IPC application, when receiving any request from the application, it sends the response from a dedicated FIFO.The example here shows only a portion of the LAD log at the very beginning of an application starting to call ipc_start() and configure multiProc module. The complete LAD log will show the entire IPC call flow between host and remote processors.

RPMsg and Resource Table

29

PresenterPresentation NotesNow let’s briefly look at how the host Linux can control and communicate with remote cores. The Linux kernel RPMsg framework and resource table make it possible.

RPMsg • remoteproc is a generic kernel component that manages remote processors and enables users

access to these remote processors. The main functionalities implemented by remoteproc:

– Device Loading & Bootup

– Power Management

– Exception Management

– Error Recovery.

• rpmsg is a virtio-based messaging bus that allows kernel drivers to communicate with remote processors available on the system. root@k2g-evm:~# lsmod

Module Size Used by

rpmsg_proto 6634 0 virtio_rpmsg_bus 11070 1 rpmsg_proto omap_remoteproc 9771 1

30

PresenterPresentation NotesremoteProc is a generic kernel component that manages remote processors and enables users access to these remote processors. The remoteproc is responsible for Device Loading & Bootup, Power Management, Exception Management, Error Recovery. rpmsg is a virtio-based messaging bus that allows kernel drivers to communicate with remote processors available on the system

After the kernel is booted up, if you list the modules, you should be able to see several modules being brought up, that include rpmsg_proto, virtio_rpmsg_bus, omap_remoteproc…these modules constructs the RPMsg.Linux documentation at Linux kernel code tree /Document/remoteproc.txt and /Document/rpmsg.txt provides the detailed APIs about the remoteProc and rpmsg drivers.

Resource Table

• Resource table is a Linux construct used to inform the remoteproc driver about the remote processor’s available resources, and typically refers to memory and local peripheral registers.

• When a remote processor image is loaded, the Linux kernel remoteproc framework will parse the system resources defined in resource table, which is linked into the remote processor image. Also, the remoteproc allocates rpmsg vring buffers, trace buffer, and configure MMUs according to the resource table.

• See default ./packages/ti/ipc/remoteproc/rsc_table_xyz_dsp.h or ./packages/ti/ipc/remoteproc/rsc_table_xyz_ipu.h

• To customize resource table entries, refer to: http://processors.wiki.ti.com/index.php/IPC_Resource_customTable

31

PresenterPresentation NotesResource table is an important topic in IPC application, users are expected to understand the definitions in the resource table and update the entries in the table according to your system configurations. The default resource table can be found at packages/ti/ipc/remoteproc。Resource table is a Linux construct used to inform the remoteproc driver about the remote processor’s available resources, and typically refers to memory and local peripheral registers. When a remote processor image is loaded, the Linux kernel remoteproc framework will parse the system resources defined in resource table, which is linked into the remote processor image.The wiki page on the slide explains the details about each entry and how to customize the resources.

http://processors.wiki.ti.com/index.php/IPC_Resource_customTable

IPC Examples

32

PresenterPresentation NotesAt this point, now let’s check it out how to write your own IPC application. IPC release provides a few examples can be used as a starting point.

IPC Examples • A few examples for HLOS and SYS/BIOS running on various devices are available in the IPC

/examples directory. For instance: – examples/DRA7XX_linux_elf$ ls

• ex02_messageq ex12_mmrpc ex41_forwardmsg ex68_power

- examples/DRA7XX_bios_elf$ ls • ex01_hello ex02_messageq ex11_ping ex13_notifypeer

• In addition, the IPC release contains source files for functional testing. – /packages/ti/ipc/tests/

• gatemp.c, messageq_multicore.c, messageq_single.c, ping_rpmsg.c …

33

PresenterPresentation NotesA few examples for Linux and sys/bios running on different platforms are available in IPC /examples directory, for instance, examples/DRA7XX_linux_elf contains ex02_messageQ examples which shows the data exchange between A15 and DSP/M4 processors. In the example, A15 runs Linux and DSP/M4 runs SYS/BIOS>Also ex02_messageq in example/DRA7xx_bios_elf shows the data exchange between A15 and DSP/M4 with all processors runing SYS/BIOS.In addition, the IPC release contains source files for functional testing, which can be also used as good reference.

MessageQ Examples

examples/DRA7XX_linux_elf/ex02_messageq/dsp1/Dsp1.cfg

BIOS.addUserStartupFunction('&IpcMgr_ipcStartup'); …… var MessageQ = xdc.useModule('ti.sdo.ipc.MessageQ'); MessageQ.registerHeapMeta(msgHeap, 0); /* Setup MessageQ transport */ var VirtioSetup = xdc.useModule('ti.ipc.transports.TransportRpmsgSetup'); MessageQ.SetupTransportProxy = VirtioSetup; /* Setup NameServer remote proxy */ var NameServer = xdc.useModule("ti.sdo.utils.NameServer"); var NsRemote = xdc.useModule("ti.ipc.namesrv.NameServerRemoteRpmsg"); NameServer.SetupProxy = NsRemote; /* Enable Memory Translation module that operates on the BIOS Resource Table */ var Resource = xdc.useModule('ti.ipc.remoteproc.Resource'); Resource.customTable = true;

34

PresenterPresentation NotesThis slide shows an example of .cfg file of IPC SYS/BIOS MessageQ application.In addition to IPC APIs that an IPC application calls, the IPC config need to be properly setup in SYS/BIOS .cfg file. The .cfg specifies the modules (MessageQ, NameServer. , as well as MessageQ transport. that need to be built in application.

The config file also indicates the application will use customized resource table rather than default one from IPC release.

GateMP Examples ./packages/ti/ipc/tests/gatempapp.cfg

/* GateMP host support */ var GateMP = xdc.useModule('ti.sdo.ipc.GateMP'); GateMP.hostSupport = true; /* shared region configuration */ var SharedRegion = xdc.useModule('ti.sdo.ipc.SharedRegion'); /* configure SharedRegion #0 (IPC) */ var SR0Mem = Program.cpu.memoryMap["SR_0"]; SharedRegion.setEntryMeta(0, new SharedRegion.Entry({ name: "SR0", base: SR0Mem.base, len: SR0Mem.len, ownerProcId: MultiProc.getIdMeta("DSP1"), cacheEnable: true, isValid: true }) ); /*ownerProcId. MultiProc ID of the processor that manages this region. If an owner is specified, the

owner creates a HeapMemMP instance at runtime. The other cores open the same HeapMemMP instance.

isValid. Boolean to specify whether the region is valid (accessible) or not on this processor. */

35

PresenterPresentation NotesThe slide shows another .cfg file for GateMP application that require shared region setup.The SharedRegion module is designed to be used in a multi-processor environment where there are memory regions that are shared and accessed across different processors.In the application's C code, a shared memory region can be modified in the SharedRegion table by calling SharedRegion_setEntry(). Typically, applications configure SharedRegion table entries statically as described here, and only modify the table entries dynamically in applications where it is possible for shared memory region availability to change dynamically.

Examples: Image Load and Debug • The remote processor images are expected in the directory /lib/firmware/ and loaded

during Linux kernel bootup via remoteproc module, e.g. on AM57x

• To check the remote processor image boot log:

36

Core Binary on the Host Binary on the Target File System

IPU1 server_ipu1.xem4 /lib/firmware/dra7-ipu1-fw.xem4

IPU2 server_ipu2.xem4 /lib/firmware/dra7-ipu2-fw.xem4

DSP1 server_dsp1.xe66 /lib/firmware/dra7-dsp1-fw.xe66

DSP2 server_dsp2.xe66 /lib/firmware/dra7-dsp2-fw.xe66

Core Debug Trace

IPU1 /sys/kernel/debug/remoteproc/remoteproc0/trace0

IPU2 /sys/kernel/debug/remoteproc/remoteproc1/trace0

DSP1 /sys/kernel/debug/remoteproc/remoteproc2/trace0

DSP2 /sys/kernel/debug/remoteproc/remoteproc3/trace0

PresenterPresentation NotesWith the help of remoteproc module, remote processor image can be loaded then run without additional utilities, e.g. JTAG emulator or bootloader.When the remote processor images have been built, user need to put them into the /lib/firmware directory as shown in the table on the slide.

Also from the remoteproc traces, you can know the remote processor status…as well as entire application status. This will greatly help debug IPC application remote processor problem if any.

IPC Benchmarking

37

PresenterPresentation NotesNow, after we are able to get the IPC application running, let’s look at the benchmarks of IPC messageQ module so you can determine if it meets your application requirement.

MessageQBench

• MessageQBench utilizes the IPC stack to communicate from the main processor to the remote processor via the MessageQ interface and measures the time required to complete. target# /usr/bin/MessageQBench

- The application will exchange 1000 messages with a payload of 8 by default with an average round trip time per message. The following is the usage parameters for the application.

- Usage: ./bin/MessageQBench [] [] [] Defaults: numLoops: 1000; payloadSize: 8, ProcId: 1

- DSP image: messageq_single.xe66

• Additional processor synchronization mechanism is required to measure the unidirectional latency.

38

PresenterPresentation NotesIPC provides a benchmark application that can be directly ported to your system to measure the round trip MessageQ latency. The benchmark application MessageQBench uses the IPC stack to communicate from the main processor to the remote processor via the MessageQ interface and measures the time required to complete.Note this application benchmarks the round trip delay. Additional processor synchronization mechanism is required for measuring the unidirectional latency

For More Information: • IPC Downloads: http://downloads.ti.com/dsps/dsps_public_sw/sdo_sb/targetcontent/ipc/

• IPC User’s Guide: http://processors.wiki.ti.com/index.php/IPC_Users_Guide

• IPC Wiki: http://processors.wiki.ti.com/index.php/Category:IPC

• IPC Transport Offerings: http://processors.wiki.ti.com/index.php/MCSDK_UG_Chapter_Developing_Transports#IPC_Transports

• For questions regarding topics covered in this training, visit the support forums at the TI E2E Community website.

39

PresenterPresentation NotesThe final slide provides some IPC reference links.For questions regarding topics covered in this training, please visit the support forums at the TI E2E Community website e2e.ti.com

http://downloads.ti.com/dsps/dsps_public_sw/sdo_sb/targetcontent/ipc/http://processors.wiki.ti.com/index.php/IPC_Users_Guidehttp://processors.wiki.ti.com/index.php/Category:IPChttp://processors.wiki.ti.com/index.php/MCSDK_UG_Chapter_Developing_Transportshttp://e2e.ti.com/http://e2e.ti.com/http://ap-fpdsp-swapps.dal.design.ti.com/index.php/Processor_SDK_RTOS_Software_Developer_Guide

Introduction to�Inter-Processor Communication (IPC)�for KeyStone and Sitara™ DevicesAgendaIPC 3.x Overview: ArchitectureMulticore ProcessingWhy IPC?IPC 3.x OverviewCurrent IPC ReleasesIPC Open Source ProjectIPC MessageQ Software Architecture: AK2H IPC MessageQ Software Architecture: AM57xIPC 3.x Overview: ModulesIPC SYS/BIOS ModulesIPC HLOS ModulesIPC Modules DefinedIpc ModuleMessageQ ModuleMessageQ HeaderUsing MessageQGateMPGateMP Ladder DiagramMultiProc ModuleMultiProc MappingLibrary and daemonIPC Library Build: products.makIPC Library Build: products.mak exampleIPC Library Build CommandsIPC DaemonIPC Daemon InitializationRPMsg and Resource TableRPMsgResource TableIPC ExamplesIPC ExamplesMessageQ ExamplesGateMP ExamplesExamples: Image Load and DebugIPC BenchmarkingMessageQBenchFor More Information: