FMC Bus Abstraction for Linux - Open Hardware Repository · FMC Bus Abstraction for Linux ... an advanced user can also use sysfs to change the binding of ... The contents of the

FMC Bus Abstraction for LinuxMay 2013

A kernel bus to support FMC mezzanines and carriers

Alessandro Rubini for CERN (BE-CO-HT)

i

Table of Contents

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1 Basic Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

2 What is FMC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

3 What is a Linux Bus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

4 What is SDB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

5 Functions Exported by fmc.ko . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

6 Module Parameters in fmc.ko . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

7 FMC Device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37.1 The API Offered by Carriers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57.2 The GPIO Abstraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67.3 fmc-fakedev . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87.4 FMC Device Incompatibilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

8 FMC Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98.1 ID Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98.2 Module Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108.3 fmc-trivial . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108.4 fmc-write-eeprom . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118.5 fmc-chardev . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

9 Writing your FMC Kernel Module . . . . . . . . . . . . . . . . . . . . . . . . . 129.1 Versions of the Bus Abstraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139.2 Git Submodule Crash Course . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

Quick Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14Supermodules and Submodules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14How to Commit to a Submodule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

9.3 Git Submodules and “make install” . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159.4 Suggestions for Mezzanine Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169.5 Suggestions for Carrier Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179.6 Work in Progress . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

10 FMC Identification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1810.1 Building the FRU Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1810.2 Using SDB-FS in the EEPROM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1910.3 Writing to the EEPROM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2010.4 Reading back the EEPROM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

ii

11 SDB Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

12 Portability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

Chapter 3: What is a Linux Bus 1

Introduction

This document describes the implementation of the fmc bus for Linux. fmc (FPGA MezzanineCard) is the standard we use for our I/O devices, in the context of White Rabbit and relatedhardware.

In our I/O environments we need to write drivers for each mezzanine card, and such driversmust work regardless of the carrier being used. To achieve this, we abstract the fmc interface.

We have a carrier for PCI-E called spec and one for VME called svec, but more are planned.Also, we support stand-alone devices (usually plugged on a spec card), controlled throughEtherbone, developed by GSI.

Code and documentation for the fmc bus was born as part of the spec-sw project, but now itlives in its own project. Other projects, i.e. software support for the various carriers, shouldinclude this as a submodule.

The most up to date version of code and documentation is always available from the repositoryyou can clone from:

git://ohwr.org/fmc-projects/fmc-bus.git (read-only)

[email protected]:fmc-projects/fmc-bus.git (read-write for developers)

Selected versions of the documentation, as well as complete tar archives for selected revisions areplaced to the Files section of the project: http://www.ohwr.org/projects/fmc-bus/files

1 Basic Concepts

2 What is FMC

fmc, as said, stands for “FPGA Mezzanine Card”. It is a standard developed by the VMEconsortium called VITA (VMEbus International Trade Association and ratified by ANSI, theAmerican National Standard Institute. The official documentation is called “ANSI-VITA 57.1”.

The fmc card is an almost square pcb, around 70x75 millimeters, that is called mezzanine inthis document. It usually lives plugged into another pcb for power supply and control; suchbigger circuit board is called carrier from now on, and a single carrier may host more than onemezzanine.

In the typical application the mezzanine is mostly analog while the carrier is mostly digital,and hosts an fpga that must be configured to match the specific mezzanine and the desiredapplication. Thus, you may need to load different fpga images to drive different instances ofthe same mezzanine.

fmc, as such, is not a bus in the usual meaning of the term, because most carriers have only oneconnector, and carriers with several connectors have completely separate electrical connectionsto them. This package, however, implements a bus as a software abstraction.

3 What is a Linux Bus

Within the Linux kernel, a bus is a data structure with a few methods. It’s main role isregistering a list of devices and a list of drivers, offering a match function that compares therespective identifiers (in a bus-specific way) to assign drivers to devices.

Activation and deactivation of devices happens through the probe and remove functions of therespective driver; an advanced user can also use sysfs to change the binding of drivers to devices(for example, if more than one driver can drive the same device you may want to force thechoice).

http://www.ohwr.org/projects/fmc-bus/files

Chapter 5: Functions Exported by fmc.ko 2

4 What is SDB

sdb (Self Describing Bus) is a set of data structures that we use for enumerating the internalstructure of an fpga image. We also use it as a filesystem inside the fmc eeprom.

sdb is not mandatory for use of this fmc kernel bus, but if you have sdb this package can makegood use of it. sdb itself is developed in the fpga-config-space OHWR project. The link tothe repository is git://ohwr.org/hdl-core-lib/fpga-config-space.git and what is usedin this project lives in the sdbfs subdirectory in there.

sdb support for fmc is described in Chapter 10 [FMC Identification], page 18 and Chapter 11[SDB Support], page 21

5 Functions Exported by fmc.ko

The fmc core exports the usual 4 functions that are needed for a bus to work, and a few more:

int fmc_driver_register(struct fmc_driver *drv);

void fmc_driver_unregister(struct fmc_driver *drv);

int fmc_device_register(struct fmc_device *fmc);

void fmc_device_unregister(struct fmc_device *fmc);

int fmc_device_register_n(struct fmc_device **fmc, int n);

void fmc_device_unregister_n(struct fmc_device **fmc, int n);

uint32_t fmc_readl(struct fmc_device *fmc, int offset);

void fmc_writel(struct fmc_device *fmc, uint32_t val, int off);

void *fmc_get_drvdata(struct fmc_device *fmc);

void fmc_set_drvdata(struct fmc_device *fmc, void *data);

int fmc_reprogram(struct fmc_device *f, struct fmc_driver *d, char *gw,

int sdb_entry);

The data structure that describe a device is detailed in Chapter 7 [FMC Device], page 3, theone that describes a driver is detailed in Chapter 8 [FMC Driver], page 9. Please note thatstructures of type fmc_device must be allocated by the caller, but must not be released afterunregistering. The fmc-bus itself takes care of releasing the structure when their use countreaches zero – actually, the device model does that in lieu of us.

The functions to register and unregister n devices are meant to be used by carriers that hostmore than one mezzanine. The devices must all be registered at the same time because if thefpga is reprogrammed, all devices in the array are affected. Usually, the driver matching thefirst device will reprogram the fpga, so other devices must know they are already driven by areprogrammed fpga.

If a carrier hosts slots that are driven by different fpga devices, it should register as a grouponly mezzanines that are driven by the same fpga, for the reason outlined above.

Finally, the fmc reprogram function calls the reprogram method (see Section 7.1 [The APIOffered by Carriers], page 5 and also scans the memory area for an sdb tree. You can pass -1 assdb_entry to disable such scan. Otherwise, the function fails if no tree is found at the specifiedentry point. The function is meant to factorize common code, and by the time you read this itis already used by the spec-sw and fine-delay modules.

git://ohwr.org/hdl-core-lib/fpga-config-space.git

Chapter 7: FMC Device 3

6 Module Parameters in fmc.ko

The core driver receives two module parameters, meant to help debugging client modules. Bothparameters can be modified by writing to /sys/module/fmc/parameters/, because they are usedwhen client drivers are devices are registered, not when fmc.ko is loaded.

dump_eeprom=

If not zero, the parameter asks the bus controller to dump the eeprom of any devicethat is registered, using printk.

dump_sdb=

If not zero, the parameter prints the sdb tree of every fpga it is loaded byfmc_reprogram(). If greater than one, it asks to dump the binary content of sdbrecords. This currently only dumps the top-level sdb array, though.

eeprom dumping avoids repeating lines, since most of the contents is usually empty and all bitsare one or zero. This is an example of the output:

[ 6625.850480] spec 0000:02:00.0: FPGA programming successful

[ 6626.139949] spec 0000:02:00.0: Manufacturer: CERN

[ 6626.144666] spec 0000:02:00.0: Product name: FmcDelay1ns4cha

[ 6626.150370] FMC: mezzanine 0: 0000:02:00.0 on SPEC

[ 6626.155179] FMC: dumping eeprom 0x2000 (8192) bytes

[ 6626.160087] 0000: 01 00 00 01 00 0b 00 f3 01 0a 00 a5 85 87 c4 43

[ 6626.167069] 0010: 45 52 4e cf 46 6d 63 44 65 6c 61 79 31 6e 73 34

[ 6626.174019] 0020: 63 68 61 c7 70 72 6f 74 6f 2d 30 cc 45 44 41 2d

[ 6626.180975] 0030: 30 32 32 36 37 2d 56 33 da 32 30 31 32 2d 31 31

[...]

[ 6626.371366] 0200: 66 64 65 6c 61 79 0a 00 00 00 00 00 00 00 00 00

[ 6626.378359] 0210: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

[ 6626.385361] [...]

[ 6626.387308] 1800: 70 6c 61 63 65 68 6f 6c 64 65 72 ff ff ff ff ff

[ 6626.394259] 1810: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

[ 6626.401250] [...]

The dump of sdb looks like the following; the example shows the simple golden gateware for thespec card, removing the leading timestamps to fit the page:

spec 0000:02:00.0: SDB: 00000651:e6a542c9 WB4-Crossbar-GSI

spec 0000:02:00.0: SDB: 0000ce42:ff07fc47 WR-Periph-Syscon (00000000-000000ff)

FMC: mezzanine 0: 0000:02:00.0 on SPEC

FMC: poor dump of sdb first level:

0000: 53 44 42 2d 00 02 01 00 00 00 00 00 00 00 00 00

0010: 00 00 00 00 00 00 01 ff 00 00 00 00 00 00 06 51

0020: e6 a5 42 c9 00 00 00 02 20 12 05 11 57 42 34 2d

0030: 43 72 6f 73 73 62 61 72 2d 47 53 49 20 20 20 00

0040: 00 00 01 01 00 00 00 07 00 00 00 00 00 00 00 00

0050: 00 00 00 00 00 00 00 ff 00 00 00 00 00 00 ce 42

0060: ff 07 fc 47 00 00 00 01 20 12 03 05 57 52 2d 50

0070: 65 72 69 70 68 2d 53 79 73 63 6f 6e 20 20 20 01

7 FMC Device

Within the Linux bus framework, the fmc device is created and registered by the carrier driver.For example, the PCI driver for the spec card fills a data structure for each spec that it drives,and registers an associated fmc device for each card. The svec driver can do exactly thesame for the VME carrier (actually, it should do it twice, because the svec carries two fmcmezzanines). Similarly, an Etherbone driver will be able to register its own fmc devices, offeringcommunication primitives through frame exchange.

The contents of the eeprom within the fmc are used for identification purposes, i.e. for matchingthe device with its own driver. For this reason the device structure includes a complete copy of


the eeprom (actually, the carrier driver may choose whether or not to return it – for examplewe most likely won’t have the whole eeprom available for Etherbone devices.

The following listing shows the current structure defining a device. Please note that all themachinery is in place but some details may still change in the future. For this reason, there isa version field at the beginning of the structure. As usual, the minor number will change forcompatible changes (like a new flag) and the major number will increase when an incompatiblechange happens (for example, a change in layout of some fmc data structures). Device writersshould just set it to the value FMC_VERSION, and be ready to get back -EINVAL at registrationtime.

struct fmc_device {

unsigned long version;

unsigned long flags;

struct module *owner; /* char device must pin it */

struct fmc_fru_id id; /* for EEPROM-based match */

struct fmc_operations *op; /* carrier-provided */

int irq; /* according to host bus. 0 == none */

int eeprom_len; /* Usually 8kB, may be less */

int eeprom_addr; /* 0x50, 0x52 etc */

uint8_t *eeprom; /* Full contents or leading part */

char *carrier_name; /* "SPEC" or similar, for special use */

void *carrier_data; /* "struct spec *" or equivalent */

__iomem void *fpga_base; /* May be NULL (Etherbone) */

__iomem void *slot_base; /* Set by the driver */

struct fmc_device **devarray; /* Allocated by the bus */

int slot_id; /* Index in the slot array */

int nr_slots; /* Number of slots in this carrier */

unsigned long memlen; /* Used for the char device */

struct device dev; /* For Linux use */

struct device *hwdev; /* The underlying hardware device */

unsigned long sdbfs_entry;

struct sdb_array *sdb;

uint32_t device_id; /* Filled by the device */

char *mezzanine_name; /* Defaults to ‘‘fmc’’ */

void *mezzanine_data;

};

The meaning of most fields is summarized in the code comment above.

The following fields must be filled by the carrier driver before registration:

• version: must be set to FMC_VERSION.

• owner: set to MODULE_OWNER.

• op: the operations to act on the device.

• irq: number for the mezzanine; may be zero.

• eeprom_len: length of the following array.

• eeprom_addr: 0x50 for first mezzanine and so on.

• eeprom: the full content of the i2c eeprom.

• carrier_name.

• carrier_data: a unique pointer for the carrier.

• fpga_base: the I/O memory address (may be NULL).

• slot_id: the index of this slot (starting from zero).

• memlen: if fpga_base is valid, the length of I/O memory.

• hwdev: to be used in dev err() calls.

• device_id: a slot-specific unique integer number.

Please note that the carrier should read its own eeprom memory before registering the device,as well as fill all other fields listed above.


The following fields should not be assigned, because they are filled later by either the bus or thedevice driver:

• flags.

• fru_id: filled by the bus, parsing the eeprom.

• slot_base: filled and used by the driver, if useful to it.

• devarray: an array og all mezzanines driven by a singe fpga.

• nr_slots: set by the core at registration time.

• dev: used by Linux.

• sdb: fpga contents, scanned according to driver’s directions.

• sdbfs_entry: sdb entry point in eeprom: autodetected.

• mezzanine_data: available for the driver.

• mezzanine_name: filled by fmc-bus during identification.

Note: mezzanine data may be redundant, because Linux offers the drvdata approach, so thefield may be removed in later versions of this bus implementation.

As I write this, she spec carrier is already completely functional in the fmc-bus environment,and is a good reference to look at.

7.1 The API Offered by Carriers

The carrier provides a number of methods by means of the fmc_operations structure, whichcurrently is defined like this (again, it is a moving target, please refer to the header rather thanthis document):

struct fmc_operations {

uint32_t (*readl)(struct fmc_device *fmc, int offset);

void (*writel)(struct fmc_device *fmc, uint32_t value, int offset);

int (*reprogram)(struct fmc_device *f, struct fmc_driver *d, char *gw);

int (*validate)(struct fmc_device *fmc, struct fmc_driver *drv);

int (*irq_request)(struct fmc_device *fmc, irq_handler_t h,

char *name, int flags);

void (*irq_ack)(struct fmc_device *fmc);

int (*irq_free)(struct fmc_device *fmc);

int (*gpio_config)(struct fmc_device *fmc, struct fmc_gpio *gpio,

int ngpio);

int (*read_ee)(struct fmc_device *fmc, int pos, void *d, int l);

int (*write_ee)(struct fmc_device *fmc, int pos, const void *d, int l);

};

The individual methods perform the following tasks:

readl

writel

These functions access fpga registers by whatever means the carrier offers. Theyare not expected to fail, and most of the time they will just make a memory accessto the host bus. If the carrier provides a fpga base pointer, the driver may use directaccess through that pointer. For this reason the header offers the inline functionsfmc readl and fmc writel that access fpga base if the respective method is NULL. Adriver that wants to be portable and efficient should use fmc readl and fmc writel.For Etherbone, or other non-local carriers, error-management is still to be defined.

validate

Module parameters are used to manage different applications for two or more boardsof the same kind. Validation is based on the busid module parameter, if provided,and returns the matching index in the associated array. See Section 8.2 [Module


Parameters], page 10 in in doubt. If no match is found, -ENOENT is returned; ifthe user didn’t pass busid=, all devices will pass validation. The value returnedby the validate method can be used as index into other parameters (for example,some drivers use the lm32= parameter in this way). Such “generic parameters”are documented in Section 8.2 [Module Parameters], page 10, below. The validatemethod is used by fmc-trivial.ko, described in Section 8.3 [fmc-trivial], page 10.

reprogram

The carrier enumerates fmc devices by loading a standard (or golden) fpga binarythat allows eeprom access. Each driver, then, will need to reprogram the fpgaby calling this function. If the name argument is NULL, the carrier should repro-gram the golden binary. If the gateware name has been overridden through moduleparameters (in a carrier-specific way) the file loaded will match the parameters.Per-device gateware names can be specified using the gateware= parameter, seeSection 8.2 [Module Parameters], page 10. Note: Clients should call rhe new helper,fmc reprogram, which both calls this method and parse the sdb tree of the fpga.

irq_request

irq_ack

irq_free

Interrupt management is carrier-specific, so it is abstracted as operations. Theinterrupt number is listed in the device structure, and for the mezzanine driver thenumber is only informative. The handler will receive the fmc pointer as dev id ; theflags argument is passed to the Linux request irq function, but fmc-specific flagsmay be added in the future. You’ll most likely want to pass the IRQF_SHARED flag.

gpio_config

The method allows to configure a GPIO pin in the carrier, and read its currentvalue if it is configured as input. See Section 7.2 [The GPIO Abstraction], page 6for details.

read_ee

write_ee

Read or write the eeprom. The functions are expected to be only called beforereprogramming and the carrier should refuse them with ENODEV after reprogram-ming. The offset is expected to be within 8kB (the current size), but addresses upto 1MB are reserved to fit bigger I2C devices in the future. Carriers may offer accessto other internal flash memories using these same methods: for example the specdriver may define that its carrier I2C memory is seen at offset 1M and the internalSPI flash is seen at offset 16M. This multiplexing of several flash memories in thesame address space is is carrier-specific and should only be used by a driver thathas verified the carrier_name field.

7.2 The GPIO Abstraction

Support for GPIO pins in the fmc-bus environment is not very straightforward and deservesspecial discussion.

While the general idea of a carrier-independent driver seems to fly, configuration of specificsignals within the carrier needs at least some knowledge of the carrier itself. For this reason, thespecific driver can request to configure carrier-specific GPIO pins, numbered from 0 to at most4095. Configuration is performed by passing a pointer to an array of struct fmc_gpio items,as well as the length of the array. This is the data structure:

struct fmc_gpio {

char *carrier_name;


int gpio;

int _gpio; /* internal use by the carrier */

int mode; /* GPIOF_DIR_OUT etc, from <linux/gpio.h> */

int irqmode; /* IRQF_TRIGGER_LOW and so on */

};

By specifying a carrier name for each pin, the driver may access different pins in different car-riers. The gpio config method is expected to return the number of pins successfully configured,ignoring requests for other carriers. However, if no pin is configured (because no structure at allrefers to the current carrier name), the operation returns an error so the caller will know thatit is running under a yet-unsupported carrier.

So, for example, a driver that has been developed and tested on both the spec and the svecmay request configuration of two different GPIO pins, and expect one such configuration tosucceed – if none succeeds it most likely means that the current carrier is a still-unknown one.

If, however, your GPIO pin has a specific known role, you can pass a special number in the gpiofield, using one of the following macros:

#define FMC_GPIO_RAW(x) (x) /* 4096 of them */

#define FMC_GPIO_IRQ(x) ((x) + 0x1000) /* 256 of them */

#define FMC_GPIO_LED(x) ((x) + 0x1100) /* 256 of them */

#define FMC_GPIO_KEY(x) ((x) + 0x1200) /* 256 of them */

#define FMC_GPIO_TP(x) ((x) + 0x1300) /* 256 of them */

#define FMC_GPIO_USER(x) ((x) + 0x1400) /* 256 of them */

Use of virtual GPIO numbers (anything but FMC_GPIO_RAW) is allowed provided the carrier namefield in the data structure is left unspecified (NULL). Each carrier is responsible for providinga mapping between virtual and physical GPIO numbers. The carrier may then use the _gpio

field to cache the result of this mapping.

All carriers must map their I/O lines to the sets above starting from zero. The spec, for example,maps interrupt pins 0 and 1, and test points 0 through 3 (even if the test points on the pcb arecalled 5,6,7,8).

If, for example, a driver requires a free LED and a test point (for a scope probe to be plugged atsome point during development) it may ask for FMC_GPIO_LED(0) and FMC_GPIO_TP(0). Eachcarrier will provide suitable GPIO pins. Clearly, the person running the drivers will know theorder used by the specific carrier driver in assigning leds and testpoints, so to make a carrier-dependent use of the diagnostic tools.

In theory, some form of autodetection should be possible: a driver like the wr-nic (which usesIRQ(1) on the spec card) should configure IRQ(0), make a test with software-generated inter-rupts and configure IRQ(1) if the test fails. This probing step should be used because even if thewr-nic gateware is known to use IRQ1 on the spec, the driver should be carrier-independent andthus use IRQ(0) as a first bet – actually, the knowledge that IRQ0 may fail is carrier-dependentinformation, but using it doesn’t make the driver unsuitable for other carriers.

The return value of gpio config is defined as follows:

• If no pin in the array can be used by the carrier, -ENODEV.

• If at least one virtual GPIO number cannot be mapped, -ENOENT.

• On success, 0 or positive. The value returned is the number of high input bits (if no inputis configured, the value for success is 0).

While I admit the procedure is not completely straightforward, it allows configuration, inputand output with a single carrier operation. Given the typical use case of fmc devices, GPIOoperations are not expected to ever by in hot paths, and GPIO access so fare has only been usedto configure the interrupt pin, mode and polarity. Especially reading inputs is not expected to


be common. If your device has GPIO capabilities in the hot path, you should consider usingthe kernel’s GPIO mechanisms.

7.3 fmc-fakedev

This package includes a software-only device, called fmc-fakedev, which is able to register up to4 mezzanines (by default it registers one). Unlike the spec driver, which creates an fmc devicefor each PCI cards it manages, this module creates a single instance of its set of mezzanines.

It is meant as the simplest possible example of how a driver should be written, and it includes afake eeprom image (built using the tools described in Chapter 10 [FMC Identification], page 18),which by default is replicated for each fake mezzanine.

You can also use this device to verify the match algorithms, by asking it to test your owneeprom image. You can provide the image by means of the eeprom= module parameter: thenew eeprom image is loaded, as usual, by means of the firmware loader. This example showsthe defaults and a custom eeprom image:

spusa.root# insmod fmc-fakedev.ko

[ 99.971247] fake-fmc-carrier: mezzanine 0

[ 99.975393] Manufacturer: fake-vendor

[ 99.979624] Product name: fake-design-for-testing

spusa.root# rmmod fmc-fakedev

spusa.root# insmod fmc-fakedev.ko eeprom=fdelay-eeprom.bin

[ 121.447464] fake-fmc-carrier: Mezzanine 0: eeprom "fdelay-eeprom.bin"

[ 121.462725] fake-fmc-carrier: mezzanine 0

[ 121.466858] Manufacturer: CERN

[ 121.470477] Product name: FmcDelay1ns4cha

spusa.root# rmmod fmc-fakedev

After loading the device, you can use the write ee method do modify its own internal fakeeeprom: whenever the image is overwritten starting at offset 0, the module will unregister andregister again the fmc device. This is shown later, in section Section 8.4 [fmc-write-eeprom],page 11.

In order to register more than one mezzanine, you can use module parameters:

ndev=<number>

Ask for a number of devices. By default it registers one device or as many as eepromfile names it receives.

eeprom=<name>[,<name>]

Name of the eeprom images to be faked in the mezzanine devices. You can pass upto 4 comma-separated names; empty names or trailing unspecified names will selectthe default built-in eeprom image. If you specify more than one eeprom name,the module will automatically register more than one mezzanine.

7.4 FMC Device Incompatibilities

During development, we had to introduce a few incompatibilities in the data structures. By usingthe field named version inside fmc_device you should be notified about such changes. Pleaseuse git blame for the details – the version is always updated in the same commit that makesthe change (for example commit 14a7b580 introduced version 3, changed the data structuresand documentation at the same time). See Chapter 7 [FMC Device], page 3 for the current listof fields.

The other incompatible change we had to introduce is in the allocation policy for fmc_devicestructures. The devices must be allocated one at a time (with kmalloc) and never released.Releasing happens by means of the release method of the device, when its use count drops tozero.

Chapter 8: FMC Driver 9

As a side effect, fmc_device_register_n must be passed an array of pointers, not an array ofdevices. Since each device is freed separately when not in use any more.

8 FMC Driver

An fmc driver is concerned with the specific mezzanine and associated gateware. As such, itis expected to be independent of the carrier being used: it will perform I/O accesses only bymeans of carrier-provided functions.

The matching between device and driver is based on the content of the eeprom (as mandatedby the fmc standard) or by the actual cores configured in the fpga; the latter technique is usedwhen the fpga is already programmed when the device is registered to the bus core.

In some special cases it is possible for a driver to directly access fpga registers, by means of thefpga_base field of the device structure. This may be needed for high-bandwidth peripheralslike fast ADC cards. If the device module registered a remote device (for example by meansof Etherbone), the fpga_base pointer will be NULL. Therefore, drivers must be ready to dealwith NULL base pointers, and fail gracefully. Most driver, however, are not expected to accessthe pointer directly but run fmc readl and fmc writel instead, which will work in any case.

In even more special cases, the driver may access carrier-specific functionality: the carrier_namestring allows the driver to check which is the current carrier and make use of the carrier_datapointer. We chose to use carrier names rather than numeric identifiers for greater flexibility,but also to avoid a central registry within the fmc.h file – we hope other users will exploit ourframework with their own carriers. An example use of carrier names is in GPIO setup (seeSection 7.2 [The GPIO Abstraction], page 6), although the name match is not expected to beperformed by the driver. If you depend on specific carriers, please check the carrier name andfail gracefully if your driver finds it is running in a yet-unknown-to-it environment.

8.1 ID Table

Like most other Linux drivers, and fmc driver must list all the devices which it is able to drive.This is usually done by means of a device table, but in fmc we can match hardware basedeither on the contents of their eeprom or on the actual fpga cores that can be enumerated.Therefore, we have two tables of identifiers.

Matching of fru information depends on two names, the manufacturer (or vendor) and thedevice (see Chapter 10 [FMC Identification], page 18); for flexibility during production (i.e.before writing to the eeprom) the bus supports a catch-all driver that specifies NULL strings.For this reason, the table is specified as pointer-and-length, not a a null-terminated array – theentry with NULL names can be a valid entry.

Matching on fpga cores depends on two numeric fields: the 64-bit vendor number and the32-bit device number. Support for matching based on class is not yet implemented. Eachdevice is expected to be uniquely identified by an array of cores (it matches if all of the coresare instantiated), and for consistency the list is passed as pointer-and-length. Several similardevices can be driven by the same driver, and thus the driver specifies and array of such arrays.

The complete set of involved data structures is thus the following:struct fmc_fru_id { char *manufacturer; char *product_name; };

struct fmc_sdb_one_id { uint64_t vendor; uint32_t device; };

struct fmc_sdb_id { struct fmc_sdb_one_id *cores; int cores_nr; };

struct fmc_device_id {

struct fmc_fru_id *fru_id; int fru_id_nr;

struct fmc_sdb_id *sdb_id; int sdb_id_nr;

};

A better reference, with full explanation, is the <linux/fmc.h> header.


8.2 Module Parameters

Most of the fmc drivers need the same set of kernel parameters. This package includes supportto implement common parameters by means of fields in the fmc_driver structure and simplemacro definitions.

The parameters are carrier-specific, in that they rely on the busid concept, that varies amongcarriers. For the spec, the identifier is a PCI bus and devfn number, 16 bits wide in total;drivers for other carriers will most likely offer something similar but not identical, and somecode duplication is unavoidable.

This is the list of parameters that are common to several modules to see how they are actuallyused, please look at spec-trivial.c.

busid=

This is an array of integers, listing carrier-specific identification numbers. For PIC,for example, 0x0400 represents bus 4, slot 0. If any such ID is specified, the driverwill only accept to drive cards that appear in the list (even if the fmc ID matches).This is accomplished by the validate carrier method.

gateware=

The argument is an array of strings. If no busid= is specified, the first string ofgateware= is used for all cards; otherwise the identifiers and gateware names arepaired one by one, in the order specified.

show_sdb=

For modules supporting it, this parameter asks to show the sdb internal structureby means of kernel messages. It is disabled by default because those lines tend tohide more important messages, if you look at the system console while loading thedrivers. Note: the parameter is being obsoleted, because fmc.ko itself now supportsdump_sdb= that applies to every client driver.

For example, if you are using the trivial driver to load two different gateware files to two differentcards, you can use the following parameters to load different binaries to the cards, after lookingup the PCI identifiers. This has been tested with a spec carrier.

insmod fmc-trivial.ko \

busid=0x0200,0x0400 \

gateware=fmc/fine-delay.bin,fmc/simple-dio.bin

Please note that not all sub-modules support all of those parameters. You can use modinfo tocheck what is supported by each module.

8.3 fmc-trivial

The simple module fmc-trivial is just a simple client that registers an interrupt handler. I usedit to verify the basic mechanism of the fmc bus and how interrupts worked.

The module implements the generic fmc parameters, so it can program a different gateware filein each card. The whole list of parameters it accepts are:

busid=

gateware=

Generic parameters. See Section 8.2 [Module Parameters], page 10.

This driver is worth reading, but it is not worth describing here.


8.4 fmc-write-eeprom

This module is designed to load a binary file from /lib/firmware and to write it to the internaleeprom of the mezzanine card. This driver uses the busid generic parameter.

Overwriting the eeprom is not something you should do daily, and it is expected to only happenduring manufacturing. For this reason, the module makes it unlikely for the random user tochange a working eeprom.

The module takes the following measures:

• It accepts a file= argument (within /lib/firmware) and if no such argument is received, itdoesn’t write anything to eeprom (i.e. there is no default file name).

• If the file name ends with .bin it is written verbatim starting at offset 0.

• If the file name ends with .tlv it is interpreted as type-length-value (i.e., it allows writev(2)-like operation).

• If the file name doesn’t match any of the patterns above, it is ignored and no write isperformed.

• Only cards listed with busid= are written to. If no busid is specified, no programming isdone (and the probe function of the driver will fail).

Each TLV tuple is formatted in this way: the header is 5 bytes, followed by data. The first byteis w for write, the next two bytes represent the address, in little-endian byte order, and the nexttwo represent the data length, in little-endian order. The length does not include the header (itis the actual number of bytes to be written).

This is a real example: that writes 5 bytes at position 0x110:

spusa.root# od -t x1 -Ax /lib/firmware/try.tlv

000000 77 10 01 05 00 30 31 32 33 34

00000a

spusa.root# insmod /tmp/fmc-write-eeprom.ko busid=0x0200 file=try.tlv

[19983.391498] spec 0000:03:00.0: write 5 bytes at 0x0110

[19983.414615] spec 0000:03:00.0: write_eeprom: success

Please note that you’ll most likely want to use sdbfs to build your eeprom image, at least ifyour mezzanines are being used in the White Rabbit environment. For this reason the TLVformat is not expected to be used much and is not expected to be developed further.

If you want to try reflashing fake eeprom devices, you can use the fmc-fakedev.ko module (seeSection 7.3 [fmc-fakedev], page 8). Whenever you change the image starting at offset 0, it willderegister and register again after two seconds. Please note, however, that if fmc-write-eepromis still loaded, the system will associate it to the new device, which will be reprogrammed andthus will be unloaded after two seconds. The following example removes the module after itreflashed fakedev the first time.

spusa.root# insmod fmc-fakedev.ko

[ 72.984733] fake-fmc: Manufacturer: fake-vendor

[ 72.989434] fake-fmc: Product name: fake-design-for-testing

spusa.root# insmod fmc-write-eeprom.ko busid=0 file=fdelay-eeprom.bin; \

rmmod fmc-write-eeprom

[ 130.874098] fake-fmc: Matching a generic driver (no ID)

[ 130.887845] fake-fmc: programming 6155 bytes

[ 130.894567] fake-fmc: write_eeprom: success

[ 132.895794] fake-fmc: Manufacturer: CERN

[ 132.899872] fake-fmc: Product name: FmcDelay1ns4cha

8.5 fmc-chardev

This is a simple generic driver, that allows user access by means of a character device (actually,one for each mezzanine it takes hold of).

Chapter 9: Writing your FMC Kernel Module 12

The char device is created as a misc device. Its name in /dev (as created by udev) is the samename as the underlying fmc device. Thus, the name can be a silly fmc-0000 look-alike if thedevice has no identifiers nor bus id, a more specific fmc-0400 if the device has a bus-specificaddress but no associated name, or something like fdelay-0400 if the fmc core can rely onboth a mezzanine name and a bus address.

Currently the driver only supports read and write: you can lseek to the desired address andread or write a register.

The driver assumes all registers are 32-bit in size, and only accepts a single read or write persystem call. However, as a result of Unix read and write semantics, users can simply fread orfwrite bigger areas in order to dump or store bigger memory areas.

There is currently no support for mmap, user-space interrupt management and DMA buffers.They may be added in later versions, if the need arises.

The example below shows raw access to a spec card programmed with its golden fpga file, thatfeatures an sdb structure at offset 256 – i.e. 64 words. The mezzanine’s eeprom in this caseis not programmed, so the default name is fmc-<bus><devfn>, and there are two cards in thesystem:

spusa.root# insmod fmc-chardev.ko

[ 1073.339332] spec 0000:02:00.0: Driver has no ID: matches all

[ 1073.345051] spec 0000:02:00.0: Created misc device "fmc-0200"

[ 1073.350821] spec 0000:04:00.0: Driver has no ID: matches all

[ 1073.356525] spec 0000:04:00.0: Created misc device "fmc-0400"

spusa.root# ls -l /dev/fmc*

crw------- 1 root root 10, 58 Nov 20 19:23 /dev/fmc-0200

crw------- 1 root root 10, 57 Nov 20 19:23 /dev/fmc-0400

spusa.root# dd bs=4 skip=64 count=1 if=/dev/fmc-0200 2> /dev/null | od -t x1z

0000000 2d 42 44 53 >-BDS<

0000004

The simple program tools/fmc-mem in this package can access an fmc char device and read orwrite a word or a whole area. Actually, the program is not specific to fmc at all, it just useslseek, read and write.

Its first argument is the device name, the second the offset, the third (if any) the value to writeand the optional last argument that must begin with “+” is the number of bytes to read or write.In case of repeated reading data is written to stdout ; repeated writes read from stdin and thevalue argument is ignored.

The following examples show reading the sdb magic number and the first sdb record from aspec device programmed with its golden image:

spusa.root# ./fmc-mem /dev/fmc-0200 100

5344422d

spusa.root# ./fmc-mem /dev/fmc-0200 100 +40 | od -Ax -t x1z

000000 2d 42 44 53 00 01 02 00 00 00 00 00 00 00 00 00 >-BDS............<

000010 00 00 00 00 ff 01 00 00 00 00 00 00 51 06 00 00 >............Q...<

000020 c9 42 a5 e6 02 00 00 00 11 05 12 20 2d 34 42 57 >.B......... -4BW<

000030 73 6f 72 43 72 61 62 73 49 53 47 2d 00 20 20 20 >sorCrabsISG-. <

000040

9 Writing your FMC Kernel Module

This chapter includes some suggestions about to write your own drivers for fmc carriers ormezzanines. It is the outcome of the experience I gathered working with fine-delay and wr-

nic.ko (currently part of spec-sw, as I write this).

Driver writing is a known problem, which only requires you to know your hardware and yourframeworks. However, the choice to split the overall fmc software design into several projects,


and thus several repositories, opens a new set of problems, related to how the projects relateone another.

A typical carrier driver will be built against a specific version of this fmc-bus package. Amezzanine driver, in addition, will most often be developed under a specific carrier and willmost likely depend on features of the the specific carrier, at least initially.

To make the problem worse, a single host in your operating environment (“in production”) mayneed tp run several fmc devices (thus several fmc drivers), that expect to be using differentversions of the base packages – according to when and where they have been written.

9.1 Versions of the Bus Abstraction

This fmc-bus package includes a version field at the beginning of the two data structures, fmc_device and fmc_driver.

This version number is there to prevent mismatches between the information passed by the fmcmodules and how it is used by the core code. The version is not expected to change often,because the code base is already pretty stable, and we don’t expect to face many new problems.

In general, if you experience a mismatch, you should just recompile your carrier or mezzaninedriver with the same version of fmc-bus that you are running as a kernel module. If one of thestructures has a new field in there, the default of all-bits-as-zero will just work. If your runningfmc.ko is older than what the driver wants to use, you may need to upgrade the core module, asthe driver wants to use data fields that are unknown to the core. We foresee to never introducebackward incompatibilities, so that step should be safe as well.

If your working place uses several drivers based on fmc-bus and you feature non-trivial depen-dencies, the safest best is using the most recent fmc-bus release you can find, and recompileall the rest against that version. Again, please remember that version changes are not frequentevents.

To help clarifying how the versions is used, here is an example.

Pretty often, the mezzanine drivers need to register a new device of some kind: a networkcard, an input device, or something else. Several mezzanines may be installed in the samehost computer, and we need a way to tell which is which: using sequential numbering is notreliable. The problem for diagnostics is already solved by the hwdev field in the device structure:mezzanine drivers use dev_err(fmc->hwdev) so the message includes proper identifiers (e.g. thebus and slot numbers for PCI cards).

When we moved the fine-delay driver from being a spec-only driver to a mezzanine fmc-busdriver, we found we needed an identifier to register our top-level device (here, a zio driver).The right solution to this problems is for carrier drivers to already spell out an identifier in thefmc_device structure, so mezzanine drivers can use it. This requires a version change.

Note: this change is not currently active, we are waiting to find other issues and cluster allrequired changes in a single version bump.

Drivers designed to work with an older versions of fmc-bus can just be recompiled: they won’texploit the new feature but they will work nonetheless. In this case, an old carrier driver willunknowingly register 0 as its identifier (and mezzanine drivers use “0” to trigger sequentialenumeration). An old mezzanine driver will not use the carrier-provided identifier and will usewhatever technique it used under the older fmc-bus version.

While writing the fine-delay driver, we chose to use some special spec-dependent code to extractthe identifier, until the core offered the new field. Carrier-specific code is explained in Section 9.4[Suggestions for Mezzanine Drivers], page 16.


9.2 Git Submodule Crash Course

The mechanism we suggest to manage dependencies is git submodule. It is already in place inthe existing fmc drivers, but using it requires some knowledge. This is some basic informationabout submodules, assuming you are already accustomed to simple git repositories. (Tutorialsexists, but they are usually too detailed as a quick start).

If you are just a user of fmc-bus, not developing drivers, you can jump to the next section.

Quick Summary

To make a long story short: a git repository can know that it uses a checkout of another gitrepository, at a specific version. The submodule is checked out using a commit identifier, so itmust be considered read-only by the supermodule (the top-level project).

If you checked out a specific branch of the submodule, the supermodule will record the specificcommit at the time of the checkout, not the branch it was using: branches are not followedautomatically nor even recorded in the supermodule.

Supermodules and Submodules

Then, here’s the longer story. When you clone the supermodule, the submodules won’t beautomatically cloned (unless you use git clone --recursive, that nobody does by defaultand is not generally good for fmc-bus drivers). To check out the submodules you need to git

submodule init; git submodule update or equivalent).

Whenever you modify the content of a checked-out submodule, it is reported by git status,that considers the directory where the submodule lives as a single file. Your established useof git status will thus continue to work. The messages you may get (if you changed thesubmodule) are either modified content or untracked content, like this (in the fine-delaymezzanine driver):

morgana$ echo > fmc-bus/new-file

morgana$ echo >> spec-sw//Makefile

morgana$ git status

[...]

# modified: fmc-bus (untracked content)

# modified: spec-sw (modified content)

If you wonder where does git store the checkout information for the submodule, it lives in the"tree object" (i.e. directory) where the submodule is checked-out.

morgana$ git ls-tree HEAD | grep commit

160000 commit dc30ce998628084c4402291e6052c7feb77be424 fmc-bus

160000 commit 35480a19bc562ebfa76f4663d41d026ff1b9df29 spec-sw

If you want a different commit in the submodule (for example, you fixed a bug in it), this istracked by a commit in the supermodule, because the hosting tree must be modified to recordthe new submodule commit. It is correct, in general, to have a commit in the supermodule thatjust updates the commit identifier of the submodule. Sometimes with, however, you commit thenew submodule’s version together with changes in the supermodule that use it (like they wereparts of the same package).

How to Commit to a Submodule

There are two situations where you need to touch a submodule: either you find a problem whileworking in the supermodule or you want to use a newer version of the submodule.

In the latter case: the submodule has new useful commits and we want to use them in thesupermodule: just check-out the new upstream commit in the submodule and then commit inthe supermodule.


The former case is trickier: since the submodule is a read-only checkout of another repository,normal users are not expected to commit to a project from the place where it is just a submodule.However, it sometimes happen when your multi-repository project is still young, like fmc-bus asof 2012.

The directory of the submodule looks like a normal git clone, so you can work in it in your usualway. However, the submodule is always referencing a read-only upstream repository (becauseeverybody should be able to get the code), so you must add a new “remote” git reference, likethis:

morgana$ git remote -v

origin git://ohwr.org/fmc-projects/fmc-bus.git (fetch)

origin git://ohwr.org/fmc-projects/fmc-bus.git (push)

morgana$ git remote add work $HOME/ohwr/fmc-bus

morgana$ git remote -v

origin git://ohwr.org/fmc-projects/fmc-bus.git (fetch)

origin git://ohwr.org/fmc-projects/fmc-bus.git (push)

work /home/rubini/ohwr/fmc-bus (fetch)

work /home/rubini/ohwr/fmc-bus (push)

Now you can work normally and push to work (if you don’t push, you may loose the commitwhen working in the supermodule, which doesn’t consider the submodule a working directory).If you end up using the new commit in the supermodule, you must commit the supermoduletoo as described above. Finally, you must ensure you push upstream the submodule before youpush the supermodule that uses the new commit.

9.3 Git Submodules and “make install”

In the context of fmc-bus drivers a mezzanine driver relies on the core fmc.ko and the carrierdriver. The user who downloads the mezzanine driver would like to “make && make install”and find everything installed.

While this approach is easily achieved by arranging for make install to install the submodulesas well, this will introduce problems for more advanced users, who are using different mezzaninesin the same host system. This is going to be pretty common, because you can drive fmcmezzanines with Etherbone, so drivers in a single host won’t be limited to the few PCI or VMEslots of your desktop computer or crate.

The suggested behaviour for an fmc driver package is using submodules for the drivers it de-pends on. However, it should only install its own kernel modules. If everything were installedautomatically, we risked overwriting already installed modules with a different version, and wecan’t know if the user really wanted that – those other modules may be already installed andverified as part of another set of drivers.

The suggested make install rule, thus, only installs the current package (the mezzanine orcarrier driver the user checked out) and prints this final line, explained by accompanying docu-mentation.

WARNING: Consider "make prereq_install"

If a user is running a host with this mezzanine alone, then installing the prerequisites is a safemove. In this case the warning message should not appear if you make install again after youdid make prereq_install once. That’s because technical users are expected to sometimes editand reinstall the specific driver, but not the ones it depends on – such drivers are consideredstable infrastructure, like the kernel or the compiler.


9.4 Suggestions for Mezzanine Drivers

A mezzanine driver should include fmc-bus and the relevant carrier drivers as submodules, inorder to easily access the needed headers. A mezzanine driver should compile the submodulesas well, in order to find the external symbols and not report scaring warnings at compile time.

While in a truly-portable mezzanine driver you won’t need to access headers of the carrier drivers,it’s still good practice to carry them as submodules, so the user can just make prereq_install

and have everything that’s needed to run the hardware item.

The examples shown here are taken from fine-delay, the first mezzanine driver that uses thismulti-level approach. That specific driver also uses zio as a submodule, but this has beenstripped from the examples as not relevant to fmc-bus. The device driver itself lives in the kernelsubdirectory, because the package also features tools and lib. This is expected to be common,so all code shown below uses this convention; adapting to a driver without subdirectories isexpected to be easy.

First of all, the toplevel Makefile should have all the standard targets, and run all of themin each subdirectory. Moreover, since most users don’t run “git submodule init” or similarcommands, it should be done automatically the first time to avoid errors that may be difficultto diagnose.

These are the relevant lines of the Makefile:.PHONY: gitmodules prereq prereq_install prereq_install_warn

all clean modules install modules_install: gitmodules

[...]

gitmodules:

@test -d fmc-bus/doc || echo "Checking out submodules"

@test -d fmc-bus/doc || git submodule update --init

# The user can override, using environment variables, all these three:

FMC_BUS ?= fmc-bus

SPEC_SW ?= spec-sw

SUBMOD = $(FMC_BUS) $(SPEC_SW)

prereq:

for d in $(SUBMOD); do $(MAKE) -C $$d || exit 1; done

prereq_install_warn:

@test -f .prereq_installed || \

echo -e "\n\n\tWARNING: Consider \"make prereq_install\"\n"

prereq_install:

for d in $(SUBMOD); do $(MAKE) -C $$d modules_install || exit 1; done

touch .prereq_installed

The snippet above implements the following:

• It checks out submodules if they are not already checked out.

• It allows the user to override the submodules and use custom versions. by setting environ-ment variables (FMC_BUS and SPEC_SW).

• It implements the warning at installation time.

• It has prereq_install, which stops the warning above.

The driver-specific makefile (here “kernel/Makefile”) must take care of finding the headers atcompile time and the external symbols at link time. This is implemented using the standardKbuild variables. Again, environment variable can be used to override the submodules withcustom clones of the repositories:

LINUX ?= /lib/modules/$(shell uname -r)/build

SPEC_SW ?= $(M)/../spec-sw

FMC_BUS ?= $(M)/../fmc-bus


KBUILD_EXTRA_SYMBOLS := \

$(FMC_BUS)/kernel/Module.symvers

ccflags-y = \

-I$(SPEC_SW)/kernel \

-I$(SPEC_SW)/kernel/include \

-I$(FMC_BUS)/kernel/include \

-I$M

If the mezzanine driver is truly portable (which fine-delay is not, as I write this), you won’tneed to point ccflags-y to $(SPEC_SW), which is only compiled and installed for the user’sconvenience.

On the other hand, your mezzanine driver might also need to point KBUILD_EXTRA_SYMBOLS tothe carrier driver, whereas the example above only uses symbols from fmc-bus.

9.5 Suggestions for Carrier Drivers

A carrier driver should include fmc-bus as a submodule, in order to easily access the relevantheader files. It should also compile the submodule, to access the symbols exported by fmc.ko

and avoid unpleasant compilation warnings.

However, the repository of a carrier driver may be used as a submodule of a mezzanine driver,which in turn already has fmc-bus as a submodule (see Section 9.4 [Suggestions for MezzanineDrivers], page 16). In general you don’t want fmc-bus to be replicated several times at differentdirectory levels if you clone a mezzanine driver. If, by chance, the user of the mezzanine driverchanges something in the fmc-bus, nobody can easily say which one is the one being re-installed.

For this reason, the carrier driver should not checkout the submodule if both the carrier itselfand the fmc-bus package are submodules of another driver. Actually, this is also why thegitmodules: rule suggested above for the mezzanine driver doesn’t use --recursive.

The following example is taken from spec-sw, the first carrier driver we developed for this busabstraction:

FMC_DRV ?= $(shell ./check-fmc-bus)

export FMC_DRV

DIRS = $(FMC_DRV) kernel tools

all clean modules install modules_install:

for d in $(DIRS); do $(MAKE) -C $$d $ || exit 1; done

As you see, the variable FMC_BUS is conditionally assigned, so the user can override it from theenvironment as usual. Moreover, the default value is returned by a local shell script.

The script check-fmc-bus verifies whether both this directory and fmc-bus are submodules ofsome other git repository living in the parent directory. If this is the case, FMC_BUS is takenfrom the parent directory, otherwise it is taken from the local subdirectory (the submodule).

9.6 Work in Progress

Unfortunately, these procedures still miss some bits, that will be dealt with as the systemincreases in complexity and number of users.

For example, the “gitmodules” target is missing from carrier drivers, so a developer workingdirectly on the carrier code should be aware of submodules, and run git submodule update

--init by hand after a fresh checkout. This is not yet performed by default because I need todifferentiate when the carrier driver is the main checkout from when it is a submodule.

As the last bits are fixed I’ll update this chapter, and I’ll remove this section when everythingis verified by a larger user base.

Chapter 10: FMC Identification 18

10 FMC Identification

The fmc standard requires every compliant mezzanine to carry identification information in anI2C eeprom. The information must be laid out according to the “ipmi Platform ManagementFRU Information”, where ipmi is a lie I’d better not expand, and FRU means “Field ReplaceableUnit”.

The FRU information is an intricate unreadable binary blob that must live at offset 0 of theeeprom, and typically extends for a few hundred bytes. The standard allows the application touse all the remaining storage area of the eeprom as it wants.

This chapter explains how to create your own eeprom image and how to write it in yourmezzanine, as well as how devices and drivers are paired at run time. eeprom programminguses tools that are part of this package and sdb (part of the fpga-config-space package).

The first sections are only interesting for manufacturers who need to write the eeprom. Ifyou are just a software developer writing an fmc device or driver, you may jump straight toChapter 11 [SDB Support], page 21.

10.1 Building the FRU Structure

If you want to know the internals of the FRU structure and despair, you can retrieve the docu-ment from http://download.intel.com/design/servers/ipmi/FRU1011.pdf . The standardis awful and difficult without reason, so we only support the minimum mandatory subset – wecreate a simple structure and parse it back at run time, but we are not able to either generateor parse more arcane features like non-english languages and 6-bit text. If you need more itemsof the FRU standard for your boards, please submit patches.

This package includes the Python script that Matthieu Cattin wrote to generate the FRU binaryblob, based on an helper libipmi by Manohar Vanga and Matthieu himself. I changed the testscript to receive parameters from the command line or from the environment (the command linetakes precedence)

To make a long story short, in order to build a standard-compliant binary file to be burned inyour eeprom, you need the following items:

Environment Opt Official Name Default

FRU VENDOR -v “Board Manufacturer” fmc-example

FRU NAME -n “Board Product Name” mezzanine

FRU SERIAL -s ‘Board Serial Number” 0001

FRU PART -p “Board Part Number” sample-part

FRU OUTPUT -o not applicable /dev/stdout

The “Official Name” above is what you find in the FRU official documentation, chapter 11, page7 (“Board Info Area Format”). The output option is used to save the generated binary to aspecific file name instead of stdout.

You can pass the items to the FRU generator either in the environment or on the commandline. This package has currently no support for specifying power consumption or such stuff, butI plan to add it as soon as I find some time for that.

FIXME: consumption etc for FRU are here or in PTS?

The following example creates a binary image for a specific board:

http://download.intel.com/design/servers/ipmi/FRU1011.pdf


./tools/fru-generator -v CERN -n FmcAdc100m14b4cha \

-s HCCFFIA___-CR000003 -p EDA-02063-V5-0 > eeprom.bin

The following example shows a script that builds several binary eeprom images for a series ofboards, changing the serial number for each of them. The script uses a mix of environmentvariables and command line options, and uses the same string patterns shown above.

#!/bin/sh

export FRU_VENDOR="CERN"

export FRU_NAME="FmcAdc100m14b4cha"

export FRU_PART="EDA-02063-V5-0"

serial="HCCFFIA___-CR"

for number in $(seq 1 50); do

# build number-string "ns"

ns="$(printf %06d $number)"

./fru-generator -s "${serial}${ns}" > eeprom-${ns}.bin

done

10.2 Using SDB-FS in the EEPROM

If you want to use sdb as a filesystem in the eeprom device within the mezzanine, you shouldcreate one such filesystem using gensdbfs, from the fpga-config-space package on OHWR.

By using an sbd filesystem you can cluster several files in a single eeprom, so both the hostsystem and a soft-core running in the fpga (if any) can access extra production-time information.

We chose to use sdb as a storage filesystem because the format is very simple, and both thehost system and the soft-core will likely already include support code for such format. The sdblibrary offered by the fpga-config-space is less than 1kB under LM32, so it proves quite up tothe task.

The sdb entry point (which acts as a directory listing) cannot live at offset zero in the flashdevice, because the FRU information must live there. To avoid wasting precious storage spacewhile still allowing for more-than-minimal FRU structures, the fmc.ko will look for the sdbrecord at address 256, 512 and 1024.

In order to generate the complete eeprom image you’ll need a configuration file for gensdbfs:you tell the program where to place the sdb entry point, and you must force the FRU data fileto be placed at the beginning of the storage device. If needed, you can also place other filesat a special offset (we sometimes do it for backward compatibility with drivers we wrote beforeimplementing sdb for flash memory).

The directory tools/sdbfs of this package includes a well-commented example that you maywant to use as a starting point (the comments are in the file called --SDB-CONFIG--). Readingdocumentation for gensdbfs is a suggested first step anyways.

This package (generic fmc bus support) only accesses two files in the eeprom: the FRU informa-tion, at offset zero, with a suggested filename of IPMI-FRU and the short name for the mezzanine,in a file called name. The IPMI-FRU name is not mandatory, but a strongly suggested choice;the name filename is mandatory, because this is the preferred short name used by the fmc core.For example, a name of “fdelay” may supplement a Product Name like “FmcDelay1ns4cha” –exactly as demonstrated in ‘tools/sdbfs’.

Note: sdb access to flash memory is not yet supported, so the short name currently in use isjust the “Product Name” FRU string.

The example in tools/sdbfs includes an extra file, that is needed by the fine-delay driver, andmust live at a known address of 0x1800. By running gensdbfs on that directory you can outputyour binary eeprom image (here below spusa$ is the shell prompt):


spusa$ ../fru-generator -v CERN -n FmcDelay1ns4cha -s proto-0 \

-p EDA-02267-V3 > IPMI-FRU

spusa$ ls -l

total 16

-rw-rw-r-- 1 rubini staff 975 Nov 19 18:08 --SDB-CONFIG--

-rw-rw-r-- 1 rubini staff 216 Nov 19 18:13 IPMI-FRU

-rw-rw-r-- 1 rubini staff 11 Nov 19 18:04 fd-calib

-rw-rw-r-- 1 rubini staff 7 Nov 19 18:04 name

spusa$ sudo gensdbfs . /lib/firmware/fdelay-eeprom.bin

spusa$ sdb-read -l -e 0x100 /lib/firmware/fdelay-eeprom.bin

/home/rubini/wip/sdbfs/userspace/sdb-read: listing format is to be defined

46696c6544617461:2e202020 00000100-000018ff .

46696c6544617461:6e616d65 00000200-00000206 name

46696c6544617461:66642d63 00001800-000018ff fd-calib

46696c6544617461:49504d49 00000000-000000d7 IPMI-FRU

spusa$ ../fru-dump /lib/firmware/fdelay-eeprom.bin

/lib/firmware/fdelay-eeprom.bin: manufacturer: CERN

/lib/firmware/fdelay-eeprom.bin: product-name: FmcDelay1ns4cha

/lib/firmware/fdelay-eeprom.bin: serial-number: proto-0

/lib/firmware/fdelay-eeprom.bin: part-number: EDA-02267-V3

As expected, the output file is both a proper sdbfs object and an ipmi fru information blob. Thefd-calib file lives at offset 0x1800 and is over-allocated to 256 bytes, according to the configurationfile for gensdbfs.

10.3 Writing to the EEPROM

Once you have created a binary file for your eeprom, you can write it to the storagemedium using the fmc-write-eeprom (See Section 8.4 [fmc-write-eeprom], page 11,while relying on a carrier driver. The procedure here shown here uses the spec driver(http://www.ohwr.org/projects/spec-sw).

The example assumes no driver is already loaded (actually, I unloaded them by hand as ev-erything loads automatically at boot time after you installed the modules), and shows kernelmessages together with commands. Here the prompt is spusa.root# and two spec cards areplugged in the system.

spusa.root# insmod fmc.ko

spusa.root# insmod spec.ko

[13972.382818] spec 0000:02:00.0: probe for device 0002:0000

[13972.392773] spec 0000:02:00.0: got file "fmc/spec-init.bin", 1484404 (0x16a674) bytes

[13972.591388] spec 0000:02:00.0: FPGA programming successful

[13972.883011] spec 0000:02:00.0: EEPROM has no FRU information

[13972.888719] spec 0000:02:00.0: No device_id filled, using index

[13972.894676] spec 0000:02:00.0: No mezzanine_name found

[13972.899863] /home/rubini/wip/spec-sw/kernel/spec-gpio.c - spec_gpio_init

[13972.906578] spec 0000:04:00.0: probe for device 0004:0000

[13972.916509] spec 0000:04:00.0: got file "fmc/spec-init.bin", 1484404 (0x16a674) bytes

[13973.115096] spec 0000:04:00.0: FPGA programming successful

[13973.401798] spec 0000:04:00.0: EEPROM has no FRU information

[13973.407474] spec 0000:04:00.0: No device_id filled, using index

[13973.413417] spec 0000:04:00.0: No mezzanine_name found

[13973.418600] /home/rubini/wip/spec-sw/kernel/spec-gpio.c - spec_gpio_init

spusa.root# ls /sys/bus/fmc/devices

fmc-0000 fmc-0001

spusa.root# insmod fmc-write-eeprom.ko busid=0x0200 file=fdelay-eeprom.bin

[14103.966259] spec 0000:02:00.0: Matching an generic driver (no ID)

[14103.975519] spec 0000:02:00.0: programming 6155 bytes

[14126.373762] spec 0000:02:00.0: write_eeprom: success

[14126.378770] spec 0000:04:00.0: Matching an generic driver (no ID)

[14126.384903] spec 0000:04:00.0: fmc_write_eeprom: no filename given: not programming

[14126.392600] fmc_write_eeprom: probe of fmc-0001 failed with error -2

http://www.ohwr.org/projects/spec-sw

Chapter 12: Portability 21

10.4 Reading back the EEPROM

In order to read back the binary content of the eeprom of your mezzanine device, the buscreates a read-only sysfs file called eeprom for each mezzanine it knows about:

spusa.root# cd /sys/bus/fmc/devices; ls -l */eeprom

-r--r--r-- 1 root root 8192 Apr 9 16:53 FmcDelay1ns4cha-f001/eeprom

-r--r--r-- 1 root root 8192 Apr 9 17:19 fake-design-for-testing-f002/eeprom

-r--r--r-- 1 root root 8192 Apr 9 17:19 fake-design-for-testing-f003/eeprom

-r--r--r-- 1 root root 8192 Apr 9 17:19 fmc-f004/eeprom

11 SDB Support

The fmc.ko bus driver exports a few functions to help drivers taking advantage of the sdbinformation that may be present in your own fpga memory image.

The module exports the following functions, in the special header <linux/fmc-sdb.h>. Thelinux/ prefix in the name is there because we plan to submit it upstream in the future, anddon’t want to force changes on our drivers if that happens.

int fmc_scan_sdb_tree(struct fmc_device *fmc, unsigned long address);

void fmc_show_sdb_tree(struct fmc_device *fmc);

signed long fmc_find_sdb_device(struct sdb_array *tree, uint64_t vendor,

uint32_t device, unsigned long *sz);

int fmc_free_sdb_tree(struct fmc_device *fmc);

To be completed.

12 Portability

This package should be portable. However I didn’t test it on a wide variety of system. I used iton a 32-bit x86 host, running version 3.4 of the kernel.

I verified that it compiles without any warning from version 2.6.35 up to 3.9, on a 32-bit x86processor. On previous versions it fails for various reasons, but the backport branch we push forevery release fixes them. The backport branch builds without warnings or error for 2.6.24 and2.6.26 up to 2.6.29 (we don’t care fixing it for 2.6.25). I don’t test kernels older than 2.6.24 asthat one is the olderst release we are using in production at the time being.

Documents

FMC Bus Abstraction for Linux - Open Hardware Repository · FMC Bus Abstraction for Linux ... an advanced user can also use sysfs to change the binding of ... The contents of the