the FC7 AMC for DAQ & control applications in CMS

project

the FC7 AMC for DAQ & control applications in CMS

Mark Pesaresi (Imperial College), Paschalis Vichoudis (CERN)

Magnus Hansen, Manoel Barros Marin, Francois Vasey (CERN) Greg Iles, Sarah Greenwood, Andrew Rose, Geoff Hall (Imperial College)

| FC7 | Mark Pesaresi | TWEPP 2014 2

content

introduction - concept - motivations - specifications

hardware overview

- highlights - high frequency performance

acceptance testing & manufacturing

- test stands - manufacturing & production experience

uses in CMS summary


concept

the FC7 AMC is a flexible, uTCA compatible card for generic CMS data acquisition & system control uses

the FC7 is targeted for

o both production systems or prototyping / bench-top use

o uTCA for CMS applications (compatibility with AMC13) or standard uTCA

o users of high speed optical links up to 10Gbps

Imperial MP7

CERN GLIB


design motivations (I)

basing on existing hardware helps to reduce development time to maturity

o especially with regards to reuse of firmware and software components

o fewer nasty surprises!

designed to be an evolution of

o the Gigabit Link Interface Board (GLIB) – a highly

successful project filling a similar role, with emphasis on GBT

link testing

using design components, novel features and layout guidelines from

o the MP7 – a high optical density O(Tbps) processing card to

be used in the upgraded CMS Calorimeter Trigger


design motivations (II)

principally designed with two CMS users in mind

o the Trigger Control & Distribution System (TCDS) – rationalisation and upgrade of existing TTC

control system for installation in 2014/5; distribution of LHC clock, fast (e.g. trigger) and slow

commands, reception of synchronous detector status, centralised trigger rules, more partitions

o the Pixel Front End Driver (pixFED) – replacement front end board to acquire data from the

upgrade pixel detector, to be installed in 2017; 400Mbps digital optics from front ends, increased

readout bandwidth to DAQ, robustness to readout errors, increased processing capability










- compatibility with uTCA for CMS (AMC13)

- low jitter clock distribution & deterministic latency

- capability to run 10Gbps SERDES links










- compatibility with uTCA for CMS (AMC13)

- low jitter clock distribution & deterministic latency

- capability to run 10Gbps SERDES links

additionally: continued capability to run GBT (4.8Gbps) systems


FC7: top view

looks like a

simple board…

what’s the

fuss?


FC7: bottom view

OK

that might

explain why

it’s so heavy…


main features

- full size, double width AMC module for uTCA

- Kintex 7 FPGA supporting serial links up to 10Gbps

- 4Gb DDR3 RAM (128M x 32b)

- 2 FMC sockets fully compatible with Low Pin Count FMCs

o additionally supports up to 20 high speed serial

lanes to the front panel

o supports dual-width, legacy & non-standard FMCs

- flexible backplane configuration

o high speed connectivity on up to 12 ports

o fully compatible with uTCA for CMS and AMC13

dual QSFP+

8 x SFP+

2x 14b A/D, 2x 16b D/A


development history

November 2012

June 2013

July /August 2013

September 2013

November 2013

January 2014

June/July 2014

Today

Spec approved

First prototypes (R0a) submitted

First prototypes received Majority of design validated, one minor design issue Some serious manufacturing issues uncovered

Begin assembly of test system & firmware Informal price enquiry to trial new manufacturers

Pre-production series (R0b) submitted with two manufacturers

Pre-production batches received Excellent manufacturing quality with one supplier Order for 70 pieces placed

70 production boards received (TCDS)

Minor design update (R1) submitted Order of 120 pieces in production (pixFED)

dev

elo

pm

ent

pro

du

ctio

n


feature highlights

Kintex 7 FPGA

high range Kintex 7 XC7K420T

pin compatible with XC7K480T (difference only in logic)

FMC 1

FC7 is FMC spec compliant but can also accommodate non-standard mezzanines &

stacking heights

‘-2’ speed grade – MGTs support 10Gbps (10.3125Gbps max)

supports almost any MGT protocol to FMCs (e.g. GBT, 10GbE, custom)

12 MGT pairs

68 user IO (34 differential)

I/O operates up to 3.3V signalling - supports legacy FMCs


feature highlights

switched JTAG header FPGA, microcontroller, FMCs

backplane JTAG option

unique laser etched QR code unique EEPROM h/w address

automatic logging & inventory,

configuration with CERN assigned MAC address on acceptance

external 12V power header

desktop/crate mode switch (requires additional h/w for GbE control)


feature highlights

DDR3SRAM

AM

C c

on

ne

cto

r

4Gb DDR3 specified up to 480MHz

30Gbps maximum bandwidth (R+W)

designed to be compatible with both uTCA for CMS and

other AMC.x standards

supports range of MGT protocols across backplane

Port 0 GbE

Port 1 GbE or AMC13-DAQ*

Port 2 SATA/SAS

Port 3 SATA/SAS or AMC13-TTC/S (LVDS)

Ports 4-7 PCIe/SRIO*

Ports 8-11 Ext Fat-Pipe (SRIO/10GbE XAUI)*

Ports 12-15 LVDS I/O

Ports 17-20 unused

FCLKA AMC13-TTC or PCIe clocks

TCLKA/TCLKC telecoms clock inputs

TCLKB/TCLKD telecoms clock outputs


feature highlights

flash

uC

uSD

firmware image repository including a fallback inviolate ‘Golden’ image

allows fast write (~10s) speeds & instant

FPGA reconfiguration over IPBus (compared to 5min for JTAG->SPI flash)

ATMEL microcontroller runs imperial_mmc v1.6 code

(shared with MP7)


feature highlights

16 layer PCB Nelco N4000 13-EPSI - low loss tangent for HF - low dielectric constant -misaligned with PCB weave

strict constraint on thickness


prototype performance testing

electrical TX testing at 10Gbps

XILINX SERDES electrical breakout FMC (XM104)

optical TX & loopback testing at 10Gbps

Faster Technology 8x SFP+ FMC (FM-S18)

20ps total jitter

1.7ps ch-ch spread

specification <27ps

BER <10-12 (PRBS31)

performance limited by

optical margin/SFPs


production acceptance testing (I)

GPIB-controllable power supply

GPIB-USB controller

ATMEL programmer (microcontroller configuration)

XILINX programmer (CPLD configuration)

SD card (FPGA configuration)


production acceptance testing (II)

AMC extender card (CERN) Ports 1-15 loopback Port 0 IPBus control

electrical loopback FMCs (commercial)


production acceptance testing (III)

- visual inspection

- scan QR code

- connect to the testbench

1. Power up 2. Configure CPLD 3. Configure microcontroller 4. Power cycle to load FPGA (from SD) 5. Check ethernet link with FPGA (ping) 6. Verify on-board voltages (via the monitoring circuitry) 7. Identify the board (i2c readable S/N) 8. Assign MAC address (from LUT based on label) 9. Run DDR3 memory test (@960MT/s) 10. Run FMC loopback test (standard I/O) 11. Increase current limit for SERDES tests 12. Run FMC loopback test (SERDES @ 10Gb/s) 13. Run AMC loopback test (SERDES @ 5Gb/s) 14. Power down


production acceptance testing (III)

- visual inspection

- scan QR code

- connect to the testbench

1. Power up 2. Configure CPLD 3. Configure microcontroller 4. Power cycle to load FPGA (from SD) 5. Check ethernet link with FPGA (ping) 6. Verify on-board voltages (via the monitoring circuitry) 7. Identify the board (i2c readable S/N) 8. Assign MAC address (from LUT based on label) 9. Run DDR3 memory test (@960MT/s) 10. Run FMC loopback test (standard I/O) 11. Increase current limit for SERDES tests 12. Run FMC loopback test (SERDES @ 10Gb/s) 13. Run AMC loopback test (SERDES @ 5Gb/s) 14. Power down


manufacturing experiences

submitted prototype and pre-production series runs with two different manufacturers

o one well known to CERN/UK, one with no prior experience of

o experimented with two PCB materials compatible with the impedances & high speed

signal integrity required

large fraction of development period dedicated to tracking down & resolving manufacturing issues

o two manufacturers extremely useful to identify manufacturing vs. design faults (whether

due to designer error or board complexity)

o have learnt the importance of good, two-way communication with your supplier


example 1: PCB issues

for 10Gbps we can’t use simple FR4…

Nelco N4000 laminate (Park) has very good high frequency characteristics

but now known to be susceptible to delamination & registration defects

o also an expensive material, low yield

o requires careful handling during manufacture & assembly;

e.g. contamination, humidity, life time, temperature profiles

o working guidelines between PCB/assembly manufacturers compiled

TU-872 SLK laminate (TUC) trialled successfully but also tends to suffer

similar registration defects – special measures during construction

even more fancy HF materials now available but even less well known

o key is building communicative relationship with supplier real x-ray representations


example 2: assembly issues

concerning assembly, one of the suppliers demonstrated an excellent

quality of work

continuous improvement in the assembly process

o detailed written feedback provided after each batch

o use X-ray imaging to identify soldering issues

o identified washing process behind an issue with a mechanical

switch – assembly process changed accordingly

o improved packaging guidelines after damage during transport


pre & production yield

best manufacturer chosen for supply of production FC7s

total 80 pre-production and production boards manufactured between Nov ’13 - Jun ‘14

70 passed all tests first time around, 8 returned to manufacturer

78 passed tests second time around

o 1 board damaged in transit

o 1 board with unidentified issue on backplane

final yield >97%


firmware

mature firmware

development

- based on IPBus

system related logic is

segregated from user

reuse of GLIB f/w blocks,

similar user logic structure

suite of scripts for

configuration over IPBus

high level f/w diagram


FC7 for CMS TCDS

with 3 custom FMCs, the FC7 satisfies the role of 2 of 3 objects for the TCDS

a local partition manager <-- interfaces with AMC13, DAQ, ext. clk/trig, partition interface cards

a partition interface <-- interfaces with partition managers, new (uTCA) & legacy FEDs

AMC13

local partition managers

partition interface cards

P5


FC7 for CMS TCDS

with 3 custom FMCs, the FC7 satisfies the role of 2 of 3 objects for the TCDS

a local partition manager <-- interfaces with AMC13, DAQ, ext. clk/trig, partition interface cards

a partition interface <-- interfaces with partition managers, new (uTCA) & legacy FEDs

AMC13

local partition managers

partition interface cards

TTC/TTS to-from AMC13

to DAQ

TTC/TTS

TTC/TTS to-from FEDs

P5


FC7 for CMS pixFED

with a single custom FMC, the FC7 can fully satisfy the role of the pixel FED

2x 12 channel parallel pluggable optical receivers (Fitel)

- running at 400Mbps, reads in data directly from detector

SFP+/QSFP+ required for DAQ (not pictured)

- 10Gbps readout, to be prototyped

total 70 boards required for pixFED, possibly more

with the TCDS FMC, the FC7 could also satisfy the role

of the pixel Front End Controller (FEC)

nothing special required, R&D ongoing

~30 boards required for pixFEC


summary

flexible Kintex 7 based uTCA compatible AMC developed for generic DAQ/detector control uses in CMS

o suitable both for small scale test systems or production systems , compatible with a range of FMCs

o able to operate up to 10Gbps line rates across the front panel

o advanced FPGA configuration features

successful production of 80 pieces to date, another batch of 120 currently in production

o short development timescale

o mature system firmware, isolated from user firmware, test script suite

stress the importance of good communication with your supplier

o especially helpful when requesting non standard PCB constructions

o automated acceptance testing for quickly identifying manufacture issues

FC7s already being installed in P5

o TCDS ready for operation in 2015

o pixel FED development under way


links

https://espace.cern.ch/project-FC7

firmware & software:

https://svnweb.cern.ch/cern/wsvn/ph-ese/be/fc7/

public documentation:

https://svnweb.cern.ch/cern/wsvn/ph-ese/be/fc7/trunk/fc7_r0/doc/

project

thank you for your attention

with special thanks to: Jan Troska, Jeroen Hegeman, Karl Gill










Documents

the FC7 AMC for DAQ & control applications in CMS