Level-1 Trigger Upgrade Status

LHC CMSDetectorUpgrade

Project

PMG: Trigger Upgrade Status 1

Level-1 Trigger Upgrade Status

Wesley Smith, U. WisconsinUS CMS Project Management GroupDecember 18, 2013

18 Dec. 2013, Wesley Smith


Project


L5 - MPCM Software and Firmware Start 401.04.03.02.01 2013-11-01 Initial firmware version is ready, software for test stand is being built.

L5 - MPCM Production Start 401.04.03.02.02 2013-11-01 Started, 2 pre-production mezzanines are at Rice and being tested. All the parts

except optical transmitters are at Rice. Should be ready to start assembly in a few weeks.

L5 -Start MPC-EMUTF Optical Fibers 401.04.03.03 2013-11-01 Short optical pigtails for MPC boards are at Rice. Out of 36 trunk cables for the

EMUTF 27 have been delivered to CERN and tested there. The remaining longest 100+ m cables are expected from Draca in January. Installation at p.5 is planned for spring.

L5 -Start Muon Sorter 401.04.03.06 2013-11-01 Preliminary work on hardware design started.

L5 -Start Trigger Production 401.04.03.06 2013-11-01 Started, covered above

L5-Start MSM Production 401.04.03.06.02 2013-11-01 Not started yet


L5 Active Mu. Trig. Milestones - Rice


Project


Muon Port Card Status (1)

Received 90 optical pigtails from CERN in November Fabricated 81 new front panels

All parts for 80 mezzanine boards have been ordered and most of them are already at Rice. Optical transmitters from Avago should arrive in the second half of December.

85 optical transmitter boards have been fabricated and assembled in October



Project


Implemented few minor layout changes in the mezzanine design.

Two pre-production boards have been assembled by Pactron in the middle of November and are being tested at Rice.

Muon Port Card Status (2)



Project


Complete testing of the two pre-production mezzanines by the end of December

Assemble 80 production boards in the 1st half of January

Test 80 new mezzanines at Rice in January-March Set up a uTCA test stand at Rice in January-March Installation of 36 optical trunk cables at p.5: spring

2014 Installation of the upgraded Muon Port Cards at p.5:

late spring- early summer 2014

Muon Port Card Mezzanine Plans



Project


L5-Start EMUTF Test, Install, and Commission 401.04.03.04.03 2013-11-04

A control PC of the type recommended by CERN was ordered for tests in Florida

L5 -Start EMUTF Infrastructure 401.04.03.05 2013-11-01 A control PC of the type recommended by CERN was ordered for tests in Florida First Optical Patch Panel Prototype delivered (next slide)

L5 -Start EMUTF Infrastructure Production 401.04.03.05.01 2013-11-01 Started, covered above

L5-Start EMUTF Software and Firmware 401.04.03.04.01 2013-11-04 A first version of the software emulator (for the initial endcap TF algorithm) has

started. It is nearly ready to be used in CMSSW, but likely will occur in early 2014. Initial firmware for all MTF cards (FPGA control board, optics board, and PT LUT

mezzanine board) has been written and already used for testing of the prototypes.

L5-Start EMUTF Production 401.04.03.04.02 2013-11-04 The preproduction of the MTF7 board with Virtex-7 has started with the submission

for PCB manufacture this December, and components procured.


L5 Active Mu. Trig. Milest. - Florida


Project


Endcap Muon TF block diagram


Optical plant(fanouts and splitters)

From MPCs60 12-core fibers, 8 cores used in each.90 trig. primitives per 60° sector3.2 Gbps

From RPCUp to 216 fibers at 1.6 Gbps (may be concentrated to higher bandwidth and fewer fibers)

Sector Processors12 units60° sector each

To Overlap Track Finder

Best 3 Muons in each sector (36 total)To GMT


Project


EMUTF Optical Patch Panel


Prototype patch panel to split, fan-out and bundle fibers to appropriate processor inputs.

Units are only 1 U wide.


Project


Sector Processor (SP) o Each occupies 2 uTCA slotso 12 units in system

All chassis use AMC13 (designed by Boston University) o Clocking, TTC, and DAQo 3 units

Plan to control boards via PCI express for development (faster)

But will be compatible with IPbus as well for operations per CMS standard

o IP-based protocol for controlling hardware devices adopted by CMS.


uTCA chassis

SPSP SP SP SP

SPSP SP SP SP

SP SP

Chassis #1

Chassis #2

Chassis #3


Project

PMG: Trigger Upgrade Status

Modular Muon Track Finder (MTF)

18 Dec. 2013, Wesley Smith 10

Modularity makes future partial upgrades easier (e.g. FPGA) Card layout is simpler Power density is less

Optical module

Custom backplane connector

Core logic module

Optical module

Core logic module

Pt LUT module

Custom backplane


Project


Optical module


uTCA connector


Backplane redrivers

Optical receivers(2 out of 7 installed)

MMC

Optical transmitters (2 out of 3 installed)

Tested, Done! Works with either V6 or V7 MTF versions


Project


MTF6 Core logic module (Virtex6)



Core logic FPGA

Control FPGA

uTCA connector

Control FPGA JTAG

PT LUT module connector

FMM connector

SD card connector

MMC USB console

MMC JTAG

MMC CPU

Estimated power consumption: ~50 W (assuming FPGAs nearly full)PT LUT mezzanine not included

1Gb FLASHMain FPGA firmware storage

MMC = Module Management Controller

Tested


Project


MTF7 design (Virtex-7 version)


Core logic module FPGAs:

Core logic: XC7VX690T-FFG1927 or XC7VX550T-FFG1927 Control: XC7K70-FB676

Inputs: 80 Virtex-7 GTH links (10 Gbps) directly to Core FPGA

o Minimal latencyo All available receivers are designated for trigger data

– Maximum flexibility 4 additional Kintex GTX links (10 Gbps) to Control FPGA

o Delivered to Core FPGA via parallel channelo Longer latency

Outputs: 28 Virtex-7 GTH links (10 Gbps)

Control: PCI express Gen 2, 2 lanes IPbus


Project


MTF7 Base board


Main FPGA

Control FPGA

DC-DC converters(switchers)

Low-DropoutLinear regulators

MMC

PT LUT connector

Custom backplaneconnector

uTCA connector

Layout finishedRFQ for board production submittedAssembled boards should arrive by end of December


Project


Glue logic FPGA(Spartan-6)

Base boardconnector

RLDRAM3 memory16 chips, 8 on each side(clamshell topology)Total size: 512M x 18 bits ≈ 1GBUpgrade possible to2 GB with bigger RLDRAM3 chips(no board redesign)

DC-DC converters

Clock synthesisand distribution

PT LUT module


Tested! Works with either V6 or V7 MTF core logic modulesInvestigating further latency reduction


Project


Optical communication test


@3.2 Gbps 47 input channels Transmission from:

o Loopbacko Muon Port Cardo Earlier VME prototype (2010)

For MPC and VME prototype clock was synchronized with VME crateo Twisted pair LVDS connection to uTCA backplane

@10 Gbps 6 input channels Asynchronous clock Transmission from:

o Loopbacko Earlier 10Gbps

prototype (2006)

Results Zero errors for hours

Eye pattern @10 Gbps.GTH receiver input.


Project


Integration tests


Mimicking real system setup: Sending data from 2 MPCs to

MTF6 Each MPC clocked by separate

CCB MTF6 receives clock from AMC13 CCBs and AMC13 receive clock

from common source (TTCvi)

Types of tests:o PRBSo Random data via test FIFOso Latest MPC data format used

Result:o No errors.


Project


Optical components tests


Trunk cable identical to what will be used for Endcap TF Total length (126 m) exceeds actual max fiber length in Endcap TF system (113 m) Results:

o @3.2 Gbps: no errors with 2-way and 4-way splitting [CSC signals]o @1.6 Gbps: 2-way splitting OK, errors with 4-way [RPC signals]o Newer optical receivers lose efficiency at low bitrates

MPCor

MTF6MTF6

63-meter trunk cable(2 lengths)

Couplers

FanoutSplitters Fanout

Endcap TF needs:2-way splitters: 3484-way splitters: 60Rate: 3.2 Gbps


Project


PT LUT tests


Parameters: RLDRAM clock : 200 MHz Address & control: 200 Mbps each bit Data: 400 Mbps each bit RLDRAM can tolerate up to ~1GHz clock. However:

o Hard to implement in FPGAo Needed for burst-oriented applications mostlyo Does not change latency for random address accesso Lower clk F lower power consumption

Tests performed (random data, full 1GB space): Writing into consecutive addresses Reading from consecutive addresses Reading from random addresses

No errors detected Except soldering defects in one RLDRAM chip


Project


L5-Start Calorimeter Trigger Processor 401.04.04.02 2013-11-01 Two Prototype Boards delivered this week

L5-Start CTP7 Software and Firmware 401.04.04.02.01 2013-11-01 Initial SW and FW for Prototype Operation is being prepared

L5 -Start CTP7 Production 401.04.04.02.02 2013-11-01 First two prototype PCB delivered, parts received and assembly underway (see next slides) Ordered and received 8 Virtex-7 FPGAs.

L5 -Start CIOx Card 401.04.04.03 2013-11-01 Prototype CIOx card made and is adequate for use.

L5 -Start CIOx Software 401.04.04.03.01 2013-11-01 Included in CTP6 software package

L5 -Start CIOx Production 401.04.04.03.02 2013-11-01 Production needed in 2016. Will reschedule.

L5 -Start CTP Infrastructure 401.04.04.04 2013-11-01 Have 3 Vadatech Crates and optical fibers for interconnects in hand.

L5 -Start oRM 401.04.04.05 2013-11-01 oRM Production underway (not US cost – this item is personnel for installation)

L5 -Start oSLB-oRM Optical Fibers401.04.04.06 2013-11-01 Fibers have been purchased (not US cost – this item is personnel for installation)


L5 Active Cal. Trig. Milest. – Wisc.


Project


Current & Stage 1 Cal. Trigger

Stage 1 Calorimeter Trigger Upgrade Improved processing of current RCT with new oRSC Converts current system to optical New HF (fwd calo) data is available to upgrade path Prepare for the Stage 2 Upgrade Layer 1 cards will also read out RCT in Stage 1



Project


Two oRSCs are at 904 since mid-October Integration Row & Lab Test Crate

Continuing to validate FPGA builds at UW

Finishing up VME access now New oRSC code to test VME VME-Spartan BE-Kintex FE working

Integrating oRSC into Trigger Supervisor Using new oRSC code

oRSC



Project


Stage 2 Calo Trigger – Layer 1

Layer 1 cards will receive the calorimeter trigger primitives via optical fiber from ECAL TCCs with oSLB 4.8 Gbps optical to oRM on legacy RCT

o To be installed this wintero Send duplicate to layer-1

HF and HCAL mHTRso Passive optical splitting before HTRs for deployment of mHTR in parallel



Project


CTP7 Concept

Main Layer-1 processor card

• Virtex-7 690T for processing

• ZYNQ for TCP/IP + Linux

• 60 10Gbps optical Input links

• 36 10Gbps opticalOutput links

• Function: Find ET clusters & transmit to Layer-2

• Card Count = 46• 36 total +

8 spares + 2 test setups

Virtex-7 VX690T FPGA

ZYNQXC7Z03

0EPP

1.5V

Su

pply

2.5V

Su

pply

3.3V

Su

pply

1V 3

0A

Supp

ly CXP Module12Tx + 12 Rx

CXP Module12Tx + 12Rx

CXP Module12Tx + 12 Rx

12XRx12XRx

(CTP-6 BG View)



Project


Calorimeter Trigger Processor Prototype for Layer-1

4 Rev. A and 8 Rev. B CTP6s have been built 48 input links in 4 Avago AFBR-820B Rx Modules 12 output links in 1 Avago AFBR-810B Tx Modules Back-end & Front-end FPGA XC6VXH 250T/380T Dual SDRAM for DAQ and TCPIP buffering Custom power modules working at 1.0V, 1.5V, & 2.5V Embedded Microblaze processor with Linux on back-end FPGA Custom MMC for control

o Phased power-supply enableso Add’l analog voltage, power good and FPGA Load Done sensors

Parallel FPGA Flash - ~6 seconds to load both FPGAs SI5338 Prog. Clock Synthesizer, multiple options for Reference clocks

o 4.8 Gbps: 120, 240, 300 MHz optionso 6.4 Gbps: 160, 320, 400 MHz options

Nelco 4000-13 SI laminateo Lower losses & better εr for 50Ω trace geometry



Project


CTP6 Layer-1 Prototype

MMC Power Supplies

BEFPGA

FEFPGA

AvagoRx (4)

Avago TxDual SDRAMFor DAQ & TCP/IP

JTAGUSBInterface



Project


CTP6 Back Side

Inter-FPGA Links

Link Clock Fanout & Distribution

12X Frontpanel Tx Link Source-Select Muxes (Front or Back-End FPGA)



Project


UW Test Crate

BU

AM

C13

Vada

tech

M

CH U

W C

TP-6

UW

CTP

-6

TTC Downlink

UW

Aux

(Final system: 3 crates w/ 12 CTP7 ea. + 2 test setups + spare = 6)

U. Wisconsin designed backplane with dense card interconnectsmanufactured & installed in commercial Vadatech VT892 Crate available in Vadatech Catalog



Project


CTP6 Status

CTP6 inter-board link tests show wide margins

HF from mHTR to CTP6 integration successful 12 links tested at 6.4 Gbps – no errors RAM based data between

One CTP6 at 904 to continue SW work, HW Integration18 Dec. 2013, Wesley Smith


Project


oRSC to CTP6 Integration Test

Data patterns successfully transferred from the oRSC, captured on the CTP6 and readout via embedded Linux

• Fiber capture RAM readout

• SW controlled capture• Capture RAM clears

on each capture• All six links capture

expected pattern from oRSC• Automatically

checked by host



Project


CTP6 and 7 Status

CTP6 at 904 at CERN since Mid-october Checked out Necessary tools work Most effort with oRSC at the moment

CTP7 prototype board in UW lab, post-production final assembly Installation of CXP modules – new innovation – see next slides



Project


Conforms to 100GbE and 12x10Gbps Infiniband standards

12 optical lanes in each direction MPO-24 optical connector Multi-rate capable from 1 Gbps to 10.5 Gbps 850 nm wavelength Single 3.3V supply Uses same Avago MicroPOD engine that is used in

the Avago MiniPODs Optically compatible with Wisconsin oRSC Card and

Imperial College MP7 Card

Pluggable frontpanel interface means module can be replaced without a card/crate PCB extraction cycle


Avago AFBR-83 CXP Module


Project


CXP Module, cont’d

Optical Tx Board

Optical Rx Board

MicroPod Optical Rx Interface

12-fiber Ribbon to Optical Tx

Engine

AC coupling caps(external caps not

req’d)

MicroPod Tx Optical

Engine (bottom

side)18 Dec. 2013, Wesley Smith


Project


CXP Cage Press-Fit Insertion Operation

Post-reflow operation

84 electrical pins, 4 screws and 2 alignment pins per housing

Electrical pins are press-fit

Requires a tool for properly force application, and a fixture for bottom side board support

Designed and made a fixture for the CTP7

Molex I-Pass Plus EFConnectorP/N 170465-0001

Tool centers insertion force over

electrical pins

CTP7 PCB

Molex Insertion Tool

Fixture



Project


CTP7 CXP Board Hole Pattern

84 Electrical holes are spec’d for 14.6 mils, ±2 mils

Finished hole diameter spec should be given to the PCB vendor, letting them work backwards from there

PCB Decal has ringed border to chassis ground

Alignment & Screw Retaining Holes

Electrical Connection Press-Fit Holes

CTP7 PCB

Ground Vias (Smaller Holes)



Project


CTP7 CXP Press-In Fixture

Milled from ¾” aluminum stock

For use after components are soldered on to the board

Clearances for bottom side components

Provides critical board support immediately under the 84 electrical pins

Custom-designed to match CTP7 board design

Relieved for Bottom Side Component Clearance

Direct board support immediately under the CXP Connector electrical Holes (3 places)

Kapton tape for anti-gouge protection

Alignment Pin



Project


CXP Modules on CTP7 Each CXP module provides 12x lanes of optical Tx and

Rx, at speeds between 1Gbps and 10.5Gbps CTP7 uses 3 CXP modules for front side optical

connection Front panel mounting simplifies PCB design Avago CXP Modules use same optical engine as Avago

MiniPODs for complete compatibility with other CMS boards

Press-fit receptacles required design of special CTP7 holding fixture during the press-in process

Front-panel pluggable design means board extraction not required to replace module



Project


Embedded processing

Xilinx provides two embedded solutions Soft – Microblaze, on CTP6 Hard – ZYNQ, on CTP7

PetaLinux is an embedded Linux product from Xilinx Runs on Microblaze and ZYNQ platforms Standard, debugged drivers for SPI, I2C, Ethernet, USART,

Flash, etc.

Communications and control functions are most easily handled by processors and software

Advantages General communications & control functions

originally requiring HDL design now can be done by SW engineers writing Linux Apps

More SW engineers than FW engineers

Deployed on CTP6 & oRSC using Microblaze

On the CTP7 using ZYNC processor



Project


CTP7 Zynq SoC Linux Control Chip

•ZYNQ has hard ARM core

•embedded Linux on transforms housekeeping tasks into Linux application

•Xilinx AXI chip-to-chip exposes Target FPGA into ZYNQ memory space

•Can store FPGA bitfiles on μSD FAT filesystem for easy upgrades via SCP/pop in a new μSD

SD

RA

M

Low Speed Peripherals (EEPROM, Clock Synths,

Optical Modules, Crosspoint Switches,

Console, etc.)

TargetFPGA

AXI Chip-to-ChipInterface

SPI Config Interface

GbE Connection

ZYNQSoC

(Linux)

microSDFlash

Memory

Target AMC Board



Project


CTP Software

CTP6 has two FPGAso Front-end connected to optical linkso Back-end connect to Etherneto Inter FPGA Communication via serial bus: UART

CTP7 follows the same model, replace BE by ZYNQ chip Development going on now should be portable



Project


CTP6 Software All of the FPGAs have a Microblaze core

On the CTP6 BE, PetaLinux is installed So we use IPBus protocol, in our “softipbus” implementation available in the

CACTUS SVN repository softipbus includes:

o A TCP server that runs on Linux and interprets the byte stream as IPBuso A TCP server that runs on Linux and forwards the byte stream along a file

descriptoro Functions to de-serialize a byte stream into IPBus transactionso Functions to handle IPBus transactions (READ/WRITE/RWM)o Functions to serialize the responses into a bytes stream

Code is written so it can run on Linux as a server or be used as library on the standalone deviceso All the standalone programs are in github

– Problem Tracking and Wiki too



Project


MMC Project: IPMI System Manager

•μTCA features Gigabit Ethernet to address individual slots and IPMI to manage the power and sensors of individual cards

•Commercial IPMI management tools available, but

•Vendor specific

•Not extensible (GUIs)


IPMI: Intelligent Platform Management InterfaceMMC: Memory Management Controller


Project


IPMI System Manger


•API wrapping a socket interface•Act as a gateway for other applications

into the IPMI system•Provide a robust IPMI-over-LAN

connection to crates•Provide a service for automatic low-

level initialization of AMC cards at startup

Crate MCH Crate MCH Crate MCH


Project


Complete oRSC testing and manufacture cards Test prototype CTP7 in Madison Lab and then bring

to CERN for integration tests with MP7 card Work on implementation and optimzation of Stage-1

algorithms Design and Performance documented in Level-1 TDR Collaboration with FNAL and UIC

Continue Firmware development for oRSC and CTP7

Integrate Software for oRSC and CTP7 with online Trigger Supervisor system


Calorimeter Trigger Plans

Documents

Level-1 Trigger Upgrade Status