Agata Week – LNL 14 November 2007

Global Readout System for the AGATA experiment

M. Bellatoa

aINFN Sez. di Padova, Padova, Italy

Agata Week – LNL 14 November 2007

AGATA Experiment:

4array of HPGe detectors for in-beam -ray spectroscopy

36 fold segmented crystal + central core contact

x 180

Digital electronics and sophisticated Pulse Shape Analysis algorithms

Operation of Ge detectors in position sensitive mode -ray tracking

Agata Week – LNL 14 November 2007

AGATA Readout:

Channel per crystal: 36 + 1 per crystalTotal channel: 6660

Local Level Trigger

Central core processing Central trigger processor

Global Trigger

Global Trigger and Synchronization control system (GTS)

Global time reference

The design of FE readout and L1 trigger follows a synchronous pipeline model: the data are collected at 100 MS/s and stored in pipeline buffers at the global AGATA frequency waiting the L1 decision

Agata Week – LNL 14 November 2007

Agata Front-end Model

Agata Week – LNL 14 November 2007

Choice of Readout Standard

Candidates

• VME/VXI• Past Experience with older detectors

• Bandwidth limited • CompactPCI

• First version of AGATA Readout electronics• Advanced TCA

• Present version

Agata Week – LNL 14 November 2007

• Pros– Very stable– Huge choice of commercial boards– Slave implementation very simple– Wide ecosystem in HEP– Recently extended with serial switched capabilities

• Cons– Slow readout (max 40-80) Mbytes/s

• Enhancements (CBLT) not fully standard

– Limited power dissipation per slot– Master implementation not trivial– If large estate needed then 9U crates expensive– Limited customizability on the bus– Crate control not standard– No power redundancy -> limited availability

VME

Agata Week – LNL 14 November 2007

• Pros– Fast ( > 500 Mbytes/s)

– Huge choice of commercial boards

– Low cost (Telecom standard)

– Wide choice of IP cores available for interfacing (both initiator/target)

– Extended to serial switched with CPCIExpress

– Highly customizable through user reserved backplane connectors

• Cons– Small estate

– Low power dissipation per slot

– Low no. of slots per bus without bridging

– No standard crate control

CompactPCI

Agata Week – LNL 14 November 2007

• Pros– Ultra high speed

– Multi standard

– Big estate / high power per slot

– Choice of star or full mesh backplane

– Redundant power supply

– Standard crate control

– Fits perfectly with modern FPGAs/ ASSPs

– Gaining momentum in HEP community

– Custom deployments can co-exist with standards

• Cons– Moderate choice of COTS cards on the market

– Commercial switches exist only for 1G/10G ethernet• Telecom driven

Advanced TCA

Agata Week – LNL 14 November 2007

Carrier Readout (1):

Main FPGA

DP RAM

Segment

Dual star ATCA network

Gigabit Ethernet Switch

LLP Carrier

PSA Farm

Agata Week – LNL 14 November 2007

Carrier Readout (2):

Main FPGA

DP RAM

Segments

Full mesh ATCA network

Optical Transceiver

PCI Express x1 @ 2.5 Gbps

PCI ExpressEndpoint

LLP CarrierPCI Express optical translator

Virtual I/O multiprocessor system

Agata Week – LNL 14 November 2007

Agata Week – LNL 14 November 2007

Clock Distribution

From GTS Tree

Agata Week – LNL 14 November 2007

MGT Clocking Layout

RocketIO101M

UX

MU

X

RocketIO102M

UX

MU

X

RocketIO103M

UX

MU

X

RocketIO105M

UX

MU

X

MGTclkM34/N34

MGTclkAP28/AP29

RocketIO106M

UX

MU

X

RocketIO109M

UX

MU

X

RocketIO110M

UX

MU

XRocketIO

112MU

XM

UX

RocketIO113M

UX

MU

X

MGTclkAP3/AP4

MGTclkJ1/K1

RocketIO114M

UX

MU

X

ATCA FABRIC CH1-CH2ATCA FABRIC CH1-CH2

ATCA FABRIC CH3-CH4ATCA FABRIC CH3-CH4

ATCA FABRIC CH5-CH6ATCA FABRIC CH5-CH6

ATCA FABRIC CH7-CH8ATCA FABRIC CH7-CH8

ATCA FABRIC CH9-CH10ATCA FABRIC CH9-CH10

ATCA FABRIC CH11-CH12ATCA FABRIC CH11-CH12

USER SFP TRANSCEIVERUSER SFP TRANSCEIVER

RTM PCI EXPRESS LANE0RTM PCI EXPRESS LANE0

RTM PCI EXPRESS LANE5RTM PCI EXPRESS LANE5

RTM PCI EXPRESS LANE1RTM PCI EXPRESS LANE1

RTM PCI EXPRESS LANE2RTM PCI EXPRESS LANE2

100250MHzPCI Express

JITTERATTENUATOR

200MHzGTS Clock

(**) The ATCA FABRIC channelsare routed from CHANNEL1 to CHANNEL12by switches

(***) User SFP could be used as 1GEnet or PCIExpressDAQ without FABRIC

RTM PCI EXPRESS LANE3RTM PCI EXPRESS LANE3

RTM PCI EXPRESS LANE4RTM PCI EXPRESS LANE4

AB

AB

AB

AB

AB

AB

AB

AB

AB

AB

INSPECTIONPADS

100MHzGTS Clock

OPTICALSFP

INSPECTIONPADS

LOCAL100MHz

(EPSON)

PHASE LOCKED

MGT clocking layout

Agata Week – LNL 14 November 2007

-48V

DC

ENABLE

P3V3-5A 16.5W MEZZANINE 1

MEZZANINE 2

MEZZANINE 3

MEZZANINE 4

MAIN BOARDP3V3-7A 23.1W

MAIN BOARDP2V5-7A 17.5W

FPGAs CORE

P1V2-7A 8.4W

FPGA MGTP1V2-4A 4.8W

P2V5-1.5A

P2V5-1.5A

P1V8-0.5APROMS

VCCAUX Fpga 1

VCCAUX MGT

P2V5-1.5A VCCAUX Fpga 2

P1V2-0.5AVTTTXs

P1V2-0.5A VTTRXs

MGT BUFFERS P1V8-6A 10.8W

P12V-14.7A 176.7(160.6)W

P3V3-5A 16.5W

P3V3-5A 16.5W

P3V3-5A 16.5W

M48V-4.0A 194.4(176.7)W

DC-DC Efficency is estimated at least 90%

ATC210 (210W)

P3V3_BOOT

4x LTM4600 55W

6x LTM4600 55W

P5V0-6A 30W

Carrier Power Supply

Agata Week – LNL 14 November 2007

• 1M x 18 true dual port RAM @ 100/200 MHZ• 800Mb/s LVDS streaming on data channels• Equalized and filtered distribution of 200MHZ

GTS clock• 1 PCI Express/ GE optical link• 15 x Full mesh connectivity on the backplane• Pervasive I2C bus for slow controls• 200W power supply• Multiple options for data readout

Carrier main features

Agata Week – LNL 14 November 2007

Fin

al R

outin

g (a

s of

apr

il 07

)

Agata Week – LNL 14 November 2007

Power and Signal Integrity Simulations

Agata Week – LNL 14 November 2007

Example Resonant mode between L6pwr/L11gnd

Agata Week – LNL 14 November 2007

Agata Week – LNL 14 November 2007

Agata Week – LNL 14 November 2007

CH-15 Eye Diagram with Equalization

Agata Week – LNL 14 November 2007

Int *ATCA0 = 0xfe001000; // DPRAM on board 0

Int *ATCA1 = 0xfe002000; // DPRAM on board 1

….

Fragment0 = memcpy(buffer0, ATCA0);

Fragment1= memcpy(buffer1, ATCA1);

PCI Express Readout Test