LEB 2002 @ ColmarLEB 2002 @ Colmar

•CMS DAQ overview•Data to Surface (D2S)

•Requirements•Topology•Hardware implementation•Measurements and simulations•Readout kit for FE designers•DAQ staging policy

•Conclusion

CMS Data to SurfaceCMS Data to Surfacetransportation transportation architecturearchitecture

attila.racz@cern.chon behalf of the CMS DAQ group

9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 2

1/8th DAQ Readout Builder:Lv-1 Max. trigger rate 12.5 kHzRU Builder (64x64) .125 Terabit/sFB fragment <size> ≈ 16 kBRU-BU systems 64Event filter power ≈ .7 TeraFlopEvent flow control ≈ 105 Mssg/sLocal mass storage 10 TByte

Data to surface:Average event size ≈ 1 MbyteFED S-link64 ports 658DAQ links (5 Gb/s) 512FED fragment <size> ≈ 2 kBFED builders (8x8) 64

TTC Timing, Trigger and Control TPD Trigger Primitive DataaTTS asynchronous Trigger Throttle SystemD2S Data to SurfaceFRL Frontend Readout LinkRU Readout UnitBU Builder UnitFU Filter UnitFFN Filter Farm NetworkEVM Event ManagerRCN Readout Control NetworkBCN Builder Control NetworkDCN Detector Control networkDSN DAQ Service Network

8-fold DAQ architecture8-fold DAQ architecture

DAQ staging

9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 3

D2S requirementsD2S requirements

• Reads the data from the FE data sources (~630 sources)

• Merging of 2 “small” sources (nominal size is 2 KBytes)

• Leveling effect on un-balanced data sources

• Staging capability according to the 8-fold DAQ architecture

9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 4

Data to Surface (D2S)Data to Surface (D2S)

Short link(20m)

Long link(200m)

Underground area

Surface area

FED/DCC

FRL

FED Builder

RU

FED/DCC

FRL

FED/DCC

FRL

FED/DCC

FRL

FED/DCC

FRL

FED/DCC

FRL

FED/DCC

FRL

FED/DCC

FRL

9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 5

Data sourcesData sources

Detector Number of data sources

Number of DAQ links

Event size per link

KB

Fluctuations RMS

Nominal event

size KB (pp run)

Number of

trigger partition

s

Pixel 40 36 (merg) 1.2 – 2.0 30% 72 2

Tracker 440 272 (merg) 0.4 – 1.5 0.27 KB 300 4

Preshower

47 47 2.3 ? 110 2

ECAL 52 (DCC) 52 2 1 KB ~100 4

HCAL 24 (DCC) 24 1.7 ? 48 6

Muons CSC

9 (DCC 4-1)

9 2 Huge… 16* 2

Muons RPC

5 (DCC) 5 .3 ? 1.5 4

Muons DT 5 5 1.6 ? 8 2

Glob. trig 4 4 2 none 8 3

DT track f.

4 4 2 none 8 2

Total 630 458 671.5 31

* Most of the time, the detector is empty: one muon produces 5.5KB

These numbers are currently revised/updated and they will change !Event sizes are strongly dependent on simulated occupancies…

9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 6

D2S in vivo…D2S in vivo…

From one end of the DAQ buildingto the end of the control room,thecable path is 187 m long and there are~90 m of verticality

9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 7

Data sources location (1)Data sources location (1)

9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 8

Data sources location (2)Data sources location (2)

9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 9

S-L

lin

k64

to

FR

L M

erg

er

FE

D B

uil

de

rR

ead

ou

t U

nit

In

pu

t

D2S baseline implementationD2S baseline implementationLVDS + Custom FRL+MyrinetMyrinet

9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 10

Short distance linkShort distance link

• LVDS signals, S-link64 compliant– 869 MB/sec @ 10m– 480 MB/sec @ 17m

9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 11

AlteraACEX

60MHz

FED

FED

AlteraACEX

60MHz

FIFO32kB

FIFO32kB

FRL

S-Link64 -64-bit up to 100MHz -Backpressure -auto-test (Slink spec.) -12 reserved pins

LVDS transfer -64-bit @ 60MHz (480MB/s) -cable length 15m max.

ALTERAStratrix

-no Slink pinout-64-bit PCI clock-able to read one port at the time

FRL merging featureFRL merging feature

9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 12

FRL baseline implementationFRL baseline implementation

PCI Myrinet board

160 mm 30 mmStandard size

9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 13

DAQ Opticable: a candidateDAQ Opticable: a candidate

•200m long•72 fibers, 50/125um multimode, LC-LC connector•32 DAQ links (4 spares) per cable,•16 cables for CMS DAQ

Polyester Tape

Aramid Yarn

PVC Coated Aramid C.S.M.

Aramid Yarn

Tight Buffer Optical Fiber

PVC Sub-unit Jacket *

Sub-unit

HFFR Jacket *

Rip Cord.

9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 14

0

10

20

30

40

50

60

70

80

90

100

0 0.25 0.5 0.75 1 1.25

RMS/average

Max

imu

m U

tilis

atio

n (

%)

EVB traffic

FED builder performances FED builder performances (Myrinet)(Myrinet)

• rms=0 (fixed size): puts itself in Barrel Shifter • variable size and no traffic shaping: ~55% usage

RU1

8x8 crossbar

RU2 RU3 RU4

9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 15

0

10

20

30

40

50

60

70

80

90

100

0 0.25 0.5 0.75 1 1.25

RMS/average

Max

imu

m U

tilis

atio

n (

%)

balanced (2.0)

unbalance ratio=2 (1.33 2.66)

unbalance ratio=3 (1.0 3.0)

FED Builder - Unbalanced Inputs FED Builder - Unbalanced Inputs

•Maximum usage: roughly 50%•No significant loss due to un-balanced sources

9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 16

• Myrinet Lanai9 (1 link @ 2.5 Gbit/sec), jumbo event (200k)

Time

Event

ID

TimeEvent

ID

no MTU with MTU

““Jumbo” fragment effect…(1)Jumbo” fragment effect…(1)

9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 17

• Myrinet Lanai10 (2 links @ 2.5 Gbit/sec), jumbo event (200k)

Time

Event

ID

TimeEvent

ID

no MTU with MTU

““Jumbo” fragment effect…(2)Jumbo” fragment effect…(2)

9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 18

Readout Kit for FED designersReadout Kit for FED designers

• PCI generic III (see D.Gigi’s talk), Tx-card (PMC form factor) + Rx-card, cable (15m max.)

• Software libraries and drivers under Linux

• Supported by X-DAQ (Common DAQ software framework)

• Documented !!!

• WEB download: http: //cern.ch/cano/fedkit/

• Provides an easy transition from local readout to central DAQ readout

• Price: ~ 2500 FS

9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 19

DAQ staging policyDAQ staging policy

Staging

FED/DCC

FRL

FED Builder

FED/DCC

FRL

FED/DCC

FRL

FED/DCC

FRL

FED/DCC

FRL

FED/DCC

FRL

FED/DCC

FRL

FED/DCC

FRL

1 RU builder 64x64, 1MByte @ 12.5KHz

RURU

2 RU builders 64x64, 1MByte @ 25KHz3 RU builders 64x64, 1MByte @ 37.5KHz

RU8 RU builders 64x64, 1MByte @ 100KHz

RURURURURU

9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 20

ConclusionConclusion

• D2S structure and implementation well defined and matches the requirements

• Minimal custom electronics developments– Can profit from the technological improvements

• Unexpected situations can be handled simply by adding devices– Event size increase– Number of sources increase

• D2S pre-building allows a progressive DAQ deployment

9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 21

FED/DCC DAQ portFED/DCC DAQ port

SpecificDetector Electronic

S-LINK64 port

LinkFPGA

FED/DCC

Detector links

Storagearea

S-LINK64 port isthe border between FEand DAQ

SetupControlMessages - out-of-sync - failureMonitoringLocal readout

VME HostInterface

Fast signals: (FMM) - busy/ready - overflow warning - out-of-synch

9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 22

S-LINK64 featuresS-LINK64 features

• Based on S-LINK

• Specifies 2 connectors but not the physical link in-between

• S-LINK is 32 bits @ 40 MHz max, flow-control, K/D words

• To match CMS needs:

– 64 bit data path, clock speed up to 100 MHz, but link speed is ~500 MB/s

– no other changes to keep compatibility with S-LINK products

• S-LINK64 specs. available at:

– http://cmsdoc.cern.ch/cms/TRIDAS/horizontal/

9-13-September 2002 Attila RACZ CERN/EP-CMD LEB 2002 @ Colmar 23

16 15

8 7

12 1132 31 4 3 060 5963 56 55

60 5963

20 19 4 3 0

032 31

Evt_stat(8) xx$$

BOE_1K LV1_id(24) BX_id(12)

CRC (16)

D Sub-detector payload

K

D Sub-detector payload

FED common data formatFED common data format

32 3160 5963

Evt_ty

56 55

EOE Evt_lgth(24)xxxx

Source_id(10+2) FOV

xxxx

$$Hx

1

BOE_2K $$

The FED is generating the above encapsulation.