DAQ for new TPC readout

Sept. 3. 2007 TPC readoutupgade meeting, Budapest 1

DAQ for new TPC readoutDAQ for new TPC readout

Ervin Dénes, Zoltán FodorKFKI, Research Institute for Particle and Nuclear Physics

Sept. 3. 2007 TPC readoutupgade meeting, Budapest 2

DDL architectureDDL architecture

PCI Bus

Concentrator

DAQ Read-out Receiver

Card (D-RORC)

SourceInterface

Unit

ForwardChannel

(Raw data)

BackwardChannel

(Pedestals, control)

DestinationInterface

Unit

Detector Data Link (DDL) :- Source Interface Unit- Transmission media- Destination Interface Unit

Standarddetector/DAQ interface

Sept. 3. 2007 TPC readoutupgade meeting, Budapest 3

ALICE Detector Data LinkALICE Detector Data Link

Sept. 3. 2007 TPC readoutupgade meeting, Budapest 4

0,00

50,00

100,00

150,00

200,00

250,00

300,00

10 100 1000 10000 100000 1000000

Block size (32-bit words)

Ba

nd

wid

th (

MB

/s)

DDLD-RORC

Read-Out Receiver Cards (D-RORC)Read-Out Receiver Cards (D-RORC)

• PCI-X adapter for 2 DDLmax 100 MHz

• PCI master:autonomous DMA

Sept. 3. 2007 TPC readoutupgade meeting, Budapest 5

DDL FeaturesDDL FeaturesInterface:• Full duplex 32-bit data path on the destination interface (DIU card) • Half duplex 32-bit data path on the source interface (SIU card) • Full duplex flow control (XON/XOFF) • Interface clock up to 66 MHz (easy integration with PCI 66) • 264 MB/s peak data rate, 240 MB/s sustained bandwidth (max.)

Implementation:• Duplex LC optical link up to 300 m• 2x FC or 2x GbE physical layer components• Small Form Factor Pluggable (SFP) optical transceivers • Bit error rate < 10-12 • Robust error detection: very low undetected bit error rate < 10-40 • Automatic link synchronization and management • Radiation tolerant Source Interface

Extras:• Stand-by support (low power consumption) • In-system reconfiguration /  Remote system upgrade • Monitoring of the aging of laser diode of optical transceivers

Sept. 3. 2007 TPC readoutupgade meeting, Budapest 6

Readout System PerformanceReadout System Performance• Motherboard with dual Xeon CPUs @ 2.4 GHz

• Six PCI-X slots, 4 bus segments (3+1+1+1), 2 controllers

• Linux OS

• ALICE Data-Acquisition software (DATE)

Sept. 3. 2007 TPC readoutupgade meeting, Budapest 7

Performance: 6 D-RORCsPerformance: 6 D-RORCs• Testing the fully populated PC using data source internal

to PCI interface – Interoperability test

– Measure the maximal input bandwidth

PCI #6

PCI #5

PCI #4

PCI #3

PCI #2

PCI #1Seg

men

t #1

#2#3

#4

Con

trol

ler

#1#2

1 Ch 1 Ch

1 Ch 1 Ch 1 Ch

1 Ch 1 Ch 1 Ch 1 Ch

1 Ch 1 Ch

1 Ch 1 Ch 1 Ch

1 Ch 1 Ch 1 Ch 1 Ch 1Ch 1 Ch 1 Ch

264 464 424 528 792 1045 840

264 232 141.3 264 264 261.3 140

Aggregate Bandwidth [MB/s]Aggregate Bandwidth [MB/s]

Normalized Bandwidth Normalized Bandwidth [MB/s/Ch][MB/s/Ch]

Sept. 3. 2007 TPC readoutupgade meeting, Budapest 8

DDL SoftwareDDL Software• All functions accessible as interactive

commands or API• Script-based interpreter for sequence of

operations:– Sending command to the FEE– Reading FEE status

• printing the status• comparing the status• polling the status

– Downloading data into the FEE from a file

– Reading data from the FEE• writing data into a file• comparing data with data in a file

• TPC configuration: < 0.3 s

FEROFERO

DDL

define pedestal_addr 0x1FFF

define enable_pedestal 0x2C

reset SIU

write_command enable_pedestal

write_block pedestal_addr pedestal.hex %x

read_and_check_block pedestal_addr pedestal.hex %x

define pedestal_addr 0x1FFF

define enable_pedestal 0x2C

reset SIU

write_command enable_pedestal

write_block pedestal_addr pedestal.hex %x

read_and_check_block pedestal_addr pedestal.hex %x

D-RORCD-RORC

LDCLDC

Sept. 3. 2007 TPC readoutupgade meeting, Budapest 9

DDL TransactionsDDL Transactions

write_command <FEE command>

FEE command

No reply from the FEE

RORC Conc. FEE

BUS

FEE command

acq

Writing a command to the FEE (e.g. CLEAR)

Sept. 3. 2007 TPC readoutupgade meeting, Budapest 10

DDL TransactionsDDL Transactions

read_and_print <FEE address> “<format>” [<stream>]

FEE addressRORC Conc. FEE

BUS

BUS

BUSFEE statusFEE status

read_and_check <FEE address> <expected status> <mask>

read_until <FEE address> <expected status> <mask> <s>

At least 6FEE clocks

FEE address

Reading a status from the FEE

acq

Sept. 3. 2007 TPC readoutupgade meeting, Budapest 11

DDL TransactionsDDL Transactions

write_block <FEE address> <file> [<format>]

FEE address

FEERORC Conc. BUS

Data

FEE address

block lengthacq

Writing data to the FEE (e.g. pedestal)

end transactionacq

acq

Sept. 3. 2007 TPC readoutupgade meeting, Budapest 12

DDL TransactionsDDL Transactions

FEE address

BUS

BUS

block length

Data

end transaction

end block

FEE address

acq

acq

Reading data from the FEE (e.g. read back pedestal)

RORC Conc. FEE

BUS

SW

read_and_check_block <FEE address> <file> [<format>]

Sept. 3. 2007 TPC readoutupgade meeting, Budapest 13

DDL TransactionsDDL Transactions

start run

BUS

BUS

event length

Data

end transaction

end of event

Rdy to receive

acq

acq

Data collectionRORC Conc. FEE

BUS

SW

Data

event lengthend of event

end run

Sept. 3. 2007 TPC readoutupgade meeting, Budapest 14

The Free FIFOThe Free FIFO

D-RORC PC memory bankFirmware

Event building

page address

page address

page address

Free FIFO

PC CPU

Allocation of free pages

Sept. 3. 2007 TPC readoutupgade meeting, Budapest 15

Direct Memory AccessDirect Memory Access

D-RORCFirmware

PC memory bank

DDL

No involvement

PC CPU

Sept. 3. 2007 TPC readoutupgade meeting, Budapest 16

The Ready FIFOThe Ready FIFO

D-RORC PC memory bank

Event building

DDL

Ready FIFOFirmware

lengthpage status

lengthpage status

lengthpage status

Delivery of filled pages

PC CPU

Sept. 3. 2007 TPC readoutupgade meeting, Budapest 17

NA61 DAQ Configuration NA61 DAQ Configuration

VTPC1 VTPC2

MTPCR

MTPCL

Motherboards and concentrator

CAMAC TOFs

Trigger system

PC

RORC

RORC

RORC

RORC

DDL links

Gate & Clock to Motherboards

Busy

VME to PCI

?

Sept. 3. 2007 TPC readoutupgade meeting, Budapest 18

Timing DiagramTiming Diagram

Gate for event

Transfer to PC

Readout of FEE

Busy

Evt 1. Evt 2. Evt 3.

Read FEE to mem 1 Read FEE to mem 2

Transfer evt 1. to PC Transfer evt 2..

Busy generally = sampling time + FEE readout time, if the data flow fluid

Busy = 1, during the readout of the FEE or no free memory to read it

Sept. 3. 2007 TPC readoutupgade meeting, Budapest 19

Bandwidth EstimationsBandwidth Estimations

DDL/PCI bus:• 100 MHz, 64 bits

800 MB/s

Memory bus:• 2.4 GHz, 64 bits

19.2 GB/s

Requirement:• 100 Hz, size: 6 MB

600 MB/s

SATA speed:• 6 Gb/s,

750 MB/s

?